gluekit 1.0.1.dev1__tar.gz

gluekit-1.0.1.dev1/PKG-INFO

@@ -0,0 +1,1176 @@
+Metadata-Version: 2.4
+Name: gluekit
+Version: 1.0.1.dev1
+Summary: A developer‑centric Glue CLI built to speed local iteration with git/uv‑style commands, while bridging to highly controlled (eg Fedramped) AWS consoles for PII‑sensitive runs and smoothing onboarding around Glue’s quirks—like the lack of script‑to‑notebook conversion and the friction of refactoring Glue scripts.
+Author-email: Mike Kranz <kranz-michael@norc.org>
+Keywords: python
+Requires-Python: <4.0,>=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: boto3>=1.28.0
+Requires-Dist: python-dotenv>=1.2.1
+Requires-Dist: python-slugify>=8.0.4
+Requires-Dist: pyyaml>=6.0.0
+Requires-Dist: rapidfuzz>=3.14.3
+Requires-Dist: typer>=0.9.0
+Provides-Extra: job-dev
+Requires-Dist: moto[s3,ssm]>=5.1.16; extra == "job-dev"
+Requires-Dist: sqlframe[duckdb]>=4.3.0; extra == "job-dev"
+
+# gluekit
+
+## Introduction
+
+`gluekit` is reusable AWS Glue job tooling for local-first development. It is designed for teams that want Glue job definitions, scripts, notebooks, and supporting artifacts in source control while still being able to pull from AWS, edit safely, validate locally, and push intentional changes back.
+
+The command line is meant to feel closer to `git` and `uv` than to a cloud console: quick local commands, explicit artifact layouts, checkout-style context, and conversion workflows that smooth over AWS Glue quirks without hiding the underlying AWS behavior. A major motivation is making script and notebook work interchangeable enough for day-to-day development while still respecting that Glue scripts and Glue interactive notebooks are not identical runtime surfaces.
+
+**Local Development Emulators:** `gluekit` provides scoped `awsglue` and `pyspark` emulators during local runs, using [SQLFrame](https://sqlframe.readthedocs.io/) when available, so you can test jobs rapidly on your local machine without AWS compute. See the [Local Development](docs/local_development.md) documentation for more details.
+
+## Suggested workflows
+
+For AWS CLI-oriented setup, sync, and deployment recipes, start with the [AWS CLI recipes](docs/aws_cli_recipes/) folder, especially [docs/aws_cli_recipes/workflows.md](docs/aws_cli_recipes/workflows.md).
+
+Common starting points:
+
+- Pull a real Glue job into source control, edit the local script/config, and push selected artifacts back with `gluekit pull`, `gluekit checkout`, `gluekit edit`, and `gluekit push`.
+- Use `gluekit local setup`, `gluekit local s3`, `gluekit local ssm`, and `gluekit run` to build a repeatable local test fixture before touching AWS.
+- Keep account-specific values such as IAM roles, temp buckets, and script buckets in profile-scoped saved params so local configs can move between dev/prod-like AWS profiles without hand-editing generated artifacts.
+
+## Script-based development with notebook interoperability
+
+`gluekit` supports a script-first workflow that can still interoperate with notebooks. Python scripts can use `#%%` cell tags to mark notebook-like cells. These tags divide a `.py` file into executable notebook cells; Markdown cells can use `#%% [markdown]`. This follows the same family of conventions described by VS Code's [Python Interactive/Jupyter documentation](https://code.visualstudio.com/docs/python/jupyter-support-py#_convert-jupyter-notebooks-to-python-code-file), where Python files can contain Jupyter-style cell delimiters and be converted back into notebooks.
+
+The intent is flexibility: you can develop in a normal Python file, use editor support for cell-by-cell execution, keep clean script diffs in git, and generate a Glue notebook when that is the deployment or debugging surface you need.
+
+If you have already selected the job with checkout:
+
+```bash
+gluekit checkout {jobname}
+gluekit convert --from script --to notebook
+```
+
+If you have not checked out the job, pass it explicitly:
+
+```bash
+gluekit convert {jobname} --from script --to notebook
+```
+
+## Gluekit vs AWS CLI terminology
+
+Some `gluekit` commands look like AWS CLI commands, but the scope is different:
+
+- `gluekit pull`, `gluekit push`, `gluekit build --push`, and `gluekit init` use real AWS Glue/S3 APIs. On those commands, `--profile` means an AWS CLI credential profile, the same profile name boto3 would use.
+- `gluekit checkout --profile` and `gluekit config set --profile` update local gluekit state in `.gluekit/glue_set.json`. The value is a gluekit profile scope for saved params; it usually matches an AWS profile name, but those commands do not contact AWS.
+- `gluekit local ...` and `gluekit run` are local test tooling. Their S3, SSM, region, job, and profile values are labels or mocked fixture values stored in JSON and used by local moto/boto3 mocks. They do not create, read, or delete real AWS resources.
+
+## Install
+
+### If using `uv` as a global tool (recommended)
+
+Ensure you have `uv` installed globally (see [uv installation docs](https://uv.pypa.io/en/latest/installation.html#installing-uv-globally)) and then run:
+
+
+```bash
+uv tool install https://github.com/NORCUChicago/gluekit.git
+```
+
+### If using `pip`:
+
+```bash
+pip install https://github.com/NORCUChicago/gluekit.git
+
+```
+
+## Quick start
+
+1. Install the AWS command line tool and configure your credentials. If using sso, need to log in with 
+```bash
+aws sso login --profile your-profile-name
+```
+
+
+2. Right now, the best way to get started is to check out the [Command line help documentation](#command-line-help-documentation) below, which is auto-generated from the live CLI `--help` output and includes usage examples for each command.
+
+## Quick start commands
+
+### Glue job management
+
+Initialize a workspace from your AWS CLI credential profiles:
+
+```bash
+gluekit init
+```
+
+Pull or check out the job you want to work on, then make focused config changes:
+
+```bash
+gluekit checkout my-job --profile survey-dev
+gluekit edit --role arn:aws:iam::111111111111:role/glue-dev --temp-dir s3://dev-bucket/tmp/
+```
+
+Add local artifacts or shared files to the checked-out job:
+
+```bash
+gluekit add dist/my_pkg-0.0.1-py3-none-any.whl --package-whl
+gluekit add glue/lib/helpers.py --as-path
+```
+
+Convert between Glue scripts and notebooks when your workflow changes:
+
+```bash
+gluekit convert my-job --from script --to notebook
+gluekit convert my-job --from notebook --to script
+```
+
+Build Python artifacts and optionally push the checked-out job updates:
+
+```bash
+gluekit build
+gluekit build --bump-version patch --package-whl --push
+```
+
+### Local Glue job development
+
+Create and check out the local setup with mocked AWS resource mappings:
+
+```bash
+gluekit local setup \
+  --profile local-dev \
+  --s3-root sources/s3 \
+  --s3-bucket local-dev=tests/fixtures \
+  --ssm-param /app/runtime=local
+```
+
+Inspect the saved setup, add SSM parameters, and run the Glue script locally:
+
+```bash
+gluekit local status
+gluekit local ssm put /app/runtime local
+gluekit run glue/scripts/my-job.py --glue-arg run_date=2026-06-18
+```
+
+Use `--s3-root sources/s3` to map an entire mocked profile/account S3 namespace, where `sources/s3/{bucket}/{prefix/key}` appears as `s3://{bucket}/{prefix/key}`. Use `--s3-bucket` as `{s3 bucket in mocked profile}={local directory}` for a single bucket, and `--ssm-param` as `{ssm parameter in mocked profile}={local parameter value}`.
+
+Local setup writes fixture data to `.gluekit/local.json` and records the active local settings under `checkout.local` in `.gluekit/glue_set.json`. The local `--profile` value is saved in both places as a cross-reference label for comparison and validation; `gluekit local setup` does not authenticate to AWS.
+
+## Generated metadata
+
+gluekit now includes two checked-in metadata artifacts used by notebook conversion and Glue job validation helpers:
+
+- `src/gluekit/resources/glue_job_schema.json` is generated from the local botocore AWS Glue service model.
+- `src/gluekit/resources/magic_map.json` is a curated mapping of Glue job config fields to AWS Glue Interactive Sessions magics.
+
+The schema artifact is generated by:
+
+```bash
+uv run python scripts/extract_glue_schema.py
+```
+
+Rerun that script when changing schema-driven validation logic, when upgrading boto3/botocore in a way that could change the Glue service model, or when you need to refresh the checked-in schema after AWS Glue API changes.
+
+The magic map is intentionally maintained by hand because it reflects documented notebook behavior rather than raw AWS API structure. When updating notebook conversion behavior, review the AWS Glue Interactive Sessions magics docs and keep `src/gluekit/resources/magic_map.json` in sync.
+
+## edit workflow
+
+`gluekit edit` is checkout-first. The default flow is to select a job once with `gluekit checkout my-job` and then run `gluekit edit ...` against that active local context.
+
+Use `gluekit add` when you want the older inferred artifact behavior for local paths, wheels, zip files, and remote module specifiers. Use `gluekit update` when you need a raw property-path setter. Both remain available for compatibility, but `gluekit edit` is the primary mutation surface.
+
+For the common package-wheel flow, run `gluekit add --package-whl` after building the project. It uses the single wheel matching `dist/*.whl`, routes it to `--additional-python-modules`, replaces the previously tracked `dist/*.whl` package wheel, updates `SourceControlDetails.AdditionalPythonFiles`, and refreshes an existing notebook's generated Glue magic cells.
+
+If you need to target a different local config for one command, pass the job name directly to `gluekit edit <job-name> ...`, or keep using `gluekit add --job-name other-job <items...>` for the compatibility workflow. That stays parallel to GitHub CLI's context-first workflow, where a flag like `--repo` overrides the ambient target without changing the saved default context.
+
+## Profile-specific params and build automation
+
+Use profile-scoped saved params for account-dependent values such as IAM roles, script buckets, temp buckets, and other Glue default arguments. These are local gluekit param scopes saved in `.gluekit/glue_set.json`; they are only used as real AWS credential profiles by commands that call AWS, such as `pull`, `push`, `build --push`, and `init`.
+
+```bash
+gluekit config set my-job --profile survey-dev --Role=arn:aws:iam::111111111111:role/glue-dev
+gluekit config set my-job --profile survey-prod --Role=arn:aws:iam::222222222222:role/glue-prod
+gluekit config set my-job --profile survey-prod --command_script_location=s3://prod-bucket/scripts/my-job.py
+```
+
+Then check out the job and gluekit profile scope together:
+
+```bash
+gluekit checkout my-job --profile survey-prod
+```
+
+After a job is checked out, switch only the active profile scope with:
+
+```bash
+gluekit checkout --profile survey-dev
+```
+
+Checkout stores the active gluekit profile scope in `.gluekit/glue_set.json` and applies the matching saved params to the local config immediately, including mapped Glue config paths such as `Role`, `Command.ScriptLocation`, `DefaultArguments.*`, and `SourceControlDetails.*`. `gluekit pull` validates existing active profile mappings against the pulled config; if the active profile has no mappings yet but another profile exists for the job, pull detects and saves role plus S3/config mappings using the same profile comparison rules as `gluekit init`. `gluekit push` validates that the active profile mappings already align with the local config before uploading, and reports every mismatch instead of rewriting the config at push time. `gluekit pull`, `gluekit push`, and `gluekit build --push` also use the active profile value as the AWS CLI credential profile when `--profile` is not passed.
+
+The common package release loop can now be a single command:
+
+```bash
+gluekit build --bump-version patch --package-whl --push
+```
+
+That bumps `pyproject.toml`, builds artifacts, refreshes the checked-out job's tracked `dist/*.whl` in `--additional-python-modules`, applies active profile params before deriving S3 paths, and pushes the selected Glue artifacts/config. Use `--dry-run` to preview the build, wheel update, and push steps.
+
+For first-time repository setup, `gluekit init` scans your AWS CLI config profiles, prompts for the Glue job to inspect in each selected profile, can save profile-specific role and S3 params locally, and sets the first checked-out job/profile scope.
+
+### Command line help documentation
+<!-- BEGIN: CLI_HELP -->
+> Auto-generated from live CLI `--help` output.
+
+### `gluekit --help`
+
+```text
+Usage: gluekit [OPTIONS] COMMAND [ARGS]...
+
+Options:
+  --install-completion  Install completion for the current shell.
+  --show-completion     Show completion for the current shell, to copy it or
+                        customize the installation.
+  --help                Show this message and exit.
+
+Commands:
+  build     Build local artifacts; only touches AWS when --push is used.
+  checkout  Replace the active local checkout selection.
+  edit      Edit one local Glue job config with explicit mutation flags.
+  clone     Clone local Glue job configurations and artifacts.
+  set       Set reusable local key/value parameters for a Glue job.
+  convert   Convert Glue jobs between script and notebook formats.
+  init      Initialize local repo-level Gluekit settings from AWS CLI...
+  list      List real AWS Glue jobs in the configured AWS account.
+  pull      Download real AWS Glue job artifacts and config into the local...
+  push      Upload local artifacts to real AWS Glue/S3 and optionally...
+  run       Run a Glue script locally with emulated Glue libraries and...
+  remove    Remove local Glue job configs and local artifacts.
+  status    Show the active local checkout and its local-to-S3 artifact...
+  sync      Sync generated Glue config metadata cells in local...
+  add       Compatibility command for inferred artifact and module additions.
+  update    Compatibility command for generic property updates.
+  config    Show and set local reusable Glue config parameters.
+  local     Manage local-only Glue development setups and mocked AWS fixtures.
+```
+
+
+### `gluekit pull --help`
+
+```text
+Usage: gluekit pull [OPTIONS] [JOB_NAME] [TARGET_JOB]
+
+  Download real AWS Glue job artifacts and config into the local workspace.
+
+Arguments:
+  [JOB_NAME]    Glue job name or pattern to pull. Use "*" for all jobs.
+  [TARGET_JOB]  Optional local job name to write pulled artifacts to.
+
+Options:
+  -f, --force                     Download even if the local config is newer or
+                                  equal to LastModifiedOn.
+  --dry-run                       Show what would be downloaded without writing
+                                  files.
+  -i, --include TEXT              Include only specific components (config,
+                                  script, notebook, extra-py-files, extra-
+                                  files).
+  -x, --exclude TEXT              Exclude specific components (config, script,
+                                  notebook, extra-py-files, extra-files).
+  --config-dir PATH               Directory containing Glue job config files.
+                                  [default: glue/configs]
+  --write-job-list / --no-write-job-list
+                                  Write glue/glue_full_job_list.csv during the
+                                  run.  [default: write-job-list]
+  -p, --profile TEXT              AWS CLI credential profile used for real AWS
+                                  Glue/S3 API calls; not a gluekit local test
+                                  profile.
+  --auto-login / --no-auto-login  For real AWS profiles, automatically run 'aws
+                                  sso login' when credentials are missing or
+                                  expired.  [default: auto-login]
+  --help                          Show this message and exit.
+
+  **Examples**
+
+  - `gluekit pull my-job --profile my-sso-profile`
+
+  - `gluekit pull source-job target-job --profile my-sso-profile`
+
+  - `gluekit pull "my-job-\*" --force --include config,script --exclude
+  notebook`
+
+  - `gluekit pull "\*" --dry-run --profile my-sso-profile --no-auto-login`
+```
+
+
+### `gluekit edit --help`
+
+```text
+Usage: gluekit edit [OPTIONS] [JOB_NAME]
+
+  Edit one local Glue job config with explicit mutation flags.
+
+Arguments:
+  [JOB_NAME]  Glue job name to edit. Defaults to the active checkout selection.
+
+Options:
+  --description TEXT              Set the job description.
+  --role TEXT                     Set the IAM role for the job.
+  --glue-version TEXT             Set the Glue version.
+  --worker-type TEXT              Set the worker type.
+  --number-of-workers INTEGER     Set the number of workers.
+  --timeout INTEGER               Set the job timeout in minutes.
+  --max-retries INTEGER           Set the max retry count.
+  --execution-class TEXT          Set the execution class.
+  --max-concurrent-runs INTEGER   Set the max concurrent runs.
+  --job-run-queuing-enabled / --no-job-run-queuing-enabled
+                                  Enable or disable job run queuing.
+  --security-configuration TEXT   Set the security configuration name.
+  --maintenance-window TEXT       Set the maintenance window.
+  --log-uri TEXT                  Set the log URI.
+  --command-name TEXT             Set Command.Name.
+  --python-version TEXT           Set Command.PythonVersion.
+  --runtime TEXT                  Set Command.Runtime.
+  --remote-script-location TEXT   Set Command.ScriptLocation.
+  --local-script-location TEXT    Set SourceControlDetails.ScriptLocation.
+  --notebook-location TEXT        Set SourceControlDetails.NotebookLocation and
+                                  NotebookPath.
+  --source-control-provider TEXT  Set SourceControlDetails.Provider.
+  --source-control-owner TEXT     Set SourceControlDetails.Owner.
+  --source-control-repository TEXT
+                                  Set SourceControlDetails.Repository.
+  --source-control-branch TEXT    Set SourceControlDetails.Branch.
+  --source-control-folder TEXT    Set SourceControlDetails.Folder.
+  --add-connection TEXT           Add one or more Glue connections.
+  --remove-connection TEXT        Remove one or more Glue connections.
+  --add-additional-python-modules TEXT
+                                  Add one or more values to --additional-python-
+                                  modules.
+  --remove-additional-python-modules TEXT
+                                  Remove one or more values from --additional-
+                                  python-modules.
+  --add-extra-py-files TEXT       Track one or more local relative Python files
+                                  in SourceControlDetails.ExtraPyFiles and
+                                  --extra-py-files.
+  --remove-extra-py-files TEXT    Remove one or more local relative Python files
+                                  from SourceControlDetails.ExtraPyFiles and
+                                  --extra-py-files.
+  --add-extra-jars TEXT           Add one or more values to --extra-jars.
+  --remove-extra-jars TEXT        Remove one or more values from --extra-jars.
+  --add-additional-python-files TEXT
+                                  Track one or more local paths in
+                                  SourceControlDetails.AdditionalPythonFiles and
+                                  --extra-py-files.
+  --remove-additional-python-files TEXT
+                                  Remove one or more local paths from
+                                  SourceControlDetails.AdditionalPythonFiles and
+                                  --extra-py-files.
+  --add-extra-files TEXT          Track one or more local paths in
+                                  SourceControlDetails.ExtraFiles and --extra-
+                                  files.
+  --remove-extra-files TEXT       Remove one or more local paths from
+                                  SourceControlDetails.ExtraFiles and --extra-
+                                  files.
+  --add-default-argument TEXT     Set a DefaultArguments entry using KEY=VALUE.
+                                  Repeat as needed.
+  --remove-default-argument TEXT  Remove a DefaultArguments entry by key. Repeat
+                                  as needed.
+  --add-non-overridable-argument TEXT
+                                  Set a NonOverridableArguments entry using
+                                  KEY=VALUE. Repeat as needed.
+  --remove-non-overridable-argument TEXT
+                                  Remove a NonOverridableArguments entry by key.
+                                  Repeat as needed.
+  --enable-metrics / --disable-metrics
+                                  Set DefaultArguments.--enable-metrics to true
+                                  or false.
+  --enable-observability-metrics / --disable-observability-metrics
+                                  Set DefaultArguments.--enable-observability-
+                                  metrics to true or false.
+  --enable-glue-datacatalog / --disable-glue-datacatalog
+                                  Set DefaultArguments.--enable-glue-datacatalog
+                                  to true or false.
+  --enable-continuous-cloudwatch-log / --disable-continuous-cloudwatch-log
+                                  Set DefaultArguments.--enable-continuous-
+                                  cloudwatch-log to true or false.
+  --enable-spark-ui / --disable-spark-ui
+                                  Set DefaultArguments.--enable-spark-ui to true
+                                  or false.
+  --enable-job-insights / --disable-job-insights
+                                  Set DefaultArguments.--enable-job-insights to
+                                  true or false.
+  --enable-auto-scaling / --disable-auto-scaling
+                                  Set DefaultArguments.--enable-auto-scaling to
+                                  true or false.
+  --job-bookmark-option TEXT      Set DefaultArguments.--job-bookmark-option.
+  --temp-dir TEXT                 Set DefaultArguments.--TempDir.
+  --spark-event-logs-path TEXT    Set DefaultArguments.--spark-event-logs-path.
+  --datalake-formats TEXT         Set DefaultArguments.--datalake-formats.
+  --spark-conf TEXT               Set DefaultArguments.--conf.
+  --continuous-log-log-group TEXT
+                                  Set DefaultArguments.--continuous-log-
+                                  logGroup.
+  --continuous-log-log-stream-prefix TEXT
+                                  Set DefaultArguments.--continuous-log-
+                                  logStreamPrefix.
+  --continuous-log-conversion-pattern TEXT
+                                  Set DefaultArguments.--continuous-log-
+                                  conversionPattern.
+  --executor-cores INTEGER        Set DefaultArguments.--executor-cores.
+  --customer-driver-env-vars TEXT
+                                  Set DefaultArguments.--customer-driver-env-
+                                  vars.
+  --customer-executor-env-vars TEXT
+                                  Set DefaultArguments.--customer-executor-env-
+                                  vars.
+  --python-modules-installer-option TEXT
+                                  Set DefaultArguments.--python-modules-
+                                  installer-option.
+  --dry-run                       Show what would be updated without writing
+                                  files.
+  --config-dir PATH               Directory containing Glue job config files.
+                                  [default: glue/configs]
+  --help                          Show this message and exit.
+
+  **Examples**
+
+  - `gluekit edit my-job --description 'Updated job description' --glue-version
+  5.0`
+
+  - `gluekit edit --add-additional-python-files glue/libs/helper.py --add-extra-
+  files glue/resources/job.json`
+
+  - `gluekit edit my-job --temp-dir s3://bucket/tmp --remove-default-argument
+  legacy-arg --enable-metrics`
+
+  - `gluekit edit my-job --enable-auto-scaling --enable-job-insights --executor-
+  cores 8`
+```
+
+
+### `gluekit add --help`
+
+```text
+Usage: gluekit add [OPTIONS] [ARGS]...
+
+  Compatibility command for inferred artifact and module additions. Prefer edit
+  for explicit config changes.
+
+Arguments:
+  [ARGS]...  Paths or module specifiers to add to the selected Glue job config.
+
+Options:
+  --job-name TEXT    Override the active checked-out job for this command only.
+  --as-path          Treat all items as local paths.
+  --as-pypi          Treat all items as PyPI modules.
+  --package-whl      Use the package wheel from dist/*.whl.
+  --dry-run          Show what would be updated without writing files.
+  --config-dir PATH  Directory containing Glue job config files.  [default:
+                     glue/configs]
+  --help             Show this message and exit.
+
+  **Examples**
+
+  - `gluekit checkout my-job`
+
+  - `gluekit add dist/my-package.whl`
+
+  - `gluekit add --job-name other-job requests==2.32.3 --as-pypi`
+```
+
+
+### `gluekit convert --help`
+
+```text
+Usage: gluekit convert [OPTIONS] [JOB_NAME]
+
+  Convert Glue jobs between script and notebook formats.
+
+Arguments:
+  [JOB_NAME]  Glue job name to convert. Defaults to the active checkout
+              selection.
+
+Options:
+  --from TEXT                   Source format (script or notebook).  [default:
+                                script]
+  --to TEXT                     Target format (script or notebook).  [default:
+                                notebook]
+  --use-config / --script-only  Use config metadata (magics and paths) when
+                                available.  [default: use-config]
+  --dry-run                     Show what would be converted without writing
+                                files.
+  --config-dir PATH             Directory containing Glue job config files.
+                                [default: glue/configs]
+  --help                        Show this message and exit.
+
+  **Examples**
+
+  - `gluekit convert my-job --from script --to notebook --use-config`
+
+  - `gluekit convert my-job --from notebook --to script --script-only`
+```
+
+
+### `gluekit sync --help`
+
+```text
+Usage: gluekit sync [OPTIONS] [JOB_NAME]
+
+  Sync generated Glue config metadata cells in local script/notebook files.
+
+Arguments:
+  [JOB_NAME]  Glue job name to sync. Defaults to the active checkout selection.
+
+Options:
+  --dry-run          Show what would be updated without writing files.
+  --config-dir PATH  Directory containing Glue job config files.  [default:
+                     glue/configs]
+  --help             Show this message and exit.
+
+  **Examples**
+
+  - `gluekit sync my-job`
+
+  - `gluekit sync my-job --dry-run`
+```
+
+
+### `gluekit status --help`
+
+```text
+Usage: gluekit status [OPTIONS]
+
+  Show the active local checkout and its local-to-S3 artifact mappings.
+
+Options:
+  --config-dir PATH  Directory containing Glue job config files.  [default:
+                     glue/configs]
+  --help             Show this message and exit.
+
+  **Examples**
+
+  - `gluekit status`
+
+  - `gluekit status --config-dir glue/configs`
+```
+
+
+### `gluekit run --help`
+
+```text
+Usage: gluekit run [OPTIONS] SCRIPT_PATH
+
+  Run a Glue script locally with emulated Glue libraries and mocked AWS
+  services.
+
+Arguments:
+  SCRIPT_PATH  Path to the Glue Python script to execute locally.  [required]
+
+Options:
+  --glue-arg TEXT       Glue script argument in KEY=VALUE form; passed through
+                        as --KEY VALUE.
+  --job-name TEXT       Default JOB_NAME used when not provided in script args.
+                        [default: local-glue-job]
+  --create-bucket TEXT  Create a mocked S3 bucket only inside the local run.
+                        Repeat as needed.
+  --ssm-parameter TEXT  Seed a mocked SSM parameter only inside the local run as
+                        NAME=VALUE. Repeat as needed.
+  --aws-region TEXT     Mock AWS region used for local boto3 clients. Defaults
+                        to us-east-1.
+  --config-file FILE    Local fixture config file for mocked S3/SSM. Defaults to
+                        .gluekit/local.json when local settings are checked out.
+  --help                Show this message and exit.
+
+  **Examples**
+
+  - `gluekit run ../ppsc-survey/glue/scripts/job_a.py --create-bucket my-input-
+  bucket`
+
+  - `gluekit run glue/scripts/job-a.py --ssm-parameter /app/env=dev --glue-arg
+  report_date=2026-06-18`
+
+  - `gluekit run glue/scripts/job-a.py -- --JOB_NAME custom-local-job
+  --report_date 2026-06-18`
+```
+
+
+### `gluekit local --help`
+
+```text
+Usage: gluekit local [OPTIONS] COMMAND [ARGS]...
+
+  Manage local-only Glue development setups and mocked AWS fixtures.
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  setup   Create or update and check out the local-only development setup.
+  status  Show the active local development setup.
+  s3      Manage local mocked S3 fixture mappings; does not call AWS S3.
+  ssm     Manage local mocked SSM Parameter Store values; does not call AWS...
+```
+
+
+### `gluekit local setup --help`
+
+```text
+Usage: gluekit local setup [OPTIONS]
+
+  Create or update and check out the local-only development setup.
+
+Options:
+  --profile TEXT    Optional gluekit/AWS profile name to associate with this
+                    local setup in checkout state and .gluekit/local.json; this
+                    command does not authenticate to AWS.
+  --s3-bucket TEXT  Map a mocked S3 bucket to a local directory in {s3 bucket in
+                    mocked profile}={local directory} form. Repeat as needed.
+  --s3-root PATH    Map a local directory as the full mocked S3 namespace for
+                    this setup/profile. Expected layout: {local s3
+                    root}/{bucket}/{prefix/key}.
+  --ssm-param TEXT  Map a mocked SSM parameter to a local value in {ssm
+                    parameter in mocked profile}={local parameter value} form.
+                    Repeat as needed.
+  --help            Show this message and exit.
+
+  **Examples**
+
+  - `gluekit local setup --profile local-dev --s3-root sources/s3`
+
+  - `gluekit local setup --s3-bucket local-dev=tests/fixtures`
+
+  - `gluekit local setup --ssm-param /app/runtime=local`
+```
+
+
+### `gluekit local status --help`
+
+```text
+Usage: gluekit local status [OPTIONS]
+
+  Show the active local development setup.
+
+Options:
+  --help  Show this message and exit.
+
+  **Examples**
+
+  - `gluekit local status`
+```
+
+
+### `gluekit local s3 --help`
+
+```text
+Usage: gluekit local s3 [OPTIONS] COMMAND [ARGS]...
+
+  Manage local mocked S3 fixture mappings; does not call AWS S3.
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  cp       Copy a local file or mocked S3 object in the fixture config;...
+  mount    Map a local directory recursively into mocked S3 keys; does not...
+  unmount  Remove a recursive local-directory-to-mocked-S3 mapping.
+  mv       Move a local file or mocked S3 object in the fixture config;...
+  rm       Remove mapped mocked S3 objects from the local fixture config.
+  ls       List mocked S3 buckets inferred from mappings, or mapped objects.
+```
+
+
+### `gluekit local s3 cp --help`
+
+```text
+Usage: gluekit local s3 cp [OPTIONS] SOURCE DESTINATION
+
+  Copy a local file or mocked S3 object in the fixture config; does not call
+  AWS.
+
+Arguments:
+  SOURCE       Local path or mocked S3 URI source.  [required]
+  DESTINATION  Local path or mocked S3 URI destination.  [required]
+
+Options:
+  --config-file PATH  Local fixture config file to update. Defaults to
+                      .gluekit/local.json.
+  --help              Show this message and exit.
+
+  **Examples**
+
+  - `gluekit local s3 cp tests/fixtures/input.json s3://my-bucket/input.json`
+
+  - `gluekit local s3 cp s3://my-bucket/input.json s3://my-bucket/copy.json`
+
+  - `gluekit local s3 cp s3://my-bucket/input.json tmp/input.json`
+```
+
+
+### `gluekit local s3 mount --help`
+
+```text
+Usage: gluekit local s3 mount [OPTIONS] SOURCE_DIR DESTINATION
+
+  Map a local directory recursively into mocked S3 keys; does not call AWS.
+
+Arguments:
+  SOURCE_DIR   Local directory to map recursively.  [required]
+  DESTINATION  Mocked S3 bucket or prefix base for mapped files.  [required]
+
+Options:
+  --config-file PATH  Local fixture config file to update. Defaults to
+                      .gluekit/local.json.
+  --help              Show this message and exit.
+
+  **Examples**
+
+  - `gluekit local s3 mount tests/fixtures s3://local-dev/input/`
+
+  - `gluekit local s3 mount data/mock s3://bucket-a`
+```
+
+
+### `gluekit local s3 unmount --help`
+
+```text
+Usage: gluekit local s3 unmount [OPTIONS] TARGET
+
+  Remove a recursive local-directory-to-mocked-S3 mapping.
+
+Arguments:
+  TARGET  Local source directory or mocked S3 destination to remove.  [required]
+
+Options:
+  --config-file PATH  Local fixture config file to update. Defaults to
+                      .gluekit/local.json.
+  --help              Show this message and exit.
+
+  **Examples**
+
+  - `gluekit local s3 unmount tests/fixtures`
+
+  - `gluekit local s3 unmount s3://local-dev/input/`
+```
+
+
+### `gluekit local s3 mv --help`
+
+```text
+Usage: gluekit local s3 mv [OPTIONS] SOURCE DESTINATION
+
+  Move a local file or mocked S3 object in the fixture config; does not call
+  AWS.
+
+Arguments:
+  SOURCE       Local path or mocked S3 URI source.  [required]
+  DESTINATION  Local path or mocked S3 URI destination.  [required]
+
+Options:
+  --delete-local      Delete the local source file after mapping a LocalPath to
+                      S3.
+  --config-file PATH  Local fixture config file to update. Defaults to
+                      .gluekit/local.json.
+  --help              Show this message and exit.
+
+  **Examples**
+
+  - `gluekit local s3 mv s3://my-bucket/input.json s3://my-bucket/renamed.json`
+
+  - `gluekit local s3 mv s3://my-bucket/input.json tmp/input.json`
+```
+
+
+### `gluekit local s3 rm --help`
+
+```text
+Usage: gluekit local s3 rm [OPTIONS] URI
+
+  Remove mapped mocked S3 objects from the local fixture config.
+
+Arguments:
+  URI  Mapped mocked S3 object or prefix to remove.  [required]
+
+Options:
+  -r, --recursive     Remove all mapped objects under the S3 prefix.
+  --config-file PATH  Local fixture config file to update. Defaults to
+                      .gluekit/local.json.
+  --help              Show this message and exit.
+
+  **Examples**
+
+  - `gluekit local s3 rm s3://my-bucket/input.json`
+
+  - `gluekit local s3 rm s3://my-bucket/prefix/ --recursive`
+```
+
+
+### `gluekit local ssm --help`
+
+```text
+Usage: gluekit local ssm [OPTIONS] COMMAND [ARGS]...
+
+  Manage local mocked SSM Parameter Store values; does not call AWS SSM.
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  put  Create or update a mocked SSM parameter with local-only syntax.
+  get  Show one mocked SSM parameter with local-only syntax.
+  rm   Delete one mocked SSM parameter with local-only syntax.
+  ls   List mocked SSM parameters with local-only syntax.
+```
+
+
+### `gluekit local ssm put --help`
+
+```text
+Usage: gluekit local ssm put [OPTIONS] NAME VALUE
+
+  Create or update a mocked SSM parameter with local-only syntax.
+
+Arguments:
+  NAME   Mocked SSM parameter name.  [required]
+  VALUE  Mocked SSM parameter value.  [required]
+
+Options:
+  --overwrite / --no-overwrite  Overwrite an existing parameter.  [default:
+                                overwrite]
+  --config-file PATH            Local fixture config file to update. Defaults to
+                                .gluekit/local.json.
+  --help                        Show this message and exit.
+
+  **Examples**
+
+  - `gluekit local ssm put /app/env dev`
+```
+
+
+### `gluekit build --help`
+
+```text
+Usage: gluekit build [OPTIONS]
+
+  Build local artifacts; only touches AWS when --push is used.
+
+Options:
+  --job-name TEXT                 Checked-out or explicit local Glue job to
+                                  update when using --package-whl or --push.
+  --dry-run                       Show what would be built and zipped without
+                                  running build commands.
+  --bump-version, --bump TEXT     Bump pyproject.toml [project].version before
+                                  building: major, minor, patch, or X.Y.Z.
+  --build-tool TEXT               Build frontend to use: auto, uv, or build.
+                                  [default: auto]
+  --out-dir PATH                  The output directory to which distributions
+                                  should be written.
+  --no-version-suffix             Remove the version suffix from generated zip
+                                  artifact names.
+  -v, --verbose                   Print build commands before execution.
+  --package-whl                   After building, replace the tracked dist/*.whl
+                                  in the selected Glue config.
+  --push                          Push the selected Glue config and artifacts
+                                  after building.
+  -i, --include TEXT              For --push, include only specific components
+                                  (script, notebook, additional-python-modules,
+                                  extra-files, job-config).
+  -x, --exclude TEXT              For --push, exclude specific components
+                                  (script, notebook, additional-python-modules,
+                                  extra-files, job-config).
+  --update-config / --no-update-config
+                                  For --push, update Glue job configuration
+                                  after uploading files.  [default: update-
+                                  config]
+  --config-dir PATH               Directory containing Glue job config files.
+                                  [default: glue/configs]
+  -p, --profile TEXT              AWS CLI credential profile for profile-scoped
+                                  config params and, with --push, real AWS
+                                  Glue/S3 API calls.
+  --auto-login / --no-auto-login  For --push with a real AWS profile,
+                                  automatically run 'aws sso login' when
+                                  credentials are missing or expired.  [default:
+                                  auto-login]
+  --help                          Show this message and exit.
+
+  **Examples**
+
+  - `gluekit build`
+
+  - `gluekit build --build-tool uv`
+
+  - `gluekit build --out-dir build/artifacts --no-version-suffix`
+
+  - `gluekit build --build-tool build --dry-run`
+```
+
+
+### `gluekit push --help`
+
+```text
+Usage: gluekit push [OPTIONS] [JOB_NAME] [TARGET_JOB]
+
+  Upload local artifacts to real AWS Glue/S3 and optionally update the job
+  config.
+
+Arguments:
+  [JOB_NAME]    Glue job name or pattern to push. Use "*" for all configs.
+  [TARGET_JOB]  Optional remote Glue job name to push to.
+
+Options:
+  --dry-run                       Show what would be uploaded without writing
+                                  files.
+  -i, --include TEXT              Include only specific components (script,
+                                  notebook, additional-python-modules, extra-
+                                  files, job-config).
+  -x, --exclude TEXT              Exclude specific components (script, notebook,
+                                  additional-python-modules, extra-files, job-
+                                  config).
+  --update-config / --no-update-config
+                                  Update Glue job configuration after uploading
+                                  files.  [default: update-config]
+  --build                         Build local package artifacts in the workspace
+                                  root before push.
+  --build-tool TEXT               Build frontend for --build: auto, uv, or
+                                  build.  [default: auto]
+  -v, --verbose                   Print local build command(s) before execution.
+  --config-dir PATH               Directory containing Glue job config files.
+                                  [default: glue/configs]
+  -p, --profile TEXT              AWS CLI credential profile used for real AWS
+                                  Glue/S3 API calls; not a gluekit local test
+                                  profile.
+  --auto-login / --no-auto-login  For real AWS profiles, automatically run 'aws
+                                  sso login' when credentials are missing or
+                                  expired.  [default: auto-login]
+  --help                          Show this message and exit.
+
+  **Examples**
+
+  - `gluekit push my-job --profile my-sso-profile`
+
+  - `gluekit push source-job target-job --profile my-sso-profile`
+
+  - `gluekit push "my-job-\*" --include script,job-config --exclude notebook`
+
+  - `gluekit push my-job --build --build-tool auto --dry-run`
+```
+
+
+### `gluekit set --help`
+
+```text
+Usage: gluekit set [OPTIONS] [JOB_NAME]
+
+  Set reusable local key/value parameters for a Glue job.
+
+Arguments:
+  [JOB_NAME]  Glue job name to save params for.
+
+Options:
+  --global            Save params as global defaults for all jobs.
+  -p, --profile TEXT  Save params under this gluekit profile scope, usually
+                      matching an AWS CLI profile name; does not contact AWS.
+  --help              Show this message and exit.
+
+  **Examples**
+
+  - `gluekit set my-job --script_location glue/scripts/my-job.py --extra_files
+  s3://my-bucket/shared/config.json`
+
+  - `gluekit pull my-job --profile my-sso-profile`
+
+  - `gluekit set --global --additional_python_modules "pydantic==2.11.7"`
+
+  - `gluekit push "my-job-\*" --profile my-sso-profile`
+
+  - `gluekit config set my-job --default_arguments.--TempDir s3://my-
+  bucket/tmp/`
+
+  - `gluekit config get my-job`
+
+  - `gluekit config get`
+
+  - `gluekit set my-job --extra_py_files s3://my-
+  bucket/dist/gluekit-0.0.1-py3-none-any.whl`
+
+  - `gluekit push my-job --build --build-tool auto --profile my-sso-profile`
+```
+
+
+### `gluekit config --help`
+
+```text
+Usage: gluekit config [OPTIONS] COMMAND [ARGS]...
+
+  Show and set local reusable Glue config parameters.
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  set  Set reusable local Glue config params by job or globally.
+  get  Show stored local Glue config params.
+```
+
+
+### `gluekit config set --help`
+
+```text
+Usage: gluekit config set [OPTIONS] [JOB_NAME]
+
+  Set reusable local Glue config params by job or globally.
+
+Arguments:
+  [JOB_NAME]  Glue job name to save params for.
+
+Options:
+  --global            Save params as global defaults for all jobs.
+  -p, --profile TEXT  Save params under this gluekit profile scope, usually
+                      matching an AWS CLI profile name; does not contact AWS.
+  --help              Show this message and exit.
+
+  **Examples**
+
+  - `gluekit config set my-job --default_arguments.--TempDir s3://my-
+  bucket/tmp/`
+
+  - `gluekit config set --global --additional_python_modules "pydantic==2.11.7"`
+```
+
+
+### `gluekit config get --help`
+
+```text
+Usage: gluekit config get [OPTIONS] [JOB_NAME]
+
+  Show stored local Glue config params.
+
+Arguments:
+  [JOB_NAME]  Optional Glue job name to view merged params (global + job).
+
+Options:
+  -p, --profile TEXT  Include params from this gluekit profile scope, usually
+                      matching an AWS CLI profile name; does not contact AWS.
+  --help              Show this message and exit.
+
+  **Examples**
+
+  - `gluekit config get my-job`
+
+  - `gluekit config get`
+```
+
+
+### `gluekit list --help`
+
+```text
+Usage: gluekit list [OPTIONS]
+
+  List real AWS Glue jobs in the configured AWS account.
+
+Options:
+  --help  Show this message and exit.
+
+  **Examples**
+
+  - `gluekit list --help`
+
+  - `gluekit init --help`
+```
+
+
+### `gluekit init --help`
+
+```text
+Usage: gluekit init [OPTIONS]
+
+  Initialize local repo-level Gluekit settings from AWS CLI profiles.
+
+Options:
+  --aws-config PATH               Local AWS CLI config file to scan for
+                                  credential profile names; defaults to
+                                  AWS_CONFIG_FILE or ~/.aws/config.
+  --config-dir PATH               Directory containing Glue job config files.
+                                  [default: glue/configs]
+  --auto-login / --no-auto-login  For selected real AWS profiles, automatically
+                                  run 'aws sso login' when credentials are
+                                  missing or expired.  [default: auto-login]
+  --detect-roles / --no-detect-roles
+                                  Fetch the selected real AWS Glue job in each
+                                  profile and save its Role as local gluekit
+                                  params.  [default: detect-roles]
+  --detect-buckets / --no-detect-buckets
+                                  Compare S3 URIs in real AWS Glue job configs
+                                  across profiles and save local gluekit profile
+                                  mappings.  [default: detect-buckets]
+  --match-threshold INTEGER       Minimum rapidfuzz score for non-exact S3 key
+                                  matches.  [default: 85]
+  --baseline-profile TEXT         Selected AWS CLI profile to store as the first
+                                  gluekit checkout scope and S3 comparison
+                                  baseline.
+  --profile-job TEXT              Real AWS Glue job to inspect for an AWS CLI
+                                  profile, as PROFILE=GLUE_JOB_NAME. May be
+                                  repeated.
+  --help                          Show this message and exit.
+
+  **Examples**
+
+  - `gluekit init`
+```
+<!-- END: CLI_HELP -->
+
+## Getting started with your project
+
+### 1. Create a New Repository
+
+First, create a repository on GitHub with the same name as this project, and then run the following commands:
+
+```bash
+git init -b main
+git add .
+git commit -m "init commit"
+git remote add origin git@github.com:mbkranz/gluekit.git
+git push -u origin main
+```
+
+### 2. Set Up Your Development Environment
+
+Then, install the environment and the pre-commit hooks with
+
+```bash
+make install
+```
+
+This will also generate your `uv.lock` file
+
+### 3. Run the pre-commit hooks
+
+Initially, the CI/CD pipeline might be failing due to formatting issues. To resolve those run:
+
+```bash
+uv run pre-commit run -a
+```
+
+### 4. Commit the changes
+
+Lastly, commit the changes made by the two steps above to your repository.
+
+```bash
+git add .
+git commit -m 'Fix formatting issues'
+git push origin main
+```
+
+You are now ready to start development on your project!
+The CI/CD pipeline will be triggered when you open a pull request, merge to main, or when you create a new release.
+
+To finalize the set-up for publishing to PyPI, see [here](https://fpgmaas.github.io/cookiecutter-uv/features/publishing/#set-up-for-pypi).
+For activating the automatic documentation with MkDocs, see [here](https://fpgmaas.github.io/cookiecutter-uv/features/mkdocs/#enabling-the-documentation-on-github).
+To enable the code coverage reports, see [here](https://fpgmaas.github.io/cookiecutter-uv/features/codecov/).
+
+## Releasing a new version
+
+- Create an API Token on [PyPI](https://pypi.org/).
+- Add the API Token to your projects secrets with the name `PYPI_TOKEN` by visiting [this page](https://github.com/mbkranz/gluekit/settings/secrets/actions/new).
+- Create a [new release](https://github.com/mbkranz/gluekit/releases/new) on Github.
+- Create a new tag in the form `*.*.*`.
+
+For more details, see [here](https://fpgmaas.github.io/cookiecutter-uv/features/cicd/#how-to-trigger-a-release).
+
+---
+
+Repository initiated with [fpgmaas/cookiecutter-uv](https://github.com/fpgmaas/cookiecutter-uv).