gen3-dataops-toolkit 2.0.0__py3-none-any.whl

gen3_dataops_toolkit-2.0.0.dist-info/METADATA

@@ -0,0 +1,125 @@
+Metadata-Version: 2.4
+Name: gen3-dataops-toolkit
+Version: 2.0.0
+Summary: Gen3 DataOps toolkit (g3dt): operate SSM-published Gen3 data pipeline environments
+License: Apache-2.0
+Author: JoshuaHarris391
+Author-email: harjo391@gmail.com
+Requires-Python: >=3.9.5,<4.0.0
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: awswrangler (>=3.14.0,<4.0.0)
+Requires-Dist: boto3
+Requires-Dist: gen3 (>=4.27.4,<5.0.0)
+Requires-Dist: gen3-metadata (>=1.4.0,<2.0.0)
+Requires-Dist: gen3_validator (>=2.0.0,<3.0.0)
+Requires-Dist: numpy (<2.0.0)
+Requires-Dist: pyarrow (>=14.0.0,<19.0.0)
+Requires-Dist: pyjwt (>=2.10.1,<3.0.0)
+Requires-Dist: python-dotenv
+Requires-Dist: pytz (>=2025.2,<2026.0)
+Requires-Dist: pyyaml (>=6.0.2,<7.0.0)
+Requires-Dist: s3fs (==2025.10.0)
+Requires-Dist: tenacity (>=8.2,<10.0)
+Requires-Dist: typer (>=0.12)
+Requires-Dist: tzlocal (>=5.3.1,<6.0.0)
+Project-URL: Repository, https://github.com/AustralianBioCommons/gen3-dataops-toolkit
+Description-Content-Type: text/markdown
+
+# gen3-dataops-toolkit (`g3dt`)
+
+Operate Gen3 AWS data-pipeline environments from one pip-installable CLI.
+
+`g3dt` is the tooling half of the Gen3 DataOps platform: the
+[gen3-aws-data-pipeline](https://github.com/AustralianBioCommons/gen3-aws-data-pipeline)
+CDK app deploys a complete pipeline per project/environment and publishes every
+resource name to AWS SSM Parameter Store; `g3dt` resolves those names at
+runtime and gives operators one command surface for dictionary deploys,
+metadata upload/delete, indexd registration, EC2 job dispatch, and Kubernetes
+restarts. The dbt half of the platform lives in
+[gen3-dbt-template](https://github.com/AustralianBioCommons/gen3-dbt-template).
+
+**No AWS resource name is compiled into this package.** The same wheel
+operates any project: it is targeted purely by `--env`, the project's SSM tree
+(`/{project}/{env}/...`), and a tiny local bootstrap marker.
+
+## Install
+
+```bash
+pip install gen3-dataops-toolkit
+```
+
+## Bootstrap (the only local configuration)
+
+`g3dt` needs to know just the project and region — everything else comes from
+SSM. Create `~/.g3dt/g3dt.yaml`:
+
+```yaml
+project: etl                # your projectId
+region: ap-southeast-2
+default_env: test
+profiles:                   # optional: AWS named profile per env
+  test: etl_test            # (omit entirely on EC2/CodeBuild — ambient
+  staging: etl_staging      #  role credentials are used)
+studies:                    # optional: the project's study registry;
+  mystudy_test:             # alternatively upload it once per env to
+    project_id: MyStudy     # s3://<metadata-bucket>/config/studies.yaml
+    program_id: program1
+    s3_metadata_path: s3://my-bucket/metadata/mystudy/
+```
+
+Search order: `./g3dt.yaml` → `~/.g3dt/g3dt.yaml` → `/etc/g3dt/g3dt.yaml`
+(the EC2 job box's copy, written by CDK user-data). Env vars override:
+`G3DT_PROJECT`, `AWS_REGION`, `G3DT_DEFAULT_ENV`.
+
+## Quick start
+
+```bash
+g3dt config envs                 # environments with a deployed SSM tree
+g3dt config show --env test      # every resolved name — the safety check
+g3dt ec2 up --env test           # start the env's job box (SSM-managed)
+g3dt metadata upload --study mystudy --env test --on ec2
+g3dt jobs logs <run-id> --follow # live logs; laptop can sleep, job keeps going
+g3dt ec2 down --env test         # or let the auto-stop alarm handle it
+g3dt docs                        # the full operations overview
+```
+
+## How configuration works
+
+There are exactly two kinds of configuration:
+
+- **INPUTS** — human-authored values, committed as
+  `config/<projectId>.<env>.json` in the CDK repo and read only by
+  `cdk deploy`. To change a deployed setting (e.g. the dictionary version),
+  edit that file and redeploy — the value flows to SSM.
+- **OUTPUTS** — every resource name the CDK creates plus the mirrored Gen3
+  app facts, published to SSM under `/{project}/{env}/...` on deploy. `g3dt`
+  reads these live (cached one round-trip per invocation) and never stores
+  them locally.
+
+Because the CLI and the infrastructure read the same parameters, they cannot
+disagree — and because each environment has its own tree (including its own
+`ec2/instanceId`), running a job against the wrong environment's resources is
+structurally impossible.
+
+## Development
+
+```bash
+poetry install
+poetry run python3 -m pytest
+```
+
+## Provenance
+
+This toolkit was ported (working tree only) from
+[AustralianBioCommons/acdc-aws-etl-pipeline](https://github.com/AustralianBioCommons/acdc-aws-etl-pipeline),
+the ACDC ETL monolith, as part of the Gen3 DataOps platform refactor (2026).
+It starts at version **2.0.0**; versions ≤ 1.2.0 on PyPI are the legacy
+`acdc_aws_etl_pipeline` package, which continues to operate the legacy ACDC
+pipeline unchanged.
+

gen3_dataops_toolkit-2.0.0.dist-info/RECORD

@@ -0,0 +1,53 @@
+g3dt/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+g3dt/cli/__init__.py,sha256=9v7e7h3EBw-Gmk7kkFnaJ5A4rMO7BC4W1Fe_XpKDBVU,190
+g3dt/cli/_internal/__init__.py,sha256=l-i4zejAuT7zliVczKBiD4wYTY6-jYdGs-ruxKfrhxA,90
+g3dt/cli/_internal/dispatch.py,sha256=HOHMGyJ5DmtCqeU46KlNYcs71HbmKv3ccaAFJ6cWHjo,15624
+g3dt/cli/_internal/registry.py,sha256=ty0uaKYKp95I9JSh8dmG2XgxdxjVioeoqbcr1gyRRz4,1686
+g3dt/cli/_internal/resolve.py,sha256=oCRj6iw3mOLzEhtmGsc5Mxdkz24TQSNCHpQjRoV38KM,614
+g3dt/cli/_internal/runner.py,sha256=YgSpwRHSXzhv1i8OLZnVDTktIHzTQOFLxRpY7jinazQ,2680
+g3dt/cli/_internal/safety.py,sha256=ibIFrB0dQYLF9JQffxmtC32oD4Wy1LE1Rmk5Nrt0VxU,3872
+g3dt/cli/config_cmds.py,sha256=bPwVTtPmuUX8iktQyFzTGJoO9xF0_owLnwEFMQ0Qduw,7366
+g3dt/cli/delete_cmds.py,sha256=QzqKj9yoEPPn8UnbaXLJMplpgRz7tILZ1tyh7zDjfIc,3686
+g3dt/cli/dict_cmds.py,sha256=mqZMmriKWqcquTChM28lxAnvFejPc0ViTZ5q4yu9QRQ,3294
+g3dt/cli/ec2_cmds.py,sha256=__ZLvSHYw9a3EF23yvGIPKD6y7Xbdx1zGpvVbsR-YkE,4256
+g3dt/cli/indexd_cmds.py,sha256=GoFs-JBQnzc6eAKuDwGNMZ_kvA3ll1eKqnpX0IIFp7M,1854
+g3dt/cli/jobs.py,sha256=DueLOdSEr__qG44mzNifly9Vnb8Vwt6Jsrtyu3z7ys0,2793
+g3dt/cli/k8s.py,sha256=1h-IqyFwhsHrMG-bjgytbz26DxnvIIMxsEfcVIeyTEA,1972
+g3dt/cli/main.py,sha256=GB91S0MmBeH3_uMV4nMiXuVGfSnrOgrHUa3FjxTAtCc,3641
+g3dt/cli/metadata.py,sha256=I-jwGS6jWsY-EtpTqXFTtqOdwzEQ3AlGsR6SmyCmC-k,2630
+g3dt/cli/synth.py,sha256=5VKRb02vYHOtHSzz5anmD7MSwKs6HLLO4_ooxurRSJQ,7231
+g3dt/config.py,sha256=KYFgnu6LYewqPDEJ_5GPt58vOwjC8upc7GcKRq71Dmk,15014
+g3dt/indexd/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+g3dt/indexd/indexd_registrar.py,sha256=EknHXv81MshZJm101UrBtQfQjgZntnOR9O6Ed_eYYng,7231
+g3dt/ingest/ingest.py,sha256=NLiJ_yij28D90UvcuCnZZmQSQQJqKCuoyQdlueNu_pY,23856
+g3dt/resolver.py,sha256=A2xgHVQrVc5YBvwJr90ITCYLX-D3kchKHz4xzJoUpHU,5965
+g3dt/services/delete/delete_all_metadata_for_project.py,sha256=tSlkFu1u35C4a_ORdYgamxOVwazTG88TUlMbCpAtkYc,4563
+g3dt/services/delete/delete_metadata.sh,sha256=5haS4K8o4zywDJCGKoWLXlFg4jeVn49zecCvJaOnaEk,4714
+g3dt/services/delete/delete_metadata_by_guid.py,sha256=0_iwhJ4KIG0ESNMcELSfrL3rXuMYW2HA5AOePUZJqAU,9967
+g3dt/services/dictionary/deploy_dd.sh,sha256=wWcne7l4qa3NAOPAC1T22OV9c4ObwVYOKZRmOQZCdJ0,2895
+g3dt/services/dictionary/pull_dict.sh,sha256=gUX_ilfoaoSS6U6ZRQNnkjFLK8AwZ3AWEUYgRoD8M6Y,1643
+g3dt/services/dictionary/upload_dictionary.py,sha256=jBFAot26xfNyqHXLtFvon9vrtKDfDriXby7wXf0Iy_g,3249
+g3dt/services/indexd/register_indexd.py,sha256=YwBLpsrL6fV1SyjHorR04YMV4HjDnNWR3sgkLm29Q3o,6698
+g3dt/services/k8s_ops/argocd_restart_etl.sh,sha256=0vRHLQw1Fpvg1myiS83bOH6hUkPFjIURCj6laSDN5Jg,5071
+g3dt/services/k8s_ops/argocd_restart_ms.sh,sha256=1HrSUPO5cUaocMpl7OgZeyBRac0I_ceAT1vbkdpXbvU,2828
+g3dt/services/k8s_ops/argocd_restart_schema.sh,sha256=5M5eEip9yOFkLL6FMlzCmyrlNMoFoMBdT2S0st8tNqE,3002
+g3dt/services/k8s_ops/login_to_pod.sh,sha256=w1qA-8MhtXM9-MnUTvqX1nMRyjBF64-8uBtMBldA7mY,3397
+g3dt/services/k8s_ops/restart_etl_and_ms.sh,sha256=VbUzEYEd1fv4VkwLcvRQS-byHwW5x_Y6OcT2SNHsqUQ,2069
+g3dt/services/synthetic_data/delete_synth_metadata_sheepdog.py,sha256=frm35EYdfXyREOUeZKO_j9pbrjKuZLRN0bPO2BeeEj0,5053
+g3dt/services/synthetic_data/full_deploy_dd_and_synth.sh,sha256=RdHLTR9wmr3PPogHV6_pD7cC7-tEOUMFn8GThB3E82I,5087
+g3dt/services/synthetic_data/generate_synth_metadata.sh,sha256=cD3TIEDBzFt9-tI7PWrkw9YI0tWMik0M86xrxtvnAKw,4996
+g3dt/services/synthetic_data/upload_synth_metadata_sheepdog.py,sha256=0JrCsBzscjESskSUKOxekWgjyHEtQc7ui0ZidLEHibM,4981
+g3dt/services/upload/metadata/upload_all_studies.sh,sha256=8ANiiRU9vKbfv174-6XCAu1wXfVdxJkfK3WOlYd2osU,2951
+g3dt/services/upload/metadata/upload_metadata.py,sha256=fTXbGpQkLRIGRIbGno5LSLUcivp8ltfHwclS3v5Yz9Q,4950
+g3dt/upload/__init__.py,sha256=hXU9IZQGMTQk7d6vo4NvNxSqH_ZDq7dYJ1oNJrv26Pc,34
+g3dt/upload/metadata_deleter.py,sha256=jxAyY1r739CEnzOuxrJv-a0AGJcVl6I5cYajnTXTlWU,8405
+g3dt/upload/metadata_submitter.py,sha256=8_QO-n_TtMm0ujoYzbCOYqLevrlXOrPABQeB7wuB3II,39920
+g3dt/upload/upload_synthdata_s3.py,sha256=Ge5TQzZkWnJNp-q0Ak-Yhv1h1eWLxg-PlWVHrd1m0B8,5155
+g3dt/utils/athena_utils.py,sha256=oxIP08-tjMdCK6egCVO2GmL8iA_1z1i9FY-fXS2j-X8,33389
+g3dt/utils/dbt_utils.py,sha256=5XRFOwNNIeuW2sQuor3h_OZTuXGg6xv2AUYwj9bMAAM,2054
+g3dt/utils/release_writer.py,sha256=I8UPNKLMdK2wEjL9KsRJGkbI_YSqpx0j6sydwLrjbas,7771
+g3dt/validate/validate.py,sha256=4maAZh14uTwBTvhdMSOiYea9Ql6YPttL69FwBlUsINU,22912
+gen3_dataops_toolkit-2.0.0.dist-info/METADATA,sha256=9saenX-9cQ-rD84pP9-7N2okNtzylIqaR7mrVvUyh8U,5076
+gen3_dataops_toolkit-2.0.0.dist-info/WHEEL,sha256=EGEvSphFYqXKs23-kQBeyNoJP1nrT8ZJKQoi5p5DYL8,88
+gen3_dataops_toolkit-2.0.0.dist-info/entry_points.txt,sha256=h105uANZs0L-lhqNPipmquf6BkzqKSSAlWaABhyljbo,43
+gen3_dataops_toolkit-2.0.0.dist-info/RECORD,,

gen3_dataops_toolkit-2.0.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: poetry-core 2.4.0
+Root-Is-Purelib: true
+Tag: py3-none-any

gen3_dataops_toolkit-2.0.0.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+g3dt=g3dt.cli.main:main
+