acdc_aws_etl_pipeline 0.7.1__py3-none-any.whl → 0.7.3__py3-none-any.whl

acdc_aws_etl_pipeline-0.7.3.dist-info/METADATA

@@ -0,0 +1,75 @@
+Metadata-Version: 2.4
+Name: acdc_aws_etl_pipeline
+Version: 0.7.3
+Summary: Tools for ACDC ETL pipeline
+Author: JoshuaHarris391
+Author-email: harjo391@gmail.com
+Requires-Python: >=3.9.5,<4.0.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: boto3
+Requires-Dist: dbt-athena (==1.9.4)
+Requires-Dist: dbt-core (==1.9.4)
+Requires-Dist: gen3 (>=4.27.4,<5.0.0)
+Requires-Dist: gen3_validator (>=2.0.0,<3.0.0)
+Requires-Dist: numpy (<2.0.0)
+Requires-Dist: pyjwt (>=2.10.1,<3.0.0)
+Requires-Dist: pytest
+Requires-Dist: python-dotenv
+Requires-Dist: pytz (>=2025.2,<2026.0)
+Requires-Dist: pyyaml (>=6.0.2,<7.0.0)
+Requires-Dist: s3fs (==2025.10.0)
+Requires-Dist: tenacity (>=8.2,<10.0)
+Requires-Dist: tzlocal (>=5.3.1,<6.0.0)
+Description-Content-Type: text/markdown
+
+# acdc-aws-etl-pipeline
+Infrastructure and code for the ACDC ETL pipeline and data operations in AWS
+
+## Documentation
+
+- [Dictionary deployment](docs/dictionary_deployment.md)
+- [Data ingestion](docs/data_ingestion.md)
+- [Data validation](docs/data_validation.md)
+- [Data transformation (dbt)](docs/data_transformation_dbt.md)
+- [Data releases](docs/write_data_release.md)
+- [Synthetic data generation](docs/synthetic_data_generation.md)
+- [REST API upload to sheepdog](docs/rest_api_sheepdog_upload.md)
+- [Data deletion](docs/data_deletion.md)
+- [IndexD file registration](docs/indexd_registration.md)
+- [Querying Athena](docs/querying_athena.md)
+- [Writing Athena queries to JSON](docs/write_athena_queries_to_json.md)
+- [Troubleshooting](docs/troubleshooting.md)
+
+## Library and source code (`src/acdc_aws_etl_pipeline`)
+
+The Python package in [`src/acdc_aws_etl_pipeline`](src/acdc_aws_etl_pipeline) provides reusable utilities for ingestion, validation, uploads, and Athena/Glue operations used across the pipeline and services.
+
+### Modules
+
+- **`ingest/`**: ingestion helpers for loading source datasets into S3/Glue (see [`ingest/ingest.py`](src/acdc_aws_etl_pipeline/ingest/ingest.py)).
+- **`upload/`**: Gen3/Sheepdog metadata submission and deletion utilities (e.g. [`upload/metadata_submitter.py`](src/acdc_aws_etl_pipeline/upload/metadata_submitter.py)).
+- **`validate/`**: schema validation utilities and helpers for validation workflows (see [`validate/validate.py`](src/acdc_aws_etl_pipeline/validate/validate.py)).
+- **`utils/`**: shared Athena/Glue/dbt/release helpers (e.g. [`utils/athena_utils.py`](src/acdc_aws_etl_pipeline/utils/athena_utils.py), [`utils/release_writer.py`](src/acdc_aws_etl_pipeline/utils/release_writer.py)).
+
+### Local development
+
+To install dependencies and run tests:
+
+```bash
+pip install poetry
+poetry install
+source $(poetry env info --path)/bin/activate
+poetry run pytest
+```
+
+### Install from PyPI
+
+Releases are published automatically, so you can also install the package directly:
+
+```bash
+pip install acdc_aws_etl_pipeline

{acdc_aws_etl_pipeline-0.7.1.dist-info → acdc_aws_etl_pipeline-0.7.3.dist-info}/RECORD

@@ -9,6 +9,6 @@ acdc_aws_etl_pipeline/utils/athena_utils.py,sha256=QJlBe-07Hkq-BqmcxBu6ZtAmVfZSH
 acdc_aws_etl_pipeline/utils/dbt_utils.py,sha256=5XRFOwNNIeuW2sQuor3h_OZTuXGg6xv2AUYwj9bMAAM,2054
 acdc_aws_etl_pipeline/utils/release_writer.py,sha256=vsxHJ6l-UWPpzeyEPHurX5iFgeCEQ-9FbySAbPNfTTM,7555
 acdc_aws_etl_pipeline/validate/validate.py,sha256=zLqK9i92FsRAaBOGdY-G7-vb0e6tmkoUXhY6zCfbjN8,24895
-acdc_aws_etl_pipeline-0.7.1.dist-info/METADATA,sha256=WddwCKf3KV4-JsKtsegk5dxu6dWKvXx8YANvZZKbRGs,2964
-acdc_aws_etl_pipeline-0.7.1.dist-info/WHEEL,sha256=zp0Cn7JsFoX2ATtOhtaFYIiE2rmFAD4OcMhtUki8W3U,88
-acdc_aws_etl_pipeline-0.7.1.dist-info/RECORD,,
+acdc_aws_etl_pipeline-0.7.3.dist-info/METADATA,sha256=4HU13kgbLqeFVwPPtjSjSgNO7M1FhxcKANJHR2_3c2k,3096
+acdc_aws_etl_pipeline-0.7.3.dist-info/WHEEL,sha256=zp0Cn7JsFoX2ATtOhtaFYIiE2rmFAD4OcMhtUki8W3U,88
+acdc_aws_etl_pipeline-0.7.3.dist-info/RECORD,,

acdc_aws_etl_pipeline-0.7.1.dist-info/METADATA

@@ -1,82 +0,0 @@
-Metadata-Version: 2.4
-Name: acdc_aws_etl_pipeline
-Version: 0.7.1
-Summary: Tools for ACDC ETL pipeline
-Author: JoshuaHarris391
-Author-email: harjo391@gmail.com
-Requires-Python: >=3.9.5,<4.0.0
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Programming Language :: Python :: 3.14
-Requires-Dist: boto3
-Requires-Dist: dbt-athena (==1.9.4)
-Requires-Dist: dbt-core (==1.9.4)
-Requires-Dist: gen3 (>=4.27.4,<5.0.0)
-Requires-Dist: gen3_validator (>=2.0.0,<3.0.0)
-Requires-Dist: numpy (<2.0.0)
-Requires-Dist: pyjwt (>=2.10.1,<3.0.0)
-Requires-Dist: pytest
-Requires-Dist: python-dotenv
-Requires-Dist: pytz (>=2025.2,<2026.0)
-Requires-Dist: pyyaml (>=6.0.2,<7.0.0)
-Requires-Dist: s3fs (==2025.10.0)
-Requires-Dist: tenacity (>=8.2,<10.0)
-Requires-Dist: tzlocal (>=5.3.1,<6.0.0)
-Description-Content-Type: text/markdown
-
-# acdc-aws-etl-pipeline
-Infrastructure and code for the ACDC ETL pipeline and data operations in AWS
-
-## Ingestion
-- [ingestion](docs/ingestion.md)
-- [upload_synthdata_s3](docs/upload_synthdata_s3.md)
-
-## DBT
-
-
-
-## Release Management
-- [Writing DBT Releases](docs/write_dbt_release_info.md)
-
-
-## Deploying the dictionary
-e.g. to testing
-
-```bash
-# Example 
-bash services/dictionary/pull_dict.sh <raw_dictionary_url>
-bash services/dictionary/upload_dictionary.py <local_dictionary_path> <s3_target_uri>
-
-# Deploying to test
-VERSION=v0.6.3
-bash services/dictionary/pull_dict.sh "https://raw.githubusercontent.com/AustralianBioCommons/acdc-schema-json/refs/tags/${VERSION}/dictionary/prod_dict/acdc_schema.json"
-python3 services/dictionary/upload_dictionary.py "services/dictionary/schemas/acdc_schema_${VERSION}.json" s3://gen3schema-cad-uat-biocommons.org.au/cad.json
-
-
-# Deploying to staging
-VERSION=v1.0.0
-bash services/dictionary/pull_dict.sh "https://raw.githubusercontent.com/AustralianBioCommons/acdc-schema-json/refs/tags/${VERSION}/dictionary/prod_dict/acdc_schema.json"
-python3 services/dictionary/upload_dictionary.py "services/dictionary/schemas/acdc_schema_${VERSION}.json" s3://gen3schema-cad-staging-biocommons.org.au/cad.json
-```
-
-## Generating synthetic metadata
-- Run this script to generate synthetic metadata for the studies in the dictionary
-
-```bash
-# this will generate 30 samples for AusDiab_Simulated and 60 samples for Baker-Biobank_Simulated
-bash services/synthetic_data/generate_synth_metadata.sh --studies "AusDiab_Simulated,Baker-Biobank_Simulated" --permute-max-samples "30,60"
-```
-
-## uploading synthetic metadata to sheepdog
-- Run this script to upload synthetic metadata to sheepdog
-
-```python
-# to see argumments
-python3 services/synthetic_data/upload_synth_metadata_sheepdog.py -h
-
-# to upload metadata for version v0.6.3
-python3 services/synthetic_data/upload_synth_metadata_sheepdog.py --version v0.6.3
-```