grz-cli 1.1.1__tar.gz → 1.2.0__tar.gz

{grz_cli-1.1.1 → grz_cli-1.2.0}/CHANGELOG.md

@@ -1,5 +1,30 @@
 # Changelog
 
+## [1.2.0](https://github.com/BfArM-MVH/grz-tools/compare/grz-cli-v1.1.2...grz-cli-v1.2.0) (2025-08-19)
+
+
+### Features
+
+* **grz-cli,grzctl:** bump grz dependencies to latest ([#359](https://github.com/BfArM-MVH/grz-tools/issues/359)) ([b31c88b](https://github.com/BfArM-MVH/grz-tools/commit/b31c88bfa32cf257e6db9b2bb80d302493dbd469))
+* **grz-cli:** add command to compute submission ID from metadata ([#348](https://github.com/BfArM-MVH/grz-tools/issues/348)) ([674d0f1](https://github.com/BfArM-MVH/grz-tools/commit/674d0f1acb3b048cdbde2eb10056faf74033c309))
+* **grz-cli:** also report grz library versions ([#355](https://github.com/BfArM-MVH/grz-tools/issues/355)) ([58d047e](https://github.com/BfArM-MVH/grz-tools/commit/58d047ec3492df172067312203da49c8af83a1a8))
+* **grz-cli:** also write submission ID to logs on upload ([#347](https://github.com/BfArM-MVH/grz-tools/issues/347)) ([1de43aa](https://github.com/BfArM-MVH/grz-tools/commit/1de43aa5807f22bff47683c2da356f607180ef51))
+* **grz-cli:** write command line to logs ([#354](https://github.com/BfArM-MVH/grz-tools/issues/354)) ([bc28b23](https://github.com/BfArM-MVH/grz-tools/commit/bc28b23ae617d7c8837e872c24858b50d35bf25a))
+* **grzctl:** also report grz library versions ([58d047e](https://github.com/BfArM-MVH/grz-tools/commit/58d047ec3492df172067312203da49c8af83a1a8))
+
+
+### Documentation
+
+* **grz-cli:** add docs for submit command to README ([#351](https://github.com/BfArM-MVH/grz-tools/issues/351)) ([8d4b138](https://github.com/BfArM-MVH/grz-tools/commit/8d4b13850da435dfee60060caf4a47ddbe59a4d4))
+* **grz-cli:** clean and update README ([#352](https://github.com/BfArM-MVH/grz-tools/issues/352)) ([5c037b7](https://github.com/BfArM-MVH/grz-tools/commit/5c037b749b5cd97b44a53c61618e35787ab11f2a))
+
+## [1.1.2](https://github.com/BfArM-MVH/grz-tools/compare/grz-cli-v1.1.1...grz-cli-v1.1.2) (2025-08-05)
+
+
+### Bug Fixes
+
+* **grz-cli,grz-common,grzctl:** fix logging and migrate to grz-common ([#319](https://github.com/BfArM-MVH/grz-tools/issues/319)) ([51ada07](https://github.com/BfArM-MVH/grz-tools/commit/51ada073a2af93ba1a1c48f069b4546ce9bd2975))
+
 ## [1.1.1](https://github.com/BfArM-MVH/grz-tools/compare/grz-cli-v1.1.0...grz-cli-v1.1.1) (2025-07-23)
 
 

{grz_cli-1.1.1 → grz_cli-1.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: grz-cli
-Version: 1.1.1
+Version: 1.2.0
 Summary: Tool for validation, encryption and upload of MV submissions to GDCs.
 Project-URL: Homepage, https://github.com/BfArM-MVH/grz-tools
 Project-URL: Repository, https://github.com/BfArM-MVH/grz-tools
@@ -16,8 +16,8 @@ Requires-Python: >=3.12
 Requires-Dist: boto3<2,>=1.36
 Requires-Dist: click<9,>=8.2
 Requires-Dist: crypt4gh<2,>=1.7
-Requires-Dist: grz-common<2,>=1.1.1
-Requires-Dist: grz-pydantic-models<3,>=2.1.2
+Requires-Dist: grz-common<2,>=1.3.0
+Requires-Dist: grz-pydantic-models<3,>=2.2.1
 Requires-Dist: jsonschema<5,>=4.23.0
 Requires-Dist: platformdirs<5,>=4.3.6
 Requires-Dist: pydantic-settings<2.10,>=2.9.0
@@ -29,93 +29,61 @@ Requires-Dist: rich==13.*
 Requires-Dist: tqdm<5,>=4.66.5
 Description-Content-Type: text/markdown
 
-# GRZ CLI
+# `grz-cli`
+
+A command-line tool for validating, encrypting, and uploading submissions to a genomDE Model Project GDC (Genome Data Center).
 
-A command-line tool for validating, encrypting, uploading and downloading submissions to/from a GDC/GRZ (Genomrechenzentrum).
 
 ## Table of Contents
 
-- [Introduction](#introduction)
-- [Features](#features)
 - [Installation](#installation)
   - [Requirements](#requirements)
-  - [End-user setup](#end-user-setup)
-    - [Installation via `conda` (recommended)](#installation-via-conda-recommended)
-    - [Installation via `pip` (not recommended)](#installation-via-pip-not-recommended)
+  - [Using Conda (recommended)](#using-conda-recommended)
+  - [Using Docker](#using-docker)
 - [Usage](#usage)
   - [Configuration](#configuration)
-  - [Exemplary submission procedure](#exemplary-submission-procedure)
+  - [Submission Layout](#submission-layout)
+  - [Example submission procedure](#example-submission-procedure)
+  - [Troubleshooting](#troubleshooting)
 - [Command-Line Interface](#command-line-interface)
-- [Testing](#testing)
 - [Contributing](#contributing)
 - [License](#license)
 - [Acknowledgements](#acknowledgements)
 
-## Introduction
-
-This tool provides a way to validate files, encrypt/decrypt files using the [crypt4gh](https://crypt4gh.readthedocs.io/en/latest/) library and upload/download the encrypted files to an S3 bucket of a GDC/GRZ. It also logs the progress and outcomes of these operations in a metadata file.
-
-It is recommended to have the following folder structure for a single submission:
-
-```
-EXAMPLE_SUBMISSION
-├── files
-│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.read1.fastq.gz
-│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.read2.fastq.gz
-│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.vcf
-│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.read1.fastq.gz
-│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.read2.fastq.gz
-│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.vcf
-│   ├── target_regions.bed
-└── metadata
-    └── metadata.json
-```
 
-The current version of the tool requires the `working_dir` to have at least as much free disk space as the total size of the data being submitted.
-## Features
+## Installation
 
-- **Validation**: Validate file checksums, basic file metadata and BfArM requirements.
-- **Encryption**: Encrypt files using `crypt4gh`.
-- **Decryption**: Encrypt files using `crypt4gh`.
-- **Upload**: Upload encrypted files directly to a GRZ either (via built-in `boto3`).
-- **Download**: Download encrypted files from a GRZ (via built-in `boto3`).
-- **Logging**: Log progress and results of operations
+### Requirements
 
+The current version of the tool requires the working directory to have at least as much free disk space as the total size of the data being submitted.
 
-## Installation
+Beside of the disk space requirements for the submission data, this tool also requires a Linux environment.
+For example:
 
-### Requirements
-Beside of the disk space requirements for the submission data, this tool also requires a linux environment, e.g.:
 - Linux server
-- Virtual machine running linux
+- Virtual machine running Linux
 - Docker container
-- Windows subsystem for linux
-- ...
-
+- Windows Subsystem for Linux
 
-### End-user setup
-The recommended method to install this tool is using the conda package manager.
 
+### Using [Conda](https://conda.io) (recommended)
 
-#### Installation via `conda` (recommended)
+If Conda is not yet available on your system, we recommend to install it through the [Miniforge Conda installer](https://github.com/conda-forge/miniforge) by running the following commands:
 
-If `conda` is not yet available on your system, we recommend to install the [Miniforge conda distribution](https://github.com/conda-forge/miniforge) by running the following commands:
 ```bash
 curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
 bash Miniforge3-$(uname)-$(uname -m).sh
 ```
-There are also alternative ways to install conda:
-- [Micromamba, a single executable that does not require a base environment](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html)
-- [Official installation instructions](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html)
 
 Next, install the `grz-cli` tool:
+
 ```bash
-# create conda environment and activate it
-conda create -n grz-tools -c conda-forge -c bioconda "grz-cli"
+conda create -n grz-tools -c conda-forge -c bioconda grz-cli
 conda activate grz-tools
+grz-cli --help
 ```
 
-##### Update instructions
+#### Updating
 
 Use the following command to update the tool:
 
@@ -124,59 +92,97 @@ conda update -n grz-tools -c conda-forge -c bioconda grz-cli
 ```
 
 
-#### Installation via `pip` (not recommended)
-While installation via `pip` is possible, it is not recommended because users must ensure
-that the correct Python version is already installed and that they are using a virtual python environment.
+### Using pip (not recommended)
+
+While installation via `pip` is possible, it is not recommended because users should create/manage a Python virtual environment and must ensure that the correct Python version is being used.
 
 ```bash
 pip install grz-cli
 ```
-##### Update instructions:
+
+#### Updating
+
 Use the following command to update the tool:
+
 ```bash
 pip upgrade grz-cli
 ```
 
 
-#### Docker
+### Using Docker
+
 Docker images are available via biocontainers at [https://biocontainers.pro/tools/grz-cli](https://biocontainers.pro/tools/grz-cli).
 
-The build process can take at least a few days after the Bioconda release, so double-check that the latest version in Bioconda is also the latest Docker image version.
+The build process can take at least a few days after the Bioconda release, so double-check that the [latest version in Bioconda](https://anaconda.org/bioconda/grz-cli) is also the latest Docker image version.
 
 ## Usage
 
 ### Configuration
-**The configuration file will be provided by your associated GRZ, please place it into `~/.config/grz-cli/config.yaml`.**
 
-The tool requires a configuration file in YAML format to specify the S3 bucket and other options.
-For an exemplary configuration, see [resources/config.yaml](resources/config.yaml).
+**The configuration file will be provided by your associated GRZ.**
+**Do not create this file as an LE.**
+
+The tool requires a configuration file in YAML format to specify the S3 API parameters and other validation options.
+
+This file may be placed at `~/.config/grz-cli/config.yaml` or provided each time to `grz-cli` using the `--config-file` option on the command line.
+
+The S3 secrets can either be directly within the config file or as defined with the usual AWS environment variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
+
+### Submission Layout
+
+It is recommended to have the following folder structure for a single submission:
+
+```
+EXAMPLE_SUBMISSION
+├── files
+│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.read1.fastq.gz
+│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.read2.fastq.gz
+│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.vcf
+│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.read1.fastq.gz
+│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.read2.fastq.gz
+│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.vcf
+│   └─── target_regions.bed
+└── metadata
+    └── metadata.json
+```
+
+The only requirements are that `metadata/metadata.json` exists and the `files/` directory contains all of the other files.
+Data files may be nested under subfolders inside `files/` for better organization.
+For example, each donor could have their own folder for files.
 
-S3 access and secret key can be listed either in the config file or as environment variable (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`).
 
-### Exemplary submission procedure
-After preparing your submission as outlined above, you can use the following commands to validate, encrypt and upload the submission: 
-```sh
-# Validate the submission
-grz-cli validate --submission-dir EXAMPLE_SUBMISSION
+### Example submission procedure
 
-# Encrypt the submission
-grz-cli encrypt --submission-dir EXAMPLE_SUBMISSION
+After preparing your submission as outlined above, you can use the following command to validate, encrypt and upload the submission:
 
-# Upload the submission
-grz-cli upload --submission-dir EXAMPLE_SUBMISSION
+```bash
+grz-cli submit --submission-dir EXAMPLE_SUBMISSION
 ```
 
 ### Troubleshooting
-**In case of issues, please re-run your commands with `grz-cli --log-level DEBUG --log-file <your-log-file.log> [...]` and submit the log file to the GRZ data steward!**
+
+In case of issues, please re-run your commands with `grz-cli --log-level DEBUG --log-file path/to/write/file.log [...]` and submit the log file to the GDC data steward.
 
 ## Command-Line Interface
 
 `grz-cli` provides a command-line interface with the following subcommands:
 
+### submit
+
+The `submit` command is the recommended command for submitting data to genome data centers.
+It combines the `validate`, `encrypt`, and `upload` commands (see below).
+
+- `-s, --submission-dir`: Path to the submission directory containing both 'metadata/' and 'files/' directories [**Required**]
+- `-c, --config-file`: Path to config file [_optional_]
+
+```bash
+grz-cli submit --submission-dir foo
+```
+
+
 ### validate
 
 It is recommended to run this command before continuing with encryption and upload.
-Progress files are stored relative to the submission directory.
 
 - `--submission-dir`: Path to the submission directory containing both 'metadata/' and 'files/' directories [**Required**]
 
@@ -188,7 +194,7 @@ grz_cli validate --submission-dir foo
 
 ### encrypt
 
-If a working directory is not provided, then the current directory is used automatically. The log-files are going to be stored in the sub-folder of the working directory.
+If a working directory is not provided, then the current directory is used automatically.
 Files are stored in a folder named `encrypted_files` as a sub-folder of the working directory.
 
 - `-s, --submission-dir`: Path to the submission directory containing both 'metadata/' and 'files/' directories [**Required**]
@@ -202,7 +208,7 @@ grz-cli encrypt --submission-dir foo
 
 Upload the submission into a S3 structure of a GRZ.
 
-- `-s, --submission-dir`: Path to the submission directory containing both 'metadata/' and 'encrypted_files/' directories [**Required**]
+- `-s, --submission-dir`: Path to the submission directory containing both `metadata/` and `encrypted_files/` directories [**Required**]
 - `-c, --config-file`: Path to config file [_optional_]
 
 Example usage:
@@ -211,16 +217,39 @@ Example usage:
 grz-cli upload --submission-dir foo
 ```
 
-## Testing
+### get-id
 
-Please note that binary files used for testing are managed with [Git LFS](https://git-lfs.com), which will be needed to clone them locally with the git repository.
+*Available in grz-cli v1.2.0 or higher.*
 
-To run the tests, navigate to the root directory of your project and invoke `pytest`.
-Alternatively, install `uv` and `tox` and run `uv run tox`.
+Compute and print the submission ID from a submission's JSON metadata.
+
+This is useful in case you forget to store the ID printed during upload.
+
+Example usage:
+
+```bash
+grz-cli get-id path/to/metadata.json
+```
 
 ## Contributing
 
-<!-- Add details about how others can contribute to the project -->
+### Running unreleased/development versions
+
+First, install `uv`.
+We recommend using Conda or [Pixi](https://pixi.sh/latest).
+
+After cloning the desired branch of the `grz-tools` repo locally, you can run `grz-cli` directly from the repo using:
+
+```
+uv run --project path/to/cloned/grz-tools grz-cli --help
+```
+
+### Testing
+
+Please note that binary files used for testing are managed with [Git LFS](https://git-lfs.com), which will be needed to clone them locally with the git repository.
+
+To run the tests, navigate to the root directory of your project and invoke `pytest`.
+Alternatively, install `uv` and `tox` and run `uv run tox`.
 
 ## License
 
@@ -228,5 +257,4 @@ This project is licensed under the MIT License — see the [LICENSE](LICENSE) fi
 
 ## Acknowledgements
 
-Parts of `crypt4gh` code is used in modified form
-
+Parts of [Crypt4GH](https://github.com/EGA-archive/crypt4gh) are used in modified form.

grz_cli-1.2.0/README.md

@@ -0,0 +1,229 @@
+# `grz-cli`
+
+A command-line tool for validating, encrypting, and uploading submissions to a genomDE Model Project GDC (Genome Data Center).
+
+
+## Table of Contents
+
+- [Installation](#installation)
+  - [Requirements](#requirements)
+  - [Using Conda (recommended)](#using-conda-recommended)
+  - [Using Docker](#using-docker)
+- [Usage](#usage)
+  - [Configuration](#configuration)
+  - [Submission Layout](#submission-layout)
+  - [Example submission procedure](#example-submission-procedure)
+  - [Troubleshooting](#troubleshooting)
+- [Command-Line Interface](#command-line-interface)
+- [Contributing](#contributing)
+- [License](#license)
+- [Acknowledgements](#acknowledgements)
+
+
+## Installation
+
+### Requirements
+
+The current version of the tool requires the working directory to have at least as much free disk space as the total size of the data being submitted.
+
+Beside of the disk space requirements for the submission data, this tool also requires a Linux environment.
+For example:
+
+- Linux server
+- Virtual machine running Linux
+- Docker container
+- Windows Subsystem for Linux
+
+
+### Using [Conda](https://conda.io) (recommended)
+
+If Conda is not yet available on your system, we recommend to install it through the [Miniforge Conda installer](https://github.com/conda-forge/miniforge) by running the following commands:
+
+```bash
+curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
+bash Miniforge3-$(uname)-$(uname -m).sh
+```
+
+Next, install the `grz-cli` tool:
+
+```bash
+conda create -n grz-tools -c conda-forge -c bioconda grz-cli
+conda activate grz-tools
+grz-cli --help
+```
+
+#### Updating
+
+Use the following command to update the tool:
+
+```bash
+conda update -n grz-tools -c conda-forge -c bioconda grz-cli
+```
+
+
+### Using pip (not recommended)
+
+While installation via `pip` is possible, it is not recommended because users should create/manage a Python virtual environment and must ensure that the correct Python version is being used.
+
+```bash
+pip install grz-cli
+```
+
+#### Updating
+
+Use the following command to update the tool:
+
+```bash
+pip upgrade grz-cli
+```
+
+
+### Using Docker
+
+Docker images are available via biocontainers at [https://biocontainers.pro/tools/grz-cli](https://biocontainers.pro/tools/grz-cli).
+
+The build process can take at least a few days after the Bioconda release, so double-check that the [latest version in Bioconda](https://anaconda.org/bioconda/grz-cli) is also the latest Docker image version.
+
+## Usage
+
+### Configuration
+
+**The configuration file will be provided by your associated GRZ.**
+**Do not create this file as an LE.**
+
+The tool requires a configuration file in YAML format to specify the S3 API parameters and other validation options.
+
+This file may be placed at `~/.config/grz-cli/config.yaml` or provided each time to `grz-cli` using the `--config-file` option on the command line.
+
+The S3 secrets can either be directly within the config file or as defined with the usual AWS environment variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
+
+### Submission Layout
+
+It is recommended to have the following folder structure for a single submission:
+
+```
+EXAMPLE_SUBMISSION
+├── files
+│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.read1.fastq.gz
+│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.read2.fastq.gz
+│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.vcf
+│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.read1.fastq.gz
+│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.read2.fastq.gz
+│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.vcf
+│   └─── target_regions.bed
+└── metadata
+    └── metadata.json
+```
+
+The only requirements are that `metadata/metadata.json` exists and the `files/` directory contains all of the other files.
+Data files may be nested under subfolders inside `files/` for better organization.
+For example, each donor could have their own folder for files.
+
+
+### Example submission procedure
+
+After preparing your submission as outlined above, you can use the following command to validate, encrypt and upload the submission:
+
+```bash
+grz-cli submit --submission-dir EXAMPLE_SUBMISSION
+```
+
+### Troubleshooting
+
+In case of issues, please re-run your commands with `grz-cli --log-level DEBUG --log-file path/to/write/file.log [...]` and submit the log file to the GDC data steward.
+
+## Command-Line Interface
+
+`grz-cli` provides a command-line interface with the following subcommands:
+
+### submit
+
+The `submit` command is the recommended command for submitting data to genome data centers.
+It combines the `validate`, `encrypt`, and `upload` commands (see below).
+
+- `-s, --submission-dir`: Path to the submission directory containing both 'metadata/' and 'files/' directories [**Required**]
+- `-c, --config-file`: Path to config file [_optional_]
+
+```bash
+grz-cli submit --submission-dir foo
+```
+
+
+### validate
+
+It is recommended to run this command before continuing with encryption and upload.
+
+- `--submission-dir`: Path to the submission directory containing both 'metadata/' and 'files/' directories [**Required**]
+
+Example usage:
+
+```bash
+grz_cli validate --submission-dir foo
+```
+
+### encrypt
+
+If a working directory is not provided, then the current directory is used automatically.
+Files are stored in a folder named `encrypted_files` as a sub-folder of the working directory.
+
+- `-s, --submission-dir`: Path to the submission directory containing both 'metadata/' and 'files/' directories [**Required**]
+- `-c, --config-file`: Path to config file [_optional_]
+
+```bash
+grz-cli encrypt --submission-dir foo
+```
+
+### upload
+
+Upload the submission into a S3 structure of a GRZ.
+
+- `-s, --submission-dir`: Path to the submission directory containing both `metadata/` and `encrypted_files/` directories [**Required**]
+- `-c, --config-file`: Path to config file [_optional_]
+
+Example usage:
+
+```bash
+grz-cli upload --submission-dir foo
+```
+
+### get-id
+
+*Available in grz-cli v1.2.0 or higher.*
+
+Compute and print the submission ID from a submission's JSON metadata.
+
+This is useful in case you forget to store the ID printed during upload.
+
+Example usage:
+
+```bash
+grz-cli get-id path/to/metadata.json
+```
+
+## Contributing
+
+### Running unreleased/development versions
+
+First, install `uv`.
+We recommend using Conda or [Pixi](https://pixi.sh/latest).
+
+After cloning the desired branch of the `grz-tools` repo locally, you can run `grz-cli` directly from the repo using:
+
+```
+uv run --project path/to/cloned/grz-tools grz-cli --help
+```
+
+### Testing
+
+Please note that binary files used for testing are managed with [Git LFS](https://git-lfs.com), which will be needed to clone them locally with the git repository.
+
+To run the tests, navigate to the root directory of your project and invoke `pytest`.
+Alternatively, install `uv` and `tox` and run `uv run tox`.
+
+## License
+
+This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.
+
+## Acknowledgements
+
+Parts of [Crypt4GH](https://github.com/EGA-archive/crypt4gh) are used in modified form.

{grz_cli-1.1.1 → grz_cli-1.2.0}/pyproject.toml

@@ -22,11 +22,11 @@ dependencies = [
     "pydantic >=2.9.2,<2.10",
     "pydantic-settings >=2.9.0,<2.10",
     "platformdirs >=4.3.6,<5",
-    "grz-pydantic-models >=2.1.2,<3",
+    "grz-pydantic-models >=2.2.1,<3",
     "pysam ==0.23.*",
     "rich ==13.*",
     "requests >=2.32.3,<3",
-    "grz-common >=1.1.1,<2",
+    "grz-common >=1.3.0,<2",
 ]
 classifiers = [
     "Programming Language :: Python :: 3",

grz_cli-1.2.0/src/grz_cli/__init__.py

@@ -0,0 +1 @@
+__version__ = "1.2.0"

{grz_cli-1.1.1 → grz_cli-1.2.0}/src/grz_cli/cli.py

@@ -4,19 +4,23 @@ CLI module for handling command-line interface operations.
 
 import logging
 import logging.config
+import shutil
+import subprocess
+import sys
 from importlib.metadata import version
+from textwrap import dedent
 
 import click
 import grz_pydantic_models.submission.metadata
+from grz_common.logging import setup_cli_logging
 
 from .commands.encrypt import encrypt
+from .commands.get_id import get_id
 from .commands.submit import submit
 from .commands.upload import upload
 from .commands.validate import validate
-from .constants import PACKAGE_ROOT
-from .logging_setup import add_filelogger
 
-log = logging.getLogger(PACKAGE_ROOT + ".cli")
+log = logging.getLogger(__name__)
 
 
 class OrderedGroup(click.Group):
@@ -41,7 +45,17 @@ def build_cli():
     @click.version_option(
         version=version("grz-cli"),
         prog_name="grz-cli",
-        message=f"%(prog)s v%(version)s (currently accepted metadata schema versions: {', '.join(grz_pydantic_models.submission.metadata.get_accepted_versions())})",
+        message=dedent(f"""\
+        %(prog)s v%(version)s
+        Currently accepted metadata schema versions: {", ".join(grz_pydantic_models.submission.metadata.get_accepted_versions())}
+        grz-common v{version("grz-common")}
+        grz-pydantic-models v{version("grz-pydantic-models")}
+        """)
+        + (
+            subprocess.run(["grz-check", "--version"], capture_output=True, text=True).stdout.strip()  # noqa: S603, S607
+            if shutil.which("grz-check") is not None
+            else ""
+        ),
     )
     @click.option("--log-file", metavar="FILE", type=str, help="Path to log file")
     @click.option(
@@ -58,26 +72,15 @@ def build_cli():
         :param log_level: Log level for the logger. It should be one of the following:
                            DEBUG, INFO, WARNING, ERROR, CRITICAL.
         """
-        if log_file:
-            add_filelogger(
-                log_file,
-                log_level.upper(),
-            )  # Add file logger
+        setup_cli_logging(log_file, log_level)
 
-        # show only time and log level in STDOUT
-        logging.basicConfig(
-            format="%(asctime)s - %(levelname)s - %(message)s",
-        )
-
-        # set the log level for this package
-        logging.getLogger(PACKAGE_ROOT).setLevel(log_level.upper())
-
-        log.debug("Logging setup complete.")
+        log.info(f"Running command: {' '.join(sys.argv)}")
 
     cli.add_command(validate)
     cli.add_command(encrypt)
     cli.add_command(upload)
     cli.add_command(submit)
+    cli.add_command(get_id)
 
     return cli
 

grz_cli-1.2.0/src/grz_cli/commands/get_id.py

@@ -0,0 +1,18 @@
+"""Command for computing the submission ID from the submission metadata."""
+
+import io
+import logging
+
+import click
+from grz_pydantic_models.submission.metadata import GrzSubmissionMetadata
+
+log = logging.getLogger(__name__)
+
+
+@click.command("get-id")
+@click.argument("metadata", type=click.File("r"))
+def get_id(metadata: io.TextIOWrapper):
+    """
+    Compute the submission ID for a given metadata JSON file.
+    """
+    click.echo(GrzSubmissionMetadata.model_validate_json(metadata.read()).submission_id)

{grz_cli-1.1.1 → grz_cli-1.2.0}/src/grz_cli/commands/upload.py

@@ -39,6 +39,8 @@ def upload(
         threads=threads,
     )
     # output the generated submission ID
-    click.echo(worker_inst.upload(config.s3))
+    submission_id = worker_inst.upload(config.s3)
+    log.info(f"Generated submission ID for upload: {submission_id}")
+    click.echo(submission_id)
 
     log.info("Upload finished!")

grz_cli-1.1.1/README.md

@@ -1,201 +0,0 @@
-# GRZ CLI
-
-A command-line tool for validating, encrypting, uploading and downloading submissions to/from a GDC/GRZ (Genomrechenzentrum).
-
-## Table of Contents
-
-- [Introduction](#introduction)
-- [Features](#features)
-- [Installation](#installation)
-  - [Requirements](#requirements)
-  - [End-user setup](#end-user-setup)
-    - [Installation via `conda` (recommended)](#installation-via-conda-recommended)
-    - [Installation via `pip` (not recommended)](#installation-via-pip-not-recommended)
-- [Usage](#usage)
-  - [Configuration](#configuration)
-  - [Exemplary submission procedure](#exemplary-submission-procedure)
-- [Command-Line Interface](#command-line-interface)
-- [Testing](#testing)
-- [Contributing](#contributing)
-- [License](#license)
-- [Acknowledgements](#acknowledgements)
-
-## Introduction
-
-This tool provides a way to validate files, encrypt/decrypt files using the [crypt4gh](https://crypt4gh.readthedocs.io/en/latest/) library and upload/download the encrypted files to an S3 bucket of a GDC/GRZ. It also logs the progress and outcomes of these operations in a metadata file.
-
-It is recommended to have the following folder structure for a single submission:
-
-```
-EXAMPLE_SUBMISSION
-├── files
-│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.read1.fastq.gz
-│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.read2.fastq.gz
-│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.vcf
-│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.read1.fastq.gz
-│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.read2.fastq.gz
-│   ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.vcf
-│   ├── target_regions.bed
-└── metadata
-    └── metadata.json
-```
-
-The current version of the tool requires the `working_dir` to have at least as much free disk space as the total size of the data being submitted.
-## Features
-
-- **Validation**: Validate file checksums, basic file metadata and BfArM requirements.
-- **Encryption**: Encrypt files using `crypt4gh`.
-- **Decryption**: Encrypt files using `crypt4gh`.
-- **Upload**: Upload encrypted files directly to a GRZ either (via built-in `boto3`).
-- **Download**: Download encrypted files from a GRZ (via built-in `boto3`).
-- **Logging**: Log progress and results of operations
-
-
-## Installation
-
-### Requirements
-Beside of the disk space requirements for the submission data, this tool also requires a linux environment, e.g.:
-- Linux server
-- Virtual machine running linux
-- Docker container
-- Windows subsystem for linux
-- ...
-
-
-### End-user setup
-The recommended method to install this tool is using the conda package manager.
-
-
-#### Installation via `conda` (recommended)
-
-If `conda` is not yet available on your system, we recommend to install the [Miniforge conda distribution](https://github.com/conda-forge/miniforge) by running the following commands:
-```bash
-curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
-bash Miniforge3-$(uname)-$(uname -m).sh
-```
-There are also alternative ways to install conda:
-- [Micromamba, a single executable that does not require a base environment](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html)
-- [Official installation instructions](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html)
-
-Next, install the `grz-cli` tool:
-```bash
-# create conda environment and activate it
-conda create -n grz-tools -c conda-forge -c bioconda "grz-cli"
-conda activate grz-tools
-```
-
-##### Update instructions
-
-Use the following command to update the tool:
-
-```bash
-conda update -n grz-tools -c conda-forge -c bioconda grz-cli
-```
-
-
-#### Installation via `pip` (not recommended)
-While installation via `pip` is possible, it is not recommended because users must ensure
-that the correct Python version is already installed and that they are using a virtual python environment.
-
-```bash
-pip install grz-cli
-```
-##### Update instructions:
-Use the following command to update the tool:
-```bash
-pip upgrade grz-cli
-```
-
-
-#### Docker
-Docker images are available via biocontainers at [https://biocontainers.pro/tools/grz-cli](https://biocontainers.pro/tools/grz-cli).
-
-The build process can take at least a few days after the Bioconda release, so double-check that the latest version in Bioconda is also the latest Docker image version.
-
-## Usage
-
-### Configuration
-**The configuration file will be provided by your associated GRZ, please place it into `~/.config/grz-cli/config.yaml`.**
-
-The tool requires a configuration file in YAML format to specify the S3 bucket and other options.
-For an exemplary configuration, see [resources/config.yaml](resources/config.yaml).
-
-S3 access and secret key can be listed either in the config file or as environment variable (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`).
-
-### Exemplary submission procedure
-After preparing your submission as outlined above, you can use the following commands to validate, encrypt and upload the submission: 
-```sh
-# Validate the submission
-grz-cli validate --submission-dir EXAMPLE_SUBMISSION
-
-# Encrypt the submission
-grz-cli encrypt --submission-dir EXAMPLE_SUBMISSION
-
-# Upload the submission
-grz-cli upload --submission-dir EXAMPLE_SUBMISSION
-```
-
-### Troubleshooting
-**In case of issues, please re-run your commands with `grz-cli --log-level DEBUG --log-file <your-log-file.log> [...]` and submit the log file to the GRZ data steward!**
-
-## Command-Line Interface
-
-`grz-cli` provides a command-line interface with the following subcommands:
-
-### validate
-
-It is recommended to run this command before continuing with encryption and upload.
-Progress files are stored relative to the submission directory.
-
-- `--submission-dir`: Path to the submission directory containing both 'metadata/' and 'files/' directories [**Required**]
-
-Example usage:
-
-```bash
-grz_cli validate --submission-dir foo
-```
-
-### encrypt
-
-If a working directory is not provided, then the current directory is used automatically. The log-files are going to be stored in the sub-folder of the working directory.
-Files are stored in a folder named `encrypted_files` as a sub-folder of the working directory.
-
-- `-s, --submission-dir`: Path to the submission directory containing both 'metadata/' and 'files/' directories [**Required**]
-- `-c, --config-file`: Path to config file [_optional_]
-
-```bash
-grz-cli encrypt --submission-dir foo
-```
-
-### upload
-
-Upload the submission into a S3 structure of a GRZ.
-
-- `-s, --submission-dir`: Path to the submission directory containing both 'metadata/' and 'encrypted_files/' directories [**Required**]
-- `-c, --config-file`: Path to config file [_optional_]
-
-Example usage:
-
-```bash
-grz-cli upload --submission-dir foo
-```
-
-## Testing
-
-Please note that binary files used for testing are managed with [Git LFS](https://git-lfs.com), which will be needed to clone them locally with the git repository.
-
-To run the tests, navigate to the root directory of your project and invoke `pytest`.
-Alternatively, install `uv` and `tox` and run `uv run tox`.
-
-## Contributing
-
-<!-- Add details about how others can contribute to the project -->
-
-## License
-
-This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.
-
-## Acknowledgements
-
-Parts of `crypt4gh` code is used in modified form
-

grz_cli-1.1.1/src/grz_cli/__init__.py

@@ -1 +0,0 @@
-__version__ = "1.1.1"

grz_cli-1.1.1/src/grz_cli/constants.py

@@ -1,38 +0,0 @@
-"""Constants for logging configuration, JSON schema validation, and other settings."""
-
-PACKAGE_ROOT = "grz_cli"
-
-LOGGING_FORMAT = "%(asctime)s [%(levelname)s] %(name)s: %(message)s"
-LOGGING_DATEFMT = "%Y-%m-%d %I:%M %p"
-LOGGING_CONFIG = {
-    "version": 1,
-    "disable_existing_loggers": True,
-    "formatters": {
-        "standard": {
-            "format": LOGGING_FORMAT,
-            "datefmt": LOGGING_DATEFMT,
-        },
-    },
-    "handlers": {
-        "default": {
-            "level": "INFO",
-            "formatter": "standard",
-            "class": "logging.StreamHandler",
-        },
-    },
-    "loggers": {
-        "": {  # root logger
-            "handlers": ["default"],
-            "level": "WARNING",
-            "propagate": False,
-        },
-        PACKAGE_ROOT: {"handlers": ["default"], "level": "INFO", "propagate": False},
-        "__main__": {  # if __name__ == '__main__'
-            "handlers": ["default"],
-            "level": "DEBUG",
-            "propagate": False,
-        },
-    },
-}
-
-TQDM_SMOOTHING: float = 0.0001

grz_cli-1.1.1/src/grz_cli/logging_setup.py

@@ -1,59 +0,0 @@
-"""
-Module: logging_setup
-
-This module provides functions for setting up logging configuration.
-"""
-
-from __future__ import annotations
-
-import logging
-from os import PathLike
-from pathlib import Path
-
-from .constants import LOGGING_DATEFMT, LOGGING_FORMAT, PACKAGE_ROOT
-
-log = logging.getLogger(__name__)
-
-
-def add_filelogger(
-    file_path: str | PathLike | None = None,
-    level: str = "INFO",
-    logger_name: str = PACKAGE_ROOT,
-) -> None:
-    """
-    Add file logging for the specified package.
-
-    This function configures a file logger to capture log messages
-    for the package specified by _PACKAGE_ROOT. If no file path
-    is provided, a default log file will be created in the user's
-    home directory.
-
-    :param file_path: Optional; the path to the log file. If None,
-                      a default path will be used.
-    :param level: Optional; the logging level. Default is 'INFO'.
-                  Must be a valid logging level name (e.g., 'DEBUG', 'INFO').
-    :param logger_name:
-    """
-    logger = logging.getLogger(logger_name)
-
-    if file_path is None:
-        default_log_dir = Path.home() / "logs"
-        default_log_dir.mkdir(parents=True, exist_ok=True)
-        file_path = default_log_dir / f"{PACKAGE_ROOT}.log"
-        log.warning("No log file path provided, using default: %s", file_path)
-    else:
-        file_path = Path(file_path)
-
-    try:
-        fh = logging.FileHandler(file_path)
-        fh.setLevel(level.upper())
-        fh.setFormatter(logging.Formatter(LOGGING_FORMAT, LOGGING_DATEFMT))
-        logger.addHandler(fh)
-        log.info(
-            "File logger added for %s at %s with level %s.",
-            logger.name,
-            file_path,
-            level.upper(),
-        )
-    except Exception as e:
-        log.error("Failed to add file logger: %s", e)