grz-cli 0.1.0__tar.gz → 0.1.2__tar.gz

{grz_cli-0.1.0/src/grz_cli.egg-info → grz_cli-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: grz-cli
-Version: 0.1.0
+Version: 0.1.2
 Summary: Tool for validation, encryption and upload of MV submissions to GDCs.
 Author-email: Koray Kirli <koraykirli@gmail.com>, Mathias Lesche <mathias.lesche@tu-dresden.de>, "Florian R. Hölzlwimmer" <git.ich@frhoelzlwimmer.de>, Till Hartmann <till.hartmann@bih-charite.de>, Thomas Sell <thomas.sell@bih-charite.de>
 License: MIT
@@ -12,22 +12,22 @@ Classifier: Operating System :: OS Independent
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: boto3~=1.35.39
-Requires-Dist: click~=8.1.7
-Requires-Dist: crypt4gh~=1.7
-Requires-Dist: jsonschema~=4.23.0
-Requires-Dist: pyyaml~=6.0.2
-Requires-Dist: tqdm~=4.66.5
-Requires-Dist: pydantic~=2.9.2
-Requires-Dist: platformdirs~=4.3.6
+Requires-Dist: boto3<2,>=1.35.39
+Requires-Dist: click<9,>=8.1.7
+Requires-Dist: crypt4gh<2,>=1.7
+Requires-Dist: jsonschema<5,>=4.23.0
+Requires-Dist: pyyaml<7,>=6.0.2
+Requires-Dist: tqdm<5,>=4.66.5
+Requires-Dist: pydantic<2.10,>=2.9.2
+Requires-Dist: platformdirs<5,>=4.3.6
 Provides-Extra: test
-Requires-Dist: pytest~=8.3.3; extra == "test"
-Requires-Dist: pytest-cov~=5.0.0; extra == "test"
-Requires-Dist: pytest-mock~=3.14.0; extra == "test"
-Requires-Dist: moto[s3]~=5.0.16; extra == "test"
-Requires-Dist: pytest-sugar~=1.0.0; extra == "test"
-Requires-Dist: ruff~=0.7.0; extra == "test"
-Requires-Dist: numpy~=2.1.2; extra == "test"
+Requires-Dist: pytest<9,>=8.3.3; extra == "test"
+Requires-Dist: pytest-cov<6,>=5.0.0; extra == "test"
+Requires-Dist: pytest-mock<4,>=3.14.0; extra == "test"
+Requires-Dist: moto[s3]<6,>=5.0.16; extra == "test"
+Requires-Dist: pytest-sugar<2,>=1.0.0; extra == "test"
+Requires-Dist: ruff<1,>=0.8.1; extra == "test"
+Requires-Dist: numpy<3,>=2.1.2; extra == "test"
 
 # GRZ CLI
 
@@ -38,6 +38,11 @@ A command-line tool for validating, encrypting, uploading and downloading submis
 - [Introduction](#introduction)
 - [Features](#features)
 - [Installation](#installation)
+  - [Requirements](#requirements)
+  - [End-user setup](#end-user)
+    - [Installation via `conda` (recommended)](#installation-via-conda-recommended)
+    - [Installation via `pip` (not recommended)](#installation-via-pip-not-recommended)
+  - [Development setup](#development-setup)
 - [Usage](#usage)
   - [Configuration](#configuration)
   - [Exemplary submission procedure](#exemplary-submission-procedure)
@@ -77,35 +82,61 @@ The current version of the tool requires the `working_dir` to have at least as m
 - **Download**: Download encrypted files from a GRZ (via built-in `boto3`).
 - **Logging**: Log progress and results of operations
 
+
 ## Installation
 
-### End-user
-This tool uses the `conda` package manager.
-If `conda` is not yet available on your system, you can install the [Miniforge conda distribution](https://github.com/conda-forge/miniforge) by running the following commands:
+### Requirements
+Beside of the disk space requirements for the submission data, this tool also requires a linux environment, e.g.:
+- Linux server
+- Virtual machine running linux
+- Docker container
+- Windows subsystem for linux
+- ...
+
+
+### End-user setup
+The recommended method to install this tool is using the conda package manager.
+
+
+#### Installation via `conda` (recommended)
+
+If `conda` is not yet available on your system, we recommend to install the [Miniforge conda distribution](https://github.com/conda-forge/miniforge) by running the following commands:
 ```bash
 curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
 bash Miniforge3-$(uname)-$(uname -m).sh
 ```
+There are also alternative ways to install conda:
+- [Micromamba, a single executable that does not require a base environment](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html)
+- [Official installation instructions](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html)
+
 Next, install the `grz-cli` tool:
 ```bash
-# TODO update: This requires membership in the repository
-# download environment.yaml from the git repository:
-git archive --remote=ssh://git@codebase.helmholtz.cloud/grz-mv-genomseq/grz-cli.git dev environment.yaml | tar -xv
-# TODO update: This will work once the repository is publicly accessible:
-# download environment.yaml from the git repository:
-curl -L "https://codebase.helmholtz.cloud/grz-mv-genomseq/grz-cli/-/raw/v0.1.0/environment.yaml?ref_type=heads" > environment.yaml
-
 # create conda environment and activate it
-conda env create -f environment.yaml -n grz-tools
+conda create -n grz-tools -c conda-forge -c bioconda "grz-cli"
 conda activate grz-tools
+```
 
-# install the grz-cli tool
-# TODO update: This requires membership in the repository
-pip install "git+ssh://git@codebase.helmholtz.cloud/grz-mv-genomseq/grz-cli.git@dev"
-# TODO update: This will work once the repository is publicly accessible:
-pip install "git+https://codebase.helmholtz.cloud/grz-mv-genomseq/grz-cli@v0.1.0"
+##### Update instructions:
+Use the following command to update the tool:
+```bash
+conda update -n grz-tools "grz-cli"
 ```
 
+
+#### Installation via `pip` (not recommended)
+While installation via `pip` is possible, it is not recommended because users must ensure
+that the correct Python version is already installed and that they are using a virtual python environment.
+
+```bash
+pip install grz-cli
+```
+##### Update instructions:
+Use the following command to update the tool:
+```bash
+pip upgrade grz-cli
+```
+
+
 ### Development setup
 For development purposes, you can clone the repository and install the package in editable mode:
 

{grz_cli-0.1.0 → grz_cli-0.1.2}/README.md

@@ -7,6 +7,11 @@ A command-line tool for validating, encrypting, uploading and downloading submis
 - [Introduction](#introduction)
 - [Features](#features)
 - [Installation](#installation)
+  - [Requirements](#requirements)
+  - [End-user setup](#end-user)
+    - [Installation via `conda` (recommended)](#installation-via-conda-recommended)
+    - [Installation via `pip` (not recommended)](#installation-via-pip-not-recommended)
+  - [Development setup](#development-setup)
 - [Usage](#usage)
   - [Configuration](#configuration)
   - [Exemplary submission procedure](#exemplary-submission-procedure)
@@ -46,35 +51,61 @@ The current version of the tool requires the `working_dir` to have at least as m
 - **Download**: Download encrypted files from a GRZ (via built-in `boto3`).
 - **Logging**: Log progress and results of operations
 
+
 ## Installation
 
-### End-user
-This tool uses the `conda` package manager.
-If `conda` is not yet available on your system, you can install the [Miniforge conda distribution](https://github.com/conda-forge/miniforge) by running the following commands:
+### Requirements
+Beside of the disk space requirements for the submission data, this tool also requires a linux environment, e.g.:
+- Linux server
+- Virtual machine running linux
+- Docker container
+- Windows subsystem for linux
+- ...
+
+
+### End-user setup
+The recommended method to install this tool is using the conda package manager.
+
+
+#### Installation via `conda` (recommended)
+
+If `conda` is not yet available on your system, we recommend to install the [Miniforge conda distribution](https://github.com/conda-forge/miniforge) by running the following commands:
 ```bash
 curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
 bash Miniforge3-$(uname)-$(uname -m).sh
 ```
+There are also alternative ways to install conda:
+- [Micromamba, a single executable that does not require a base environment](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html)
+- [Official installation instructions](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html)
+
 Next, install the `grz-cli` tool:
 ```bash
-# TODO update: This requires membership in the repository
-# download environment.yaml from the git repository:
-git archive --remote=ssh://git@codebase.helmholtz.cloud/grz-mv-genomseq/grz-cli.git dev environment.yaml | tar -xv
-# TODO update: This will work once the repository is publicly accessible:
-# download environment.yaml from the git repository:
-curl -L "https://codebase.helmholtz.cloud/grz-mv-genomseq/grz-cli/-/raw/v0.1.0/environment.yaml?ref_type=heads" > environment.yaml
-
 # create conda environment and activate it
-conda env create -f environment.yaml -n grz-tools
+conda create -n grz-tools -c conda-forge -c bioconda "grz-cli"
 conda activate grz-tools
+```
 
-# install the grz-cli tool
-# TODO update: This requires membership in the repository
-pip install "git+ssh://git@codebase.helmholtz.cloud/grz-mv-genomseq/grz-cli.git@dev"
-# TODO update: This will work once the repository is publicly accessible:
-pip install "git+https://codebase.helmholtz.cloud/grz-mv-genomseq/grz-cli@v0.1.0"
+##### Update instructions:
+Use the following command to update the tool:
+```bash
+conda update -n grz-tools "grz-cli"
 ```
 
+
+#### Installation via `pip` (not recommended)
+While installation via `pip` is possible, it is not recommended because users must ensure
+that the correct Python version is already installed and that they are using a virtual python environment.
+
+```bash
+pip install grz-cli
+```
+##### Update instructions:
+Use the following command to update the tool:
+```bash
+pip upgrade grz-cli
+```
+
+
 ### Development setup
 For development purposes, you can clone the repository and install the package in editable mode:
 

{grz_cli-0.1.0 → grz_cli-0.1.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "grz-cli"
-version = "0.1.0"
+version = "0.1.2"
 description = "Tool for validation, encryption and upload of MV submissions to GDCs."
 readme = "README.md"
 license = { text = "MIT" }
@@ -16,14 +16,14 @@ authors = [
     { name = "Thomas Sell", email = "thomas.sell@bih-charite.de" },
 ]
 dependencies = [
-    "boto3 ~=1.35.39",
-    "click ~=8.1.7",
-    "crypt4gh ~=1.7",
-    "jsonschema ~=4.23.0",
-    "pyyaml ~=6.0.2",
-    "tqdm ~=4.66.5",
-    "pydantic ~=2.9.2",
-    "platformdirs ~=4.3.6",
+    "boto3 >=1.35.39,<2",
+    "click >=8.1.7,<9",
+    "crypt4gh >=1.7,<2",
+    "jsonschema >=4.23.0,<5",
+    "pyyaml >=6.0.2,<7",
+    "tqdm >=4.66.5,<5",
+    "pydantic >=2.9.2,<2.10",
+    "platformdirs >=4.3.6,<5",
 ]
 classifiers = [
     "Programming Language :: Python :: 3",
@@ -41,13 +41,13 @@ grz-cli = "grz_cli.cli:cli"
 
 [project.optional-dependencies]
 test = [
-    "pytest ~=8.3.3",
-    "pytest-cov ~=5.0.0",
-    "pytest-mock ~=3.14.0",
-    "moto[s3] ~=5.0.16",
-    "pytest-sugar ~=1.0.0",
-    "ruff ~=0.7.0",
-    "numpy ~=2.1.2"
+    "pytest >=8.3.3,<9",
+    "pytest-cov >=5.0.0,<6",
+    "pytest-mock >=3.14.0,<4",
+    "moto[s3] >=5.0.16,<6",
+    "pytest-sugar >=1.0.0,<2",
+    "ruff >=0.8.1,<1",
+    "numpy >=2.1.2,<3"
 ]
 
 [dependency-groups]
@@ -177,6 +177,7 @@ commands = [["mypy"]]
 dev-dependencies = [
     "datamodel-code-generator>=0.26.2",
     "mypy>=1.13.0",
+    "ruff>=0.8.1",
     "tox>=4.23.2",
     "types-pyyaml>=6.0.12.20240917",
     "types-tqdm>=4.66.0.20240417",

{grz_cli-0.1.0 → grz_cli-0.1.2}/src/grz_cli/download.py

@@ -18,6 +18,7 @@ from tqdm.auto import tqdm
 
 from .models.config import ConfigModel
 from .progress_logging import FileProgressLogger
+from .states import DownloadState
 
 MULTIPART_THRESHOLD = 8 * 1024 * 1024  # 8MiB, boto3 default
 MULTIPART_CHUNKSIZE = 8 * 1024 * 1024  # 8MiB, boto3 default
@@ -174,7 +175,7 @@ class S3BotoDownloadWorker:
         Upload an encrypted submission
         :param encrypted_submission: The encrypted submission to upload
         """
-        progress_logger = FileProgressLogger(self._status_file_path)
+        progress_logger = FileProgressLogger[DownloadState](self._status_file_path)
 
         encrypted_files_key_prefix = f"{submission_id}/files/"
 
@@ -216,7 +217,7 @@ class S3BotoDownloadWorker:
                             progress_logger.set_state(
                                 full_path,
                                 file_metadata,
-                                state={"download_successful": True},
+                                state=DownloadState(download_successful=True),
                             )
                         except Exception as e:
                             self.__log.error("Download failed for '%s'", str(full_path))
@@ -224,7 +225,9 @@ class S3BotoDownloadWorker:
                             progress_logger.set_state(
                                 full_path,
                                 file_metadata,
-                                state={"download_successful": False, "error": str(e)},
+                                state=DownloadState(
+                                    download_successful=False, errors=[str(e)]
+                                ),
                             )
 
                             raise e

{grz_cli-0.1.0 → grz_cli-0.1.2}/src/grz_cli/fastq_validation.py

@@ -2,9 +2,18 @@
 
 import gzip
 import logging
+import os
+import typing
 from collections.abc import Generator
 from contextlib import contextmanager
+from gzip import GzipFile
+from io import RawIOBase
 from os import PathLike
+from typing import TextIO
+
+from tqdm.auto import tqdm
+
+from .file_operations import TqdmIOWrapper
 
 log = logging.getLogger(__name__)
 
@@ -20,21 +29,41 @@ def is_gzipped(file_path: str | PathLike) -> bool:
 
 
 @contextmanager
-def open_fastq(file_path: str | PathLike):
+def open_fastq(
+    file_path: str | PathLike, progress=True
+) -> Generator[TextIO, None, None]:
     """
     Open a FASTQ file, handling both regular and gzipped formats.
 
     :param file_path: Path to the FASTQ file
+    :param progress: Whether to show a progress bar
     :return: A file object opened in the appropriate mode (gzipped or plain text)
     """
-    if is_gzipped(file_path):
-        with gzip.open(file_path, "rb") as f:
-            # Open gzipped FASTQ
-            yield f
-    else:
-        with open(file_path) as f:
-            # Open regular FASTQ
-            yield f
+    handle: TqdmIOWrapper | GzipFile | typing.BinaryIO | None = None
+    with open(file_path, "rb") as fd:
+        # Open FASTQ
+        total_size = os.stat(file_path).st_size
+        if progress:
+            handle = TqdmIOWrapper(
+                typing.cast(RawIOBase, fd),
+                tqdm(
+                    total=total_size,
+                    desc="Reading FASTQ: ",
+                    unit="B",
+                    unit_scale=True,
+                    # unit_divisor=1024,  # make use of standard units e.g. KB, MB, etc.
+                    miniters=1,
+                ),
+            )
+        else:
+            handle = fd
+
+        if is_gzipped(file_path):
+            # decompress
+            with gzip.open(typing.cast(RawIOBase, handle), "rb") as decompressed_fd:
+                yield typing.cast(TextIO, decompressed_fd)
+        else:
+            yield typing.cast(TextIO, handle)
 
 
 def calculate_fastq_stats(file_path) -> tuple[int, set[int]]:

{grz_cli-0.1.0 → grz_cli-0.1.2}/src/grz_cli/models/v1_0_0/metadata.py

@@ -16,6 +16,7 @@
 
 from __future__ import annotations
 
+import logging
 from collections.abc import Generator
 from datetime import date
 from enum import StrEnum
@@ -33,6 +34,8 @@ from pydantic.alias_generators import to_camel
 
 from grz_cli.file_operations import calculate_sha256  # type: ignore
 
+log = logging.getLogger(__name__)
+
 
 class StrictBaseModel(BaseModel):
     model_config = ConfigDict(
@@ -831,10 +834,11 @@ class GrzSubmissionMetadata(StrictBaseModel):
                 threshold = thresholds.get(
                     (lab_datum.library_type, lab_datum.sequence_subtype)
                 )
-                if not threshold:
-                    raise ValueError(
-                        f"Threshold for {lab_datum.library_type} {lab_datum.sequence_subtype} not found."
+                if threshold is None:
+                    log.warning(
+                        f"Threshold for {lab_datum.library_type} - {lab_datum.sequence_subtype} not found!"
                     )
+
                 sequence_data = lab_datum.sequence_data
                 if sequence_data.mean_depth_of_coverage < threshold:
                     raise ValueError(

{grz_cli-0.1.0 → grz_cli-0.1.2}/src/grz_cli/parser.py

@@ -13,15 +13,17 @@ from .download import S3BotoDownloadWorker
 from .fastq_validation import validate_paired_end_reads, validate_single_end_reads
 from .file_operations import Crypt4GH, calculate_sha256
 from .models.config import ConfigModel
-from .models.v1_0_0.metadata import File as SubmissionFileMetadata
 from .models.v1_0_0.metadata import (
+    File,
     FileType,
     GrzSubmissionMetadata,
     ReadOrder,
     SequenceData,
-    SequenceType,
     SequencingLayout,
 )
+from .models.v1_0_0.metadata import File as SubmissionFileMetadata
+from .progress_logging import FileProgressLogger
+from .states import DecryptionState, EncryptionState, ValidationState
 from .upload import S3BotoUploadWorker
 
 log = logging.getLogger(__name__)
@@ -168,9 +170,9 @@ class Submission:
 
         :return: Generator of errors
         """
-        from .progress_logging import FileProgressLogger
-
-        progress_logger = FileProgressLogger(log_file_path=progress_log_file)
+        progress_logger = FileProgressLogger[ValidationState](
+            log_file_path=progress_log_file
+        )
         # cleanup log file and keep only files listed here
         progress_logger.cleanup(
             keep=[
@@ -182,43 +184,25 @@ class Submission:
         # - "errors": List[str]
         # - "validation_passed": bool
 
-        for local_file_path, file_metadata in self.files.items():
-            logged_state = progress_logger.get_state(local_file_path, file_metadata)
-
-            # determine if we can skip the verification
-            if logged_state is None:
-                self.__log.debug("State for %s not calculated yet", local_file_path)
-            elif not logged_state.get("validation_passed", False):
-                errors = logged_state.get("errors", [])
-                yield from errors
-
-                # skip re-verification
-                continue
-            else:
-                self.__log.debug(
-                    "Validation for %s already passed, skipping...",
-                    str(local_file_path),
-                )
-
-                # skip re-verification
-                continue
-
+        def validate_file(local_file_path, file_metadata):
             self.__log.debug("Validating '%s'...", str(local_file_path))
+
             # validate the file
             errors = list(file_metadata.validate_data(local_file_path))
             validation_passed = len(errors) == 0
 
-            # log state
-            progress_logger.set_state(
+            # return log state
+            return ValidationState(errors=errors, validation_passed=validation_passed)
+
+        for local_file_path, file_metadata in self.files.items():
+            logged_state = progress_logger.get_state(
                 local_file_path,
                 file_metadata,
-                state={
-                    "errors": errors,
-                    "validation_passed": validation_passed,
-                },
+                default=validate_file,  # validate the file if the state was not calculated yet
             )
 
-            yield from errors
+            if logged_state:
+                yield from logged_state["errors"]
 
     def validate_sequencing_data(
         self, progress_log_file: str | PathLike
@@ -228,51 +212,131 @@ class Submission:
 
         :return: Generator of errors
         """
+        from .progress_logging import FileProgressLogger
 
-        def find_fastq_files(sequence_data: SequenceData):
+        progress_logger = FileProgressLogger[ValidationState](
+            log_file_path=progress_log_file
+        )
+        # cleanup log file and keep only files listed here
+        progress_logger.cleanup(
+            keep=[
+                (file_path, file_metadata)
+                for file_path, file_metadata in self.files.items()
+            ]
+        )
+        # fields:
+        # - "errors": List[str]
+        # - "validation_passed": bool
+
+        def find_fastq_files(sequence_data: SequenceData) -> list[File]:
             return [f for f in sequence_data.files if f.file_type == FileType.fastq]
 
         for donor in self.metadata.content.donors:
             for lab_data in donor.lab_data:
-                sequence_type = lab_data.sequence_type
                 sequencing_layout = lab_data.sequencing_layout
                 sequence_data = lab_data.sequence_data
-                if sequence_type == SequenceType.dna:
-                    # find all FASTQ files
-                    fastq_files = find_fastq_files(sequence_data)
+                # find all FASTQ files
+                fastq_files = find_fastq_files(sequence_data)
+
+                match sequencing_layout:
+                    case (
+                        SequencingLayout.single_end
+                        | SequencingLayout.reverse
+                        | SequencingLayout.other
+                    ):
+                        yield from self._validate_single_end(
+                            fastq_files, progress_logger
+                        )
 
-                    if sequencing_layout == SequencingLayout.single_end:
-                        yield from validate_single_end_reads(
-                            self.files_dir / fastq_files[0].file_path
+                    case SequencingLayout.paired_end:
+                        yield from self._validate_paired_end(
+                            fastq_files, progress_logger
                         )
-                    else:
-                        # group reads by flowcell id and lane id
-                        key = lambda f: (f.flowcell_id, f.lane_id)
-                        fastq_files.sort(key=key)
-                        for _key, group in groupby(fastq_files, key):
-                            files = list(group)
-
-                            # separate R1 and R2 files
-                            fastq_r1_files = [
-                                f for f in files if f.read_order == ReadOrder.r1
-                            ]
-                            fastq_r2_files = [
-                                f for f in files if f.read_order == ReadOrder.r2
-                            ]
-
-                            for fastq_r1, fastq_r2 in zip(
-                                fastq_r1_files, fastq_r2_files, strict=True
-                            ):
-                                yield from validate_paired_end_reads(
-                                    # fastq R1
-                                    self.files_dir / fastq_r1.file_path,
-                                    # fastq R2
-                                    self.files_dir / fastq_r2.file_path,
-                                )
-
-                elif sequence_type == SequenceType.rna:
-                    # TODO: What to check here?
-                    pass
+
+    def _validate_single_end(
+        self,
+        fastq_files: list[File],
+        progress_logger: FileProgressLogger[ValidationState],
+    ) -> Generator[str, None, None]:
+        def validate_file(local_file_path, _file_metadata) -> ValidationState:
+            self.__log.debug("Validating '%s'...", str(local_file_path))
+
+            # validate the file
+            errors = list(validate_single_end_reads(local_file_path))
+            validation_passed = len(errors) == 0
+
+            # return log state
+            return ValidationState(
+                errors=errors,
+                validation_passed=validation_passed,
+            )
+
+        for fastq_file in fastq_files:
+            logged_state = progress_logger.get_state(
+                self.files_dir / fastq_file.file_path,
+                fastq_file,
+                default=validate_file,  # validate the file if the state was not calculated yet
+            )
+            if logged_state:
+                yield from logged_state["errors"]
+
+    def _validate_paired_end(
+        self,
+        fastq_files: list[File],
+        progress_logger: FileProgressLogger[ValidationState],
+    ) -> Generator[str, None, None]:
+        key = lambda f: (f.flowcell_id, f.lane_id)
+        fastq_files.sort(key=key)
+        for _key, group in groupby(fastq_files, key):
+            files = list(group)
+
+            # separate R1 and R2 files
+            fastq_r1_files = [f for f in files if f.read_order == ReadOrder.r1]
+            fastq_r2_files = [f for f in files if f.read_order == ReadOrder.r2]
+
+            for fastq_r1, fastq_r2 in zip(fastq_r1_files, fastq_r2_files, strict=True):
+                local_fastq_r1_path = self.files_dir / fastq_r1.file_path
+                local_fastq_r2_path = self.files_dir / fastq_r2.file_path
+
+                # get saved state
+                logged_state_r1 = progress_logger.get_state(
+                    local_fastq_r1_path,
+                    fastq_r1,
+                )
+                logged_state_r2 = progress_logger.get_state(
+                    local_fastq_r2_path,
+                    fastq_r2,
+                )
+                if (
+                    logged_state_r1 is None
+                    or logged_state_r2 is None
+                    or logged_state_r1 != logged_state_r2
+                ):
+                    # calculate state
+                    errors = list(
+                        validate_paired_end_reads(
+                            local_fastq_r1_path,  # fastq R1
+                            local_fastq_r2_path,  # fastq R2
+                        )
+                    )
+                    validation_passed = len(errors) == 0
+
+                    state = ValidationState(
+                        errors=errors,
+                        validation_passed=validation_passed,
+                    )
+                    # update state for both files
+                    progress_logger.set_state(  # fastq R1
+                        local_fastq_r1_path, fastq_r1, state
+                    )
+                    progress_logger.set_state(  # fastq R2
+                        local_fastq_r2_path, fastq_r2, state
+                    )
+
+                    yield from state["errors"]
+                else:
+                    # both fastq states are equal, so simply yield one of them
+                    yield from logged_state_r1["errors"]
 
     def encrypt(
         self,
@@ -312,7 +376,9 @@ class Submission:
 
         from .progress_logging import FileProgressLogger
 
-        progress_logger = FileProgressLogger(log_file_path=progress_log_file)
+        progress_logger = FileProgressLogger[EncryptionState](
+            log_file_path=progress_log_file
+        )
 
         try:
             public_keys = Crypt4GH.prepare_c4gh_keys(recipient_public_key_path)
@@ -347,7 +413,9 @@ class Submission:
 
                     self.__log.info(f"Encryption complete for {str(file_path)}. ")
                     progress_logger.set_state(
-                        file_path, file_metadata, state={"encryption_successful": True}
+                        file_path,
+                        file_metadata,
+                        state=EncryptionState(encryption_successful=True),
                     )
                 except Exception as e:
                     self.__log.error("Encryption failed for '%s'", str(file_path))
@@ -355,7 +423,9 @@ class Submission:
                     progress_logger.set_state(
                         file_path,
                         file_metadata,
-                        state={"encryption_successful": False, "error": str(e)},
+                        state=EncryptionState(
+                            encryption_successful=False, errors=[str(e)]
+                        ),
                     )
 
                     raise e
@@ -476,7 +546,9 @@ class EncryptedSubmission:
 
         from .progress_logging import FileProgressLogger
 
-        progress_logger = FileProgressLogger(log_file_path=progress_log_file)
+        progress_logger = FileProgressLogger[DecryptionState](
+            log_file_path=progress_log_file
+        )
 
         try:
             private_key = Crypt4GH.retrieve_private_key(recipient_private_key_path)
@@ -516,7 +588,7 @@ class EncryptedSubmission:
                     progress_logger.set_state(
                         encrypted_file_path,
                         file_metadata,
-                        state={"decryption_successful": True},
+                        state=DecryptionState(decryption_successful=True),
                     )
                 except Exception as e:
                     self.__log.error(
@@ -526,7 +598,9 @@ class EncryptedSubmission:
                     progress_logger.set_state(
                         encrypted_file_path,
                         file_metadata,
-                        state={"decryption_successful": False, "error": str(e)},
+                        state=DecryptionState(
+                            decryption_successful=False, errors=[str(e)]
+                        ),
                     )
 
                     raise e
@@ -665,21 +739,21 @@ class Worker:
             raise SubmissionValidationError(error_msg)
         else:
             self.__log.info("Checksum validation successful!")
-        self.__log.info("Starting checksum validation...")
 
+        self.__log.info("Starting sequencing data validation...")
         if errors := list(
             submission.validate_sequencing_data(
                 progress_log_file=self.progress_file_sequencing_data_validation
             )
         ):
-            error_msg = "\n".join(["Checksum validation failed! Errors:", *errors])
+            error_msg = "\n".join(
+                ["Sequencing data validation failed! Errors:", *errors]
+            )
             self.__log.error(error_msg)
 
             raise SubmissionValidationError(error_msg)
         else:
-            self.__log.info("Checksum validation successful!")
-
-        # TODO: validate FASTQ
+            self.__log.info("Sequencing data validation successful!")
 
     def encrypt(
         self,

{grz_cli-0.1.0 → grz_cli-0.1.2}/src/grz_cli/progress_logging.py

@@ -5,17 +5,18 @@ from __future__ import annotations
 import copy
 import json
 import typing
+from collections.abc import Callable
 from os import PathLike
 from pathlib import Path
 
 from .file_operations import read_multiple_json
 from .models.v1_0_0.metadata import File as SubmissionFileMetadata
+from .states import State
 
 Index = tuple[str, float, int]
-State = dict
 
 
-class FileProgressLogger:
+class FileProgressLogger[T: State]:
     """
     A class to log and track the state of files over time. It stores file states in a log file of JSON entries
     and allows querying the state based on the file path and modification time.
@@ -23,7 +24,7 @@ class FileProgressLogger:
 
     _index = {"file_path": str, "modification_time": float, "size": int}
     # mapping of index -> (metadata, data)
-    _file_states: dict[Index, tuple[SubmissionFileMetadata, State]]
+    _file_states: dict[Index, tuple[SubmissionFileMetadata, T]]
 
     def __init__(self, log_file_path: str | PathLike):
         """
@@ -95,12 +96,17 @@ class FileProgressLogger:
         self,
         file_path: str | PathLike,
         file_metadata: dict | SubmissionFileMetadata,
-    ) -> State | None:
+        default: T | Callable[[Path, SubmissionFileMetadata], T] | None = None,
+    ) -> T | None:
         """
         Retrieves the stored state of a file if it exists in the log.
 
         :param file_path: The file path to query for the state.
         :param file_metadata: The metadata of the file to query for the state.
+        :param default: Default state to use if the file does not exist.
+            Can be a Callable that takes the file path and the file metadata as input and returns some state.
+
+            The default state gets automatically saved as the state for this file in case there is no stored state.
         :return: A dictionary representing the file's state, or None if the file's state isn't logged.
         """
         file_path = Path(file_path)
@@ -110,7 +116,7 @@ class FileProgressLogger:
         stored_metadata, stored_data = self._file_states.get(index, (None, None))
 
         # check if metadata matches
-        if file_metadata and not isinstance(file_metadata, SubmissionFileMetadata):
+        if not isinstance(file_metadata, SubmissionFileMetadata):
             file_metadata = SubmissionFileMetadata(**file_metadata)
 
         if stored_metadata and not isinstance(stored_metadata, SubmissionFileMetadata):
@@ -118,14 +124,28 @@ class FileProgressLogger:
 
         if stored_metadata and file_metadata == stored_metadata:
             return copy.deepcopy(stored_data)
-        # metadata mismatch -> no valid stored state -> return None
-        return None
+
+        # metadata mismatch -> no valid stored state -
+        if file_metadata and default:
+            if callable(default):
+                state = default(file_path, file_metadata)
+                self.set_state(file_path, file_metadata, state)
+                return state
+            else:
+                if not default:
+                    raise ValueError("Default state must be provided if not callable")
+                    # return None
+                default = typing.cast(T, default)
+                self.set_state(file_path, file_metadata, default)
+                return default
+        else:
+            return None
 
     def set_state(
         self,
         file_path: str | PathLike,
         file_metadata: dict | SubmissionFileMetadata,
-        state: State,
+        state: T,
     ):
         """
         Log the state of a file:

grz_cli-0.1.2/src/grz_cli/states.py

@@ -0,0 +1,53 @@
+"""
+This module contains the type definitions for the progress logging states.
+"""
+
+from typing import TypedDict
+
+
+class State(TypedDict, total=False):
+    """
+    Base state class for progress logging.
+    """
+
+    errors: list[str]
+
+
+class ValidationState(State):
+    """
+    State for validation progress.
+    """
+
+    validation_passed: bool
+
+
+class UploadState(State):
+    """
+    State for upload progress.
+    """
+
+    upload_successful: bool
+
+
+class EncryptionState(State):
+    """
+    State for encryption progress.
+    """
+
+    encryption_successful: bool
+
+
+class DecryptionState(State):
+    """
+    State for decryption progress.
+    """
+
+    decryption_successful: bool
+
+
+class DownloadState(State):
+    """
+    State for download progress.
+    """
+
+    download_successful: bool

{grz_cli-0.1.0 → grz_cli-0.1.2}/src/grz_cli/upload.py

@@ -20,6 +20,7 @@ from tqdm.auto import tqdm
 
 from .models.config import ConfigModel
 from .progress_logging import FileProgressLogger
+from .states import UploadState
 
 MULTIPART_THRESHOLD = 8 * 1024 * 1024  # 8MiB, boto3 default
 MULTIPART_CHUNKSIZE = 8 * 1024 * 1024  # 8MiB, boto3 default
@@ -159,7 +160,7 @@ class S3BotoUploadWorker(UploadWorker):
         Upload an encrypted submission
         :param encrypted_submission: The encrypted submission to upload
         """
-        progress_logger = FileProgressLogger(self._status_file_path)
+        progress_logger = FileProgressLogger[UploadState](self._status_file_path)
         metadata_file_path, metadata_s3_object_id = (
             encrypted_submission.get_metadata_file_path_and_object_id()
         )
@@ -190,7 +191,9 @@ class S3BotoUploadWorker(UploadWorker):
 
                     self.__log.info(f"Upload complete for {str(file_path)}. ")
                     progress_logger.set_state(
-                        file_path, file_metadata, state={"upload_successful": True}
+                        file_path,
+                        file_metadata,
+                        state=UploadState(upload_successful=True),
                     )
                 except Exception as e:
                     self.__log.error("Upload failed for '%s'", str(file_path))
@@ -198,7 +201,7 @@ class S3BotoUploadWorker(UploadWorker):
                     progress_logger.set_state(
                         file_path,
                         file_metadata,
-                        state={"upload_successful": False, "error": str(e)},
+                        state=UploadState(upload_successful=False, errors=[str(e)]),
                     )
 
                     raise e

{grz_cli-0.1.0 → grz_cli-0.1.2/src/grz_cli.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: grz-cli
-Version: 0.1.0
+Version: 0.1.2
 Summary: Tool for validation, encryption and upload of MV submissions to GDCs.
 Author-email: Koray Kirli <koraykirli@gmail.com>, Mathias Lesche <mathias.lesche@tu-dresden.de>, "Florian R. Hölzlwimmer" <git.ich@frhoelzlwimmer.de>, Till Hartmann <till.hartmann@bih-charite.de>, Thomas Sell <thomas.sell@bih-charite.de>
 License: MIT
@@ -12,22 +12,22 @@ Classifier: Operating System :: OS Independent
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: boto3~=1.35.39
-Requires-Dist: click~=8.1.7
-Requires-Dist: crypt4gh~=1.7
-Requires-Dist: jsonschema~=4.23.0
-Requires-Dist: pyyaml~=6.0.2
-Requires-Dist: tqdm~=4.66.5
-Requires-Dist: pydantic~=2.9.2
-Requires-Dist: platformdirs~=4.3.6
+Requires-Dist: boto3<2,>=1.35.39
+Requires-Dist: click<9,>=8.1.7
+Requires-Dist: crypt4gh<2,>=1.7
+Requires-Dist: jsonschema<5,>=4.23.0
+Requires-Dist: pyyaml<7,>=6.0.2
+Requires-Dist: tqdm<5,>=4.66.5
+Requires-Dist: pydantic<2.10,>=2.9.2
+Requires-Dist: platformdirs<5,>=4.3.6
 Provides-Extra: test
-Requires-Dist: pytest~=8.3.3; extra == "test"
-Requires-Dist: pytest-cov~=5.0.0; extra == "test"
-Requires-Dist: pytest-mock~=3.14.0; extra == "test"
-Requires-Dist: moto[s3]~=5.0.16; extra == "test"
-Requires-Dist: pytest-sugar~=1.0.0; extra == "test"
-Requires-Dist: ruff~=0.7.0; extra == "test"
-Requires-Dist: numpy~=2.1.2; extra == "test"
+Requires-Dist: pytest<9,>=8.3.3; extra == "test"
+Requires-Dist: pytest-cov<6,>=5.0.0; extra == "test"
+Requires-Dist: pytest-mock<4,>=3.14.0; extra == "test"
+Requires-Dist: moto[s3]<6,>=5.0.16; extra == "test"
+Requires-Dist: pytest-sugar<2,>=1.0.0; extra == "test"
+Requires-Dist: ruff<1,>=0.8.1; extra == "test"
+Requires-Dist: numpy<3,>=2.1.2; extra == "test"
 
 # GRZ CLI
 
@@ -38,6 +38,11 @@ A command-line tool for validating, encrypting, uploading and downloading submis
 - [Introduction](#introduction)
 - [Features](#features)
 - [Installation](#installation)
+  - [Requirements](#requirements)
+  - [End-user setup](#end-user)
+    - [Installation via `conda` (recommended)](#installation-via-conda-recommended)
+    - [Installation via `pip` (not recommended)](#installation-via-pip-not-recommended)
+  - [Development setup](#development-setup)
 - [Usage](#usage)
   - [Configuration](#configuration)
   - [Exemplary submission procedure](#exemplary-submission-procedure)
@@ -77,35 +82,61 @@ The current version of the tool requires the `working_dir` to have at least as m
 - **Download**: Download encrypted files from a GRZ (via built-in `boto3`).
 - **Logging**: Log progress and results of operations
 
+
 ## Installation
 
-### End-user
-This tool uses the `conda` package manager.
-If `conda` is not yet available on your system, you can install the [Miniforge conda distribution](https://github.com/conda-forge/miniforge) by running the following commands:
+### Requirements
+Beside of the disk space requirements for the submission data, this tool also requires a linux environment, e.g.:
+- Linux server
+- Virtual machine running linux
+- Docker container
+- Windows subsystem for linux
+- ...
+
+
+### End-user setup
+The recommended method to install this tool is using the conda package manager.
+
+
+#### Installation via `conda` (recommended)
+
+If `conda` is not yet available on your system, we recommend to install the [Miniforge conda distribution](https://github.com/conda-forge/miniforge) by running the following commands:
 ```bash
 curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
 bash Miniforge3-$(uname)-$(uname -m).sh
 ```
+There are also alternative ways to install conda:
+- [Micromamba, a single executable that does not require a base environment](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html)
+- [Official installation instructions](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html)
+
 Next, install the `grz-cli` tool:
 ```bash
-# TODO update: This requires membership in the repository
-# download environment.yaml from the git repository:
-git archive --remote=ssh://git@codebase.helmholtz.cloud/grz-mv-genomseq/grz-cli.git dev environment.yaml | tar -xv
-# TODO update: This will work once the repository is publicly accessible:
-# download environment.yaml from the git repository:
-curl -L "https://codebase.helmholtz.cloud/grz-mv-genomseq/grz-cli/-/raw/v0.1.0/environment.yaml?ref_type=heads" > environment.yaml
-
 # create conda environment and activate it
-conda env create -f environment.yaml -n grz-tools
+conda create -n grz-tools -c conda-forge -c bioconda "grz-cli"
 conda activate grz-tools
+```
 
-# install the grz-cli tool
-# TODO update: This requires membership in the repository
-pip install "git+ssh://git@codebase.helmholtz.cloud/grz-mv-genomseq/grz-cli.git@dev"
-# TODO update: This will work once the repository is publicly accessible:
-pip install "git+https://codebase.helmholtz.cloud/grz-mv-genomseq/grz-cli@v0.1.0"
+##### Update instructions:
+Use the following command to update the tool:
+```bash
+conda update -n grz-tools "grz-cli"
 ```
 
+
+#### Installation via `pip` (not recommended)
+While installation via `pip` is possible, it is not recommended because users must ensure
+that the correct Python version is already installed and that they are using a virtual python environment.
+
+```bash
+pip install grz-cli
+```
+##### Update instructions:
+Use the following command to update the tool:
+```bash
+pip upgrade grz-cli
+```
+
+
 ### Development setup
 For development purposes, you can clone the repository and install the package in editable mode:
 

{grz_cli-0.1.0 → grz_cli-0.1.2}/src/grz_cli.egg-info/SOURCES.txt

@@ -12,6 +12,7 @@ src/grz_cli/file_operations.py
 src/grz_cli/logging_setup.py
 src/grz_cli/parser.py
 src/grz_cli/progress_logging.py
+src/grz_cli/states.py
 src/grz_cli/upload.py
 src/grz_cli.egg-info/PKG-INFO
 src/grz_cli.egg-info/SOURCES.txt

grz_cli-0.1.2/src/grz_cli.egg-info/requires.txt

@@ -0,0 +1,17 @@
+boto3<2,>=1.35.39
+click<9,>=8.1.7
+crypt4gh<2,>=1.7
+jsonschema<5,>=4.23.0
+pyyaml<7,>=6.0.2
+tqdm<5,>=4.66.5
+pydantic<2.10,>=2.9.2
+platformdirs<5,>=4.3.6
+
+[test]
+pytest<9,>=8.3.3
+pytest-cov<6,>=5.0.0
+pytest-mock<4,>=3.14.0
+moto[s3]<6,>=5.0.16
+pytest-sugar<2,>=1.0.0
+ruff<1,>=0.8.1
+numpy<3,>=2.1.2

grz_cli-0.1.0/src/grz_cli.egg-info/requires.txt

@@ -1,17 +0,0 @@
-boto3~=1.35.39
-click~=8.1.7
-crypt4gh~=1.7
-jsonschema~=4.23.0
-pyyaml~=6.0.2
-tqdm~=4.66.5
-pydantic~=2.9.2
-platformdirs~=4.3.6
-
-[test]
-pytest~=8.3.3
-pytest-cov~=5.0.0
-pytest-mock~=3.14.0
-moto[s3]~=5.0.16
-pytest-sugar~=1.0.0
-ruff~=0.7.0
-numpy~=2.1.2