metatrawl 0.1.0__tar.gz

metatrawl-0.1.0/PKG-INFO

@@ -0,0 +1,220 @@
+Metadata-Version: 2.4
+Name: metatrawl
+Version: 0.1.0
+Summary: Mutable registry and workflow bookkeeping for large ZipStrain/SRA profiling projects.
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: click>=8.1
+Requires-Dist: duckdb>=1.1
+Requires-Dist: polars>=1.0
+Requires-Dist: rich>=13.0
+Provides-Extra: test
+Requires-Dist: pytest>=8.0; extra == "test"
+
+# MetaTrawl
+
+MetaTrawl is a mutable DuckDB project store for SRA-scale ZipStrain projects.
+It tracks run IDs, imports completed ZipStrain/Sylph outputs into real database
+tables, coordinates shared genome cache preparation, and builds ZipStrain matrix
+stores from selected samples.
+
+The core idea is simple: many SRA workers can run in parallel, but one cache
+owner prepares genome and Prodigal outputs for shared accessions. Workers create
+per-sample concatenated references in scratch space, profile the sample, import
+the final tables into DuckDB, and then delete scratch files.
+
+## Install For Development
+
+```bash
+pip install -e ".[test]"
+```
+
+MetaTrawl checks for the external tools used by the full workflow:
+`zipstrain`, `sylph`, `samtools`, `bowtie2`, `prefetch`, `fasterq-dump`,
+`datasets`, and `prodigal`.
+
+```bash
+metatrawl test
+```
+
+Use the strict checker before long jobs. It exits non-zero if anything required
+is missing:
+
+```bash
+metatrawl check
+```
+
+## Database Workflow
+
+Initialize a project database:
+
+```bash
+metatrawl init --db metatrawl.duckdb
+```
+
+Add SRA run IDs:
+
+```bash
+metatrawl runs add --db metatrawl.duckdb SRR000001 SRR000002
+```
+
+Export only runs that are not yet complete:
+
+```bash
+metatrawl profiles remaining \
+  --db metatrawl.duckdb \
+  --output-file remaining_runs.csv
+```
+
+The CSV contains one column:
+
+```csv
+run_id
+SRR000001
+SRR000002
+```
+
+Run the high-level sync. This gets remaining runs from DuckDB, runs the SRA
+profiling lifecycle, finds completed profile outputs, imports them into DuckDB,
+deletes the imported per-sample files, and logs each step:
+
+```bash
+metatrawl sync \
+  --db metatrawl.duckdb \
+  --cache-dir cache \
+  --scratch-dir scratch \
+  --output-dir outputs \
+  --threads 16
+```
+
+`sync` expects per-run outputs in `--output-dir` using these conventional names:
+
+```text
+SRR000001.profile.parquet
+SRR000001.genome_stats.parquet
+SRR000001.gene_stats.parquet      # optional
+SRR000001.sylph.csv               # csv, tsv, or parquet
+```
+
+After a successful import, these per-sample outputs are removed because DuckDB is
+the durable project store. The durable cache is left intact. Use
+`--keep-profile-outputs` only when debugging a failed or suspicious run.
+
+After profiling, import completed outputs into DuckDB tables:
+
+```bash
+metatrawl profiles import \
+  --db metatrawl.duckdb \
+  --run-id SRR000001 \
+  --profile-file outputs/SRR000001.profile.parquet \
+  --genome-stats-file outputs/SRR000001.genome_stats.parquet \
+  --gene-stats-file outputs/SRR000001.gene_stats.parquet \
+  --sylph-abundance-file outputs/SRR000001.sylph.csv
+```
+
+Or import many samples from a manifest:
+
+```bash
+metatrawl profiles add \
+  --db metatrawl.duckdb \
+  --manifest completed_profiles.csv
+```
+
+Manifest columns:
+
+```csv
+run_id,profile_file,genome_stats_file,gene_stats_file,sylph_abundance_file
+SRR000001,/path/profile.parquet,/path/genome_stats.parquet,/path/gene_stats.parquet,/path/sylph.csv
+```
+
+`gene_stats_file` is optional. A run is complete after profile positions, genome
+stats, and Sylph abundance have been imported.
+
+## Cache Workflow
+
+Prepare one sample reference from accessions using a shared cache:
+
+```bash
+metatrawl cache prepare \
+  --cache-dir cache \
+  --accessions accessions.csv \
+  --output-dir scratch/SRR000001/reference
+```
+
+For parallel workers, start a local cache server:
+
+```bash
+metatrawl cache serve \
+  --cache-dir cache \
+  --host 127.0.0.1 \
+  --port 8765
+```
+
+The cache keeps only durable per-accession files:
+
+```text
+cache/genomes/GCF_xxx.fna
+cache/genes/GCF_xxx.genes.fna
+```
+
+Per-sample concatenated references are scratch outputs and should be deleted
+after import.
+
+## SRA Worker Lifecycle
+
+`profile-sra` wires the worker lifecycle around remaining runs and scratch
+cleanup:
+
+```bash
+metatrawl profile-sra \
+  --db metatrawl.duckdb \
+  --remaining-csv remaining_runs.csv \
+  --cache-dir cache \
+  --scratch-dir scratch \
+  --threads 8
+```
+
+Long-running steps emit compact cluster-friendly logs:
+
+```text
+METATRAWL sample=SRR123 step=sylph status=done genomes=12 elapsed=4.2s
+METATRAWL sample=SRR123 step=cache status=done accessions=10 elapsed=28.9s
+METATRAWL sample=SRR123 step=cleanup status=done removed=scratch/SRR123
+```
+
+## Matrix Workflow
+
+Build a ZipStrain matrix from complete DuckDB samples. Thresholds are applied
+before temporary profile parquets are exported:
+
+```bash
+metatrawl matrix build \
+  --db metatrawl.duckdb \
+  --genome GCF_000269965.1_ASM26996v1_genomic.fna \
+  --bed-file reference/genomes.bed \
+  --stb-file reference/genomes.stb \
+  --output-file matrices/binfantis.h5 \
+  --min-coverage 1 \
+  --min-breadth 0.2 \
+  --min-ber 0.77 \
+  --min-sylph-abundance 0.001 \
+  --sparse
+```
+
+Append newly imported complete samples to a registered matrix:
+
+```bash
+metatrawl matrix append \
+  --db metatrawl.duckdb \
+  --matrix-id binfantis
+```
+
+Compare a registered matrix:
+
+```bash
+metatrawl matrix compare \
+  --db metatrawl.duckdb \
+  --matrix-id binfantis \
+  --output-file compares/binfantis.duckdb \
+  --calculate all
+```

metatrawl-0.1.0/README.md

@@ -0,0 +1,207 @@
+# MetaTrawl
+
+MetaTrawl is a mutable DuckDB project store for SRA-scale ZipStrain projects.
+It tracks run IDs, imports completed ZipStrain/Sylph outputs into real database
+tables, coordinates shared genome cache preparation, and builds ZipStrain matrix
+stores from selected samples.
+
+The core idea is simple: many SRA workers can run in parallel, but one cache
+owner prepares genome and Prodigal outputs for shared accessions. Workers create
+per-sample concatenated references in scratch space, profile the sample, import
+the final tables into DuckDB, and then delete scratch files.
+
+## Install For Development
+
+```bash
+pip install -e ".[test]"
+```
+
+MetaTrawl checks for the external tools used by the full workflow:
+`zipstrain`, `sylph`, `samtools`, `bowtie2`, `prefetch`, `fasterq-dump`,
+`datasets`, and `prodigal`.
+
+```bash
+metatrawl test
+```
+
+Use the strict checker before long jobs. It exits non-zero if anything required
+is missing:
+
+```bash
+metatrawl check
+```
+
+## Database Workflow
+
+Initialize a project database:
+
+```bash
+metatrawl init --db metatrawl.duckdb
+```
+
+Add SRA run IDs:
+
+```bash
+metatrawl runs add --db metatrawl.duckdb SRR000001 SRR000002
+```
+
+Export only runs that are not yet complete:
+
+```bash
+metatrawl profiles remaining \
+  --db metatrawl.duckdb \
+  --output-file remaining_runs.csv
+```
+
+The CSV contains one column:
+
+```csv
+run_id
+SRR000001
+SRR000002
+```
+
+Run the high-level sync. This gets remaining runs from DuckDB, runs the SRA
+profiling lifecycle, finds completed profile outputs, imports them into DuckDB,
+deletes the imported per-sample files, and logs each step:
+
+```bash
+metatrawl sync \
+  --db metatrawl.duckdb \
+  --cache-dir cache \
+  --scratch-dir scratch \
+  --output-dir outputs \
+  --threads 16
+```
+
+`sync` expects per-run outputs in `--output-dir` using these conventional names:
+
+```text
+SRR000001.profile.parquet
+SRR000001.genome_stats.parquet
+SRR000001.gene_stats.parquet      # optional
+SRR000001.sylph.csv               # csv, tsv, or parquet
+```
+
+After a successful import, these per-sample outputs are removed because DuckDB is
+the durable project store. The durable cache is left intact. Use
+`--keep-profile-outputs` only when debugging a failed or suspicious run.
+
+After profiling, import completed outputs into DuckDB tables:
+
+```bash
+metatrawl profiles import \
+  --db metatrawl.duckdb \
+  --run-id SRR000001 \
+  --profile-file outputs/SRR000001.profile.parquet \
+  --genome-stats-file outputs/SRR000001.genome_stats.parquet \
+  --gene-stats-file outputs/SRR000001.gene_stats.parquet \
+  --sylph-abundance-file outputs/SRR000001.sylph.csv
+```
+
+Or import many samples from a manifest:
+
+```bash
+metatrawl profiles add \
+  --db metatrawl.duckdb \
+  --manifest completed_profiles.csv
+```
+
+Manifest columns:
+
+```csv
+run_id,profile_file,genome_stats_file,gene_stats_file,sylph_abundance_file
+SRR000001,/path/profile.parquet,/path/genome_stats.parquet,/path/gene_stats.parquet,/path/sylph.csv
+```
+
+`gene_stats_file` is optional. A run is complete after profile positions, genome
+stats, and Sylph abundance have been imported.
+
+## Cache Workflow
+
+Prepare one sample reference from accessions using a shared cache:
+
+```bash
+metatrawl cache prepare \
+  --cache-dir cache \
+  --accessions accessions.csv \
+  --output-dir scratch/SRR000001/reference
+```
+
+For parallel workers, start a local cache server:
+
+```bash
+metatrawl cache serve \
+  --cache-dir cache \
+  --host 127.0.0.1 \
+  --port 8765
+```
+
+The cache keeps only durable per-accession files:
+
+```text
+cache/genomes/GCF_xxx.fna
+cache/genes/GCF_xxx.genes.fna
+```
+
+Per-sample concatenated references are scratch outputs and should be deleted
+after import.
+
+## SRA Worker Lifecycle
+
+`profile-sra` wires the worker lifecycle around remaining runs and scratch
+cleanup:
+
+```bash
+metatrawl profile-sra \
+  --db metatrawl.duckdb \
+  --remaining-csv remaining_runs.csv \
+  --cache-dir cache \
+  --scratch-dir scratch \
+  --threads 8
+```
+
+Long-running steps emit compact cluster-friendly logs:
+
+```text
+METATRAWL sample=SRR123 step=sylph status=done genomes=12 elapsed=4.2s
+METATRAWL sample=SRR123 step=cache status=done accessions=10 elapsed=28.9s
+METATRAWL sample=SRR123 step=cleanup status=done removed=scratch/SRR123
+```
+
+## Matrix Workflow
+
+Build a ZipStrain matrix from complete DuckDB samples. Thresholds are applied
+before temporary profile parquets are exported:
+
+```bash
+metatrawl matrix build \
+  --db metatrawl.duckdb \
+  --genome GCF_000269965.1_ASM26996v1_genomic.fna \
+  --bed-file reference/genomes.bed \
+  --stb-file reference/genomes.stb \
+  --output-file matrices/binfantis.h5 \
+  --min-coverage 1 \
+  --min-breadth 0.2 \
+  --min-ber 0.77 \
+  --min-sylph-abundance 0.001 \
+  --sparse
+```
+
+Append newly imported complete samples to a registered matrix:
+
+```bash
+metatrawl matrix append \
+  --db metatrawl.duckdb \
+  --matrix-id binfantis
+```
+
+Compare a registered matrix:
+
+```bash
+metatrawl matrix compare \
+  --db metatrawl.duckdb \
+  --matrix-id binfantis \
+  --output-file compares/binfantis.duckdb \
+  --calculate all
+```

metatrawl-0.1.0/pyproject.toml

@@ -0,0 +1,27 @@
+[project]
+name = "metatrawl"
+version = "0.1.0"
+description = "Mutable registry and workflow bookkeeping for large ZipStrain/SRA profiling projects."
+readme = "README.md"
+requires-python = ">=3.12"
+dependencies = [
+    "click>=8.1",
+    "duckdb>=1.1",
+    "polars>=1.0",
+    "rich>=13.0",
+]
+
+[project.optional-dependencies]
+test = [
+    "pytest>=8.0",
+]
+
+[project.scripts]
+metatrawl = "metatrawl.cli:cli"
+
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src"]

metatrawl-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

metatrawl-0.1.0/src/metatrawl/__init__.py

@@ -0,0 +1,9 @@
+"""MetaTrawl: registry tooling for SRA-scale ZipStrain profiling projects."""
+
+from importlib.metadata import PackageNotFoundError, version
+
+
+try:
+    __version__ = version("metatrawl")
+except PackageNotFoundError:
+    __version__ = "0.1.0"

metatrawl-0.1.0/src/metatrawl/cache.py

@@ -0,0 +1,204 @@
+"""Genome cache preparation for concurrent MetaTrawl workers."""
+
+from __future__ import annotations
+
+from concurrent.futures import Future, ThreadPoolExecutor
+from dataclasses import dataclass
+from pathlib import Path
+import shutil
+import subprocess
+import threading
+from typing import Callable
+
+import polars as pl
+
+from metatrawl.logging import WorkflowLogger
+
+
+Downloader = Callable[[str, Path], None]
+ProdigalRunner = Callable[[Path, Path], None]
+
+
+@dataclass(frozen=True)
+class PreparedReference:
+    """Per-sample temporary reference files returned by the cache manager."""
+
+    reference_fasta: Path
+    gene_fasta: Path
+
+
+class GenomeCache:
+    """Single-process authoritative writer for genome/prodigal cache files."""
+
+    def __init__(
+        self,
+        cache_dir: Path,
+        *,
+        downloader: Downloader | None = None,
+        prodigal_runner: ProdigalRunner | None = None,
+        logger: WorkflowLogger | None = None,
+    ) -> None:
+        self.cache_dir = Path(cache_dir)
+        self.genomes_dir = self.cache_dir / "genomes"
+        self.genes_dir = self.cache_dir / "genes"
+        self.genomes_dir.mkdir(parents=True, exist_ok=True)
+        self.genes_dir.mkdir(parents=True, exist_ok=True)
+        self.downloader = downloader or download_genome_with_datasets
+        self.prodigal_runner = prodigal_runner or run_prodigal_gene_fasta
+        self.logger = logger or WorkflowLogger()
+        self._executor = ThreadPoolExecutor(max_workers=4)
+        self._lock = threading.Lock()
+        self._in_flight: dict[str, Future[tuple[Path, Path]]] = {}
+
+    def prepare_accession(self, accession: str) -> tuple[Path, Path]:
+        """Ensure one accession has cached genome and gene FASTA files."""
+        accession = accession.strip()
+        if not accession:
+            raise ValueError("Empty accession requested from genome cache.")
+        genome_fasta = self.genome_fasta_path(accession)
+        gene_fasta = self.gene_fasta_path(accession)
+        if genome_fasta.exists() and gene_fasta.exists():
+            self.logger.emit(accession=accession, step="cache", status="cached")
+            return genome_fasta, gene_fasta
+        with self._lock:
+            future = self._in_flight.get(accession)
+            if future is None:
+                self.logger.emit(accession=accession, step="cache", status="queued")
+                future = self._executor.submit(self._prepare_accession_uncached, accession)
+                self._in_flight[accession] = future
+        try:
+            return future.result()
+        finally:
+            if future.done():
+                with self._lock:
+                    self._in_flight.pop(accession, None)
+
+    def prepare_reference(self, *, accessions: list[str], output_dir: Path, sample: str | None = None) -> PreparedReference:
+        """Build per-sample concatenated genome and gene FASTA files."""
+        clean_accessions = _dedupe(accessions)
+        if not clean_accessions:
+            raise ValueError("Cannot prepare a reference with no accessions.")
+        output_dir = Path(output_dir)
+        output_dir.mkdir(parents=True, exist_ok=True)
+        self.logger.emit(sample=sample, step="cache", status="start", accessions=len(clean_accessions))
+        prepared = [self.prepare_accession(accession) for accession in clean_accessions]
+        reference_fasta = output_dir / "reference.fna"
+        gene_fasta = output_dir / "genes.fna"
+        _concatenate_fastas([pair[0] for pair in prepared], reference_fasta)
+        _concatenate_fastas([pair[1] for pair in prepared], gene_fasta)
+        self.logger.emit(sample=sample, step="cache", status="done", accessions=len(clean_accessions))
+        return PreparedReference(reference_fasta=reference_fasta, gene_fasta=gene_fasta)
+
+    def genome_fasta_path(self, accession: str) -> Path:
+        return self.genomes_dir / f"{_safe_name(accession)}.fna"
+
+    def gene_fasta_path(self, accession: str) -> Path:
+        return self.genes_dir / f"{_safe_name(accession)}.genes.fna"
+
+    def _prepare_accession_uncached(self, accession: str) -> tuple[Path, Path]:
+        genome_fasta = self.genome_fasta_path(accession)
+        gene_fasta = self.gene_fasta_path(accession)
+        if not genome_fasta.exists():
+            self.logger.emit(accession=accession, step="download", status="start")
+            tmp_genome = genome_fasta.with_suffix(".tmp.fna")
+            self.downloader(accession, tmp_genome)
+            _atomic_publish(tmp_genome, genome_fasta)
+            self.logger.emit(accession=accession, step="download", status="done", file=genome_fasta)
+        if not gene_fasta.exists():
+            self.logger.emit(accession=accession, step="prodigal", status="start")
+            tmp_gene = gene_fasta.with_suffix(".tmp.fna")
+            self.prodigal_runner(genome_fasta, tmp_gene)
+            _atomic_publish(tmp_gene, gene_fasta)
+            self.logger.emit(accession=accession, step="prodigal", status="done", file=gene_fasta)
+        return genome_fasta, gene_fasta
+
+
+def prepare_cache_reference(
+    *,
+    cache_dir: Path,
+    accessions: list[str],
+    output_dir: Path,
+    logger: WorkflowLogger | None = None,
+) -> PreparedReference:
+    """Prepare a per-sample reference using the default cache manager."""
+    return GenomeCache(cache_dir, logger=logger).prepare_reference(accessions=accessions, output_dir=output_dir)
+
+
+def read_accessions_file(accessions_file: Path) -> list[str]:
+    """Read accessions from a one-column CSV/TSV/text file."""
+    path = Path(accessions_file)
+    if path.suffix.lower() == ".csv":
+        df = pl.read_csv(path)
+        column = "accession" if "accession" in df.columns else df.columns[0]
+        return [str(value) for value in df[column].to_list()]
+    return [line.strip().split()[0] for line in path.read_text().splitlines() if line.strip()]
+
+
+def download_genome_with_datasets(accession: str, output_fasta: Path) -> None:
+    """Download a genome FASTA with NCBI datasets CLI when available."""
+    output_fasta.parent.mkdir(parents=True, exist_ok=True)
+    archive = output_fasta.with_suffix(".zip")
+    tmp_dir = output_fasta.parent / f"{output_fasta.stem}.datasets"
+    try:
+        subprocess.run(
+            ["datasets", "download", "genome", "accession", accession, "--filename", str(archive)],
+            check=True,
+            capture_output=True,
+            text=True,
+        )
+        shutil.unpack_archive(str(archive), str(tmp_dir))
+        fasta_files = sorted(tmp_dir.rglob("*.fna"))
+        if not fasta_files:
+            raise RuntimeError(f"datasets did not produce a FASTA for accession {accession}")
+        shutil.copy2(fasta_files[0], output_fasta)
+    finally:
+        if archive.exists():
+            archive.unlink()
+        if tmp_dir.exists():
+            shutil.rmtree(tmp_dir)
+
+
+def run_prodigal_gene_fasta(genome_fasta: Path, output_gene_fasta: Path) -> None:
+    """Run Prodigal and keep only nucleotide gene FASTA output."""
+    output_gene_fasta.parent.mkdir(parents=True, exist_ok=True)
+    proteins = output_gene_fasta.with_suffix(".faa")
+    genes = output_gene_fasta
+    subprocess.run(
+        ["prodigal", "-i", str(genome_fasta), "-d", str(genes), "-a", str(proteins), "-p", "meta", "-q"],
+        check=True,
+        capture_output=True,
+        text=True,
+    )
+    if proteins.exists():
+        proteins.unlink()
+
+
+def _concatenate_fastas(inputs: list[Path], output: Path) -> None:
+    tmp = output.with_suffix(output.suffix + ".tmp")
+    with tmp.open("w") as out:
+        for fasta in inputs:
+            out.write(Path(fasta).read_text())
+            if not out.tell():
+                continue
+            out.write("\n")
+    _atomic_publish(tmp, output)
+
+
+def _atomic_publish(tmp: Path, final: Path) -> None:
+    final.parent.mkdir(parents=True, exist_ok=True)
+    tmp.replace(final)
+
+
+def _safe_name(value: str) -> str:
+    return "".join(char if char.isalnum() or char in "._-" else "_" for char in value)
+
+
+def _dedupe(values: list[str]) -> list[str]:
+    seen: set[str] = set()
+    result: list[str] = []
+    for value in values:
+        clean = value.strip()
+        if clean and clean not in seen:
+            seen.add(clean)
+            result.append(clean)
+    return result