bioflowkit 0.2.1__tar.gz → 0.3.0__tar.gz

bioflowkit-0.3.0/.gitattributes

@@ -0,0 +1,20 @@
+# Test fixtures are read by Linux containers (FastANI, CAFE5, …).  Some
+# tool parsers don't strip a stray CR, so a CRLF checkout on Windows
+# silently corrupts them (CAFE5 read the last column header as "D\r" and
+# reported "D was not found").  Force LF for text fixtures everywhere,
+# and mark the compressed reads binary so git never touches them.
+data/test/**/*.fna   text eol=lf
+data/test/**/*.fa    text eol=lf
+data/test/**/*.fasta text eol=lf
+data/test/**/*.tsv   text eol=lf
+data/test/**/*.csv   text eol=lf
+data/test/**/*.nwk   text eol=lf
+data/test/**/*.txt   text eol=lf
+data/test/**/*.gff   text eol=lf
+data/test/**/*.ffn   text eol=lf
+data/test/**/*.gz    binary
+
+# Registry + recipe shell commands also end up inside Linux containers;
+# keep them LF regardless of host.
+*.yaml text eol=lf
+registry/** text eol=lf

{bioflowkit-0.2.1 → bioflowkit-0.3.0}/.github/workflows/nightly-smoke.yml

@@ -48,12 +48,20 @@ jobs:
             -v -m docker \
             --junitxml=reports/smoke.xml
 
-      - name: Upload junit report
+      - name: Run full-pipeline e2e (all 9 recipes with committed fixtures)
+        env:
+          BIOFLOW_LOG_LEVEL: INFO
+        run: |
+          python -m pytest tests/integration/test_full_pipeline_e2e.py \
+            -v -m docker \
+            --junitxml=reports/full_e2e.xml
+
+      - name: Upload junit reports
         if: always()
         uses: actions/upload-artifact@v4
         with:
           name: smoke-junit
-          path: reports/smoke.xml
+          path: reports/*.xml
 
       - name: Fail the job on red results
         if: failure()

{bioflowkit-0.2.1 → bioflowkit-0.3.0}/.gitignore

@@ -98,3 +98,4 @@ build/
 
 # MkDocs build output
 site/
+data/test/**/_e2e_out/

{bioflowkit-0.2.1 → bioflowkit-0.3.0}/CHANGELOG.md

@@ -12,6 +12,208 @@ ship bug fixes only.  Breaking changes to the documented public API
 
 ---
 
+## [Unreleased]
+
+## [0.3.0] — 2026-06-18
+
+### Fixed — aligner images had no samtools (5 recipes broke at alignment)
+The plain single-tool aligner BioContainers (`bwa`, `bowtie2`,
+`minimap2`) do **not** bundle samtools, yet five recipes ran
+`aligner | samtools sort` / `samtools index` inside them — so every one
+failed at its alignment stage with `samtools: command not found`.  The
+smoke matrix only exercises each recipe's *first* stage, so this stayed
+invisible.  (See #2.)  Each aligner stage now uses an image that carries
+both tools:
+- **germline_variants**, **joint_genotyping**: a mulled
+  `bwa 0.7.19 + samtools 1.22` image.  A new **`prepare_reference`**
+  stage builds the BWA index + `.fai` + `.dict` **once** (in the cohort
+  recipe, before the per-sample fan-out — previously each parallel
+  sample raced to index the *shared* reference), and the gatk call stage
+  drops its own `samtools` calls (`MarkDuplicates --CREATE_INDEX`
+  indexes the dedup BAM; the gatk4 image ships no samtools either).
+  Verified end-to-end on phiX: prep → bwa-mem → sort/index →
+  MarkDuplicates → HaplotypeCaller now produces a valid VCF.
+- **chip_seq**, **atac_seq**: `staphb/bowtie2:2.5.4` (bundles samtools).
+- **metagenome_assembly**: a mulled `minimap2 2.31 + samtools 1.23` image.
+- `registry/tools/alignment/bowtie2.yaml` had the same broken
+  assumption (and `samtools.yaml` documented it as fact) — both fixed.
+
+### Fixed — static audit of the never-e2e'd recipes (round 1)
+A static pass over the 10 recipes that have no committed full-e2e fixture
+(they need external reference data) turned up several latent defects:
+- **proteomics_dda**: the percolator FDR cut used ``awk -F\t``, whose
+  backslash bash strips before awk sees it — so the field separator
+  became the literal letter ``t`` instead of a tab, and
+  ``passing_psms.tsv`` was filtered on garbage columns.  Now ``-F"\t"``,
+  which reaches awk as a real tab (verified).
+- **eukaryote_assembly**: the docstring advertised ``polish=False`` to
+  skip Medaka for HiFi reads, but no such parameter existed.  Added it
+  (the ``polish`` stage is renamed ``polish_consensus`` to free the
+  name); ``assess`` already falls back to Flye's ``assembly.fasta``.
+- **chip_seq**: docstring promised ``--ctrl-r1 / --ctrl-r2`` raw-control
+  alignment the recipe never implemented — corrected to document the
+  actual ``--ctrl-bam`` (pre-aligned control) input.
+- **metagenomics_profile**: ``bioflow db fetch`` example used a
+  catalog key that doesn't exist (``kraken2_standard`` →
+  ``kraken2_standard_8gb``) and now notes Bracken's ``kmer_distrib`` need.
+
+### Docs — strict build fixed + e2e-coverage page
+- `mkdocs build --strict` was aborting: three docs links pointed at repo
+  files outside the `docs/` tree (`conda-recipe/meta.yaml`,
+  `scripts/compare_nfcore.py`, the nf-core workflow), which strict mode
+  flags as unresolved.  Re-pointed them at GitHub blob URLs.
+- New **`reference/e2e-coverage.md`** documents which 9 recipes have a
+  committed full end-to-end fixture and which 10 are gated on external
+  reference data (with the `bioflow db fetch` key for each), plus the one
+  utility recipe.  Regenerated `reference/recipes.md` / `tools.md` from
+  the registry (now 20 recipes; `joint_genotyping` was missing and image
+  tags were stale).
+
+### Fixed — ani_matrix broken for genomes outside the workspace
+- **Bug a full e2e caught**: FastANI reads genome paths from a *list
+  file*, not the command, so the SDK's command-path translator and
+  auto-mount never applied to them — every external genome failed with
+  `Could not open <host path>`.  Since genomes normally live outside the
+  output workspace, this broke the recipe's primary documented use.
+- New SDK helper **`stage_input(path)`** copies an external file into the
+  active workspace (always mounted at `/work`) and returns its container
+  path — the clean primitive for any recipe that feeds a tool a list
+  file of paths.  Also exported **`container_path(path)`**.
+- `ani_matrix` now stages genomes via `stage_input` and writes container
+  paths into the FastANI list; verified end-to-end (genome1 vs genome2 =
+  99.5% ANI).
+
+### Added — full e2e for the comparative-genomics recipes
+- `tests/integration/test_full_pipeline_e2e.py` gains real end-to-end
+  tests for **amr_vf_catalogue** (ABRicate fan-out, bundled DBs),
+  **ani_matrix** (all-vs-all FastANI), **pangenome** (Prokka × N →
+  Roary), and **gwas** (Scoary on a synthetic Roary GPA + phenotype,
+  recovers a planted association).  Fixtures:
+  `data/test/genomes_small/` (phiX174 + a 25-SNP variant) and
+  `data/test/gwas_small/` (12-gene × 10-sample GPA).  Recipes validated
+  end-to-end: 1 (prokaryote) → 9 (see below).
+- **cafe_evolution** (CAFE5 gene-family expansion/contraction) added as
+  the 6th, on `data/test/cafe_small/` (ultrametric 4-taxon tree + 60
+  families).
+- **phylogeny** (single-copy core → MAFFT × N → IQ-TREE) added as the
+  7th, on `data/test/phylo_small/` (Prokka GFF + CDS + Roary GPA for 4
+  phiX strains; IQ-TREE recovers a 4-taxon ML tree).
+- **rnaseq_deg** (fastp → Salmon → DESeq2 → enrichment + MultiQC) added
+  as the 8th, on `data/test/rnaseq_small/` (60 synthetic transcripts, 4
+  samples, 10 transcripts planted ~4× up in the treated group).  DESeq2
+  recovers the planted signal (`tx0001` log2FC ≈ 2) and the run finishes
+  in seconds.  The sample sheet is built by the test at run time so no
+  machine-specific paths are committed.
+- **methylation_wgbs** (TrimGalore → Bismark → methylKit) added as the
+  9th, on `data/test/methyl_small/` (phiX174 + 3,000 synthetic
+  directional bisulfite read pairs, ~70 % CpG-methylated).  The reads
+  map at 100 % and Bismark produces a real cytosine report; the genome
+  is **not** committed pre-prepared — the new `bismark_prep` stage
+  (below) bisulfite-converts it at run time, so no version-tied bowtie2
+  index lands in git.  Regenerated deterministically by
+  `scripts/gen_methyl_fixture.py`.
+
+### Added — methylation_wgbs prepares its genome (matches the docs)
+- The recipe's docstring promised automatic genome preparation, but the
+  pipeline had no such stage — it silently required a pre-prepared
+  `Bisulfite_Genome/` directory, so running from a plain reference FASTA
+  failed.  A new **`bismark_prep`** stage now runs
+  `bismark_genome_preparation` when `--bismark-genome` is a FASTA (or a
+  directory holding one); an already-prepared directory is detected and
+  used directly, skipping preparation.  `methylation_wgbs` is now 4
+  stages (trim → bismark_prep → bismark → methylkit).
+
+### Fixed — methylKit CpG-report glob (shell ate the regex escape)
+- `methylkit_dmr` matched the cytosine report with `pattern='…txt(\.gz)?$'`,
+  but the `\.` escape was stripped by the shell before R parsed the
+  string, so R aborted with *"'\.' is an unrecognized escape in character
+  string"* and the whole recipe failed at the final stage.  Replaced with
+  a `[.]` character class (no backslash to escape), which matches the
+  report — with or without a `.gz` suffix — robustly.
+
+### Fixed — rnaseq_deg DESeq2 step (two latent bugs)
+- The `deseq2_diff` stage required **tximport**, which the
+  `bioconductor-deseq2` BioContainer does not ship — every run failed
+  with "there is no package called 'tximport'".  Rewritten to assemble
+  the count matrix in base R straight from each sample's `quant.sf`
+  (`NumReads`, rounded) and feed `DESeqDataSetFromMatrix`, dropping the
+  tximport dependency entirely.
+- A second, masked bug: `samples$sample_id` inside the `Rscript -e "…"`
+  body (run via `sh -c`) was shell-expanded because the `$` was
+  unescaped, so the `file.path(...)` of `quant.sf` paths was wrong.
+  Escaped to `\$sample_id`.  The pipeline now also **fails fast** if the
+  DESeq2 stage exits non-zero (the downstream Enrichr step tolerates an
+  empty gene list, which previously masked a broken DEG table).
+
+### Fixed — LF line endings for container-read fixtures
+- A new `.gitattributes` pins text test fixtures (and registry YAMLs) to
+  LF.  CAFE5 doesn't strip a trailing CR, so a CRLF checkout on Windows
+  made it read the last species column as `D\r` and fail with "D was not
+  found in gene family …".  FASTA parsers tolerated the CR, but the
+  table parser did not — LF is now enforced so fixtures are safe on
+  every host.
+
+### Fixed — bounded stdout retention (no orchestrator OOM on chatty tools)
+- `DockerBackend.run` accumulated **every** stdout line in memory; a
+  tool that emits millions of lines (Roary, IQ-TREE) could OOM the
+  orchestrator.  It now retains only the trailing `_STDOUT_TAIL_LINES`
+  (5000) via a bounded `deque` for the diagnostic `CommandResult.stdout`
+  — every line still streams live to `log_callback`, and real artifacts
+  go to files in the workspace.
+- Tests: +1 (`test_docker_timeout.py`) asserting the tail is kept.
+
+### Fixed — clear error for shell-unsafe external input filenames
+- An external input file whose **basename** contained a space or shell
+  metacharacter silently corrupted the recipe's command — bioflow mounts
+  the file's parent at the space-free `/inputs/<n>` and splices the
+  basename in unquoted, and it can't be quoted generically because many
+  recipes wrap the whole command in `bash -c '…'`.  (A spaced *directory*
+  was already fine — only the basename survives into the command.)
+- `_collect_external_mounts` now raises an actionable `ValueError`
+  naming the offending characters and telling the user to rename /
+  symlink to a safe name.
+- Tests: +12 (`tests/unit/test_unsafe_paths.py`), incl. confirmation
+  that spaced *directories* and workspace-internal paths are unaffected.
+
+### Fixed — stage_timeout now actually bounds runtime
+- **Latent bug**: `run_plan(stage_timeout=…)` never worked.  The log
+  loop (`container.logs(stream=True, follow=True)`) blocks until the
+  container exits, so the subsequent `container.wait(timeout=…)` — a
+  docker-py HTTP read timeout, not a runtime cap — could never fire for
+  a runaway container; it would hang forever.
+- `DockerBackend.run` now starts a watchdog `threading.Timer` that
+  `container.kill()`s the stage when the timeout elapses, returning the
+  conventional exit code **124** with a clear message.
+- Tests: +3 (`tests/unit/test_docker_timeout.py`, fake-container based,
+  deterministic).
+
+### Fixed — DockerBackend now clamps CPU/RAM to host capacity
+- **Bug the full-pipeline e2e caught**: a stage declaring `cpu=8` (e.g.
+  SPAdes in `prokaryote_assembly`) failed to even *start* on any host
+  with fewer cores — Docker rejects a container whose `--cpus` exceeds
+  the host count ("range of CPUs is from 0.01 to N.00"), so all 3 retry
+  attempts died instantly.  Passed locally (12 cores) but failed on the
+  4-core CI runner — and would hit any user on a small workstation.
+- `DockerBackend.run` now clamps the requested CPU to the host core count
+  and RAM to ~90% of host memory (`_clamp_resources`), so an
+  over-ambitious resource request degrades to "use what's available"
+  instead of crashing.
+- Tests: +5 (`tests/unit/test_resource_clamp.py`).
+
+### Added — first full-pipeline end-to-end test
+- `tests/integration/test_full_pipeline_e2e.py`: runs the **entire**
+  `prokaryote_assembly` recipe (fastp → SPAdes → QUAST → Prokka) against
+  real BioContainers — the first time a complete pipeline (not just a
+  first stage, as the smoke matrix does) is validated end-to-end.
+- New `data/test/phix_small/` fixture: phiX174 (`NC_001422.1`, 5386 bp) +
+  1000 wgsim-simulated 150 bp pairs (~56×, seed 42 → deterministic).
+  phiX assembles into a single ~5.4 kb contig in <1 min, so the whole
+  chain finishes in ~45 s.
+- Asserts real data flow: assembled contig length 4.5–6 kb, QUAST
+  `report.tsv`, and Prokka annotation with ≥1 CDS.  Verified locally
+  (phiX → 1 contig 5377 bp, 6 CDS).
+- Wired into the nightly-smoke workflow as a second step.
+
 ## [0.2.1] — 2026-06-12
 
 > **Why upgrade from 0.2.0**: the `v0.2.0` tag predated the registry

{bioflowkit-0.2.1 → bioflowkit-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: bioflowkit
-Version: 0.2.1
+Version: 0.3.0
 Summary: bioflow: one-line comparative-genomics recipes + Tier-A SDK, orchestrated over per-tool Docker BioContainers.
 Project-URL: Homepage, https://github.com/hope9901/bioflow
 Project-URL: Repository, https://github.com/hope9901/bioflow
@@ -35,7 +35,7 @@ Description-Content-Type: text/markdown
 [![PyPI](https://img.shields.io/pypi/v/bioflowkit.svg)](https://pypi.org/project/bioflowkit/)
 [![Downloads](https://img.shields.io/pypi/dm/bioflowkit.svg)](https://pypi.org/project/bioflowkit/)
 [![python](https://img.shields.io/pypi/pyversions/bioflowkit.svg)](https://pypi.org/project/bioflowkit/)
-[![tests](https://img.shields.io/badge/tests-609%20passed-brightgreen)](tests/)
+[![tests](https://img.shields.io/badge/tests-631%20passed-brightgreen)](tests/)
 [![nightly smoke](https://github.com/hope9901/bioflow/actions/workflows/nightly-smoke.yml/badge.svg)](https://github.com/hope9901/bioflow/actions/workflows/nightly-smoke.yml)
 [![license](https://img.shields.io/badge/license-MIT-lightgrey)](LICENSE)
 [![docs](https://img.shields.io/badge/docs-mkdocs-blue)](https://hope9901.github.io/bioflow/)

{bioflowkit-0.2.1 → bioflowkit-0.3.0}/README.md

@@ -3,7 +3,7 @@
 [![PyPI](https://img.shields.io/pypi/v/bioflowkit.svg)](https://pypi.org/project/bioflowkit/)
 [![Downloads](https://img.shields.io/pypi/dm/bioflowkit.svg)](https://pypi.org/project/bioflowkit/)
 [![python](https://img.shields.io/pypi/pyversions/bioflowkit.svg)](https://pypi.org/project/bioflowkit/)
-[![tests](https://img.shields.io/badge/tests-609%20passed-brightgreen)](tests/)
+[![tests](https://img.shields.io/badge/tests-631%20passed-brightgreen)](tests/)
 [![nightly smoke](https://github.com/hope9901/bioflow/actions/workflows/nightly-smoke.yml/badge.svg)](https://github.com/hope9901/bioflow/actions/workflows/nightly-smoke.yml)
 [![license](https://img.shields.io/badge/license-MIT-lightgrey)](LICENSE)
 [![docs](https://img.shields.io/badge/docs-mkdocs-blue)](https://hope9901.github.io/bioflow/)

{bioflowkit-0.2.1 → bioflowkit-0.3.0}/bioflow/__init__.py

@@ -1,6 +1,6 @@
 """bioflow - bioinformatics pipeline platform."""
 
-__version__ = "0.2.1"
+__version__ = "0.3.0"
 
 # Tier-A SDK — @stage / @pipeline / runtime config
 from bioflow.sdk import (  # noqa: E402,F401
@@ -16,6 +16,8 @@ from bioflow.sdk import (  # noqa: E402,F401
     clear_cache,
     set_log_streaming,
     is_log_streaming_enabled,
+    container_path,
+    stage_input,
     MockBackend,
     DockerBackend,
 )

{bioflowkit-0.2.1 → bioflowkit-0.3.0}/bioflow/core/runner.py

@@ -25,6 +25,7 @@ Log streaming
 from __future__ import annotations
 
 import math
+import os
 from dataclasses import dataclass
 from pathlib import Path
 from typing import TYPE_CHECKING, Callable, Optional, Protocol, runtime_checkable
@@ -115,6 +116,37 @@ class MockBackend:
         return CommandResult(exit_code=0)
 
 
+# How many trailing stdout lines a stage retains in memory for error
+# reporting.  Live output still streams in full to log_callback; only the
+# in-memory copy returned in CommandResult is capped, so a tool that emits
+# millions of lines can't OOM the orchestrator.
+_STDOUT_TAIL_LINES = 5000
+
+
+def _clamp_resources(cpu: int, ram_gb: float) -> "tuple[int, float]":
+    """Clamp a stage's requested CPU / RAM to the host's capacity.
+
+    Docker refuses to create a container whose ``--cpus`` exceeds the host
+    core count, so a stage declaring ``cpu=8`` must not be sent verbatim to
+    a 4-core host — it would fail to start.  Memory is clamped too (a
+    ``mem_limit`` above host RAM is meaningless and risks an instant OOM).
+    Both floor at 1 so a container is always launchable.
+    """
+    host_cpu = os.cpu_count() or 1
+    eff_cpu = max(1, min(int(cpu), host_cpu))
+
+    eff_ram = ram_gb
+    try:
+        import psutil  # noqa: PLC0415
+
+        host_ram_gb = psutil.virtual_memory().total / (1024 ** 3)
+        # Leave a little headroom for the host/OS.
+        eff_ram = max(1.0, min(float(ram_gb), host_ram_gb * 0.9))
+    except Exception:
+        eff_ram = max(1.0, float(ram_gb))
+    return eff_cpu, eff_ram
+
+
 # ---------------------------------------------------------------------------
 # Docker backend (production)
 # ---------------------------------------------------------------------------
@@ -146,8 +178,6 @@ class DockerBackend:
     _STREAMING_SUPPORTED = True    # sentinel for run_plan
 
     def __init__(self, base_url: Optional[str] = None) -> None:
-        import os  # noqa: PLC0415
-
         import docker  # type: ignore[import-not-found]
 
         url = (
@@ -189,36 +219,85 @@ class DockerBackend:
             dr = self._gpu_device_requests()
             if dr is not None:
                 extra["device_requests"] = dr
+
+        # Clamp the requested CPU/RAM to what the host actually has.  Docker
+        # *rejects* a container whose --cpus exceeds the host core count
+        # ("range of CPUs is from 0.01 to N.00"), so a stage declaring
+        # cpu=8 would fail to even start on a 4-core CI runner or a small
+        # workstation.  A request larger than the host should degrade to
+        # "use everything available", not crash.
+        eff_cpu, eff_ram = _clamp_resources(cpu, ram_gb)
+
         container = None
+        timer = None
+        timed_out = {"flag": False}
         try:
             container = self.client.containers.run(
                 image=image,
                 command=["sh", "-c", command],
                 volumes=volumes,
                 working_dir=workdir,
-                mem_limit=f"{max(math.ceil(ram_gb), 1)}g",
-                nano_cpus=int(cpu * 1_000_000_000),
+                mem_limit=f"{max(math.ceil(eff_ram), 1)}g",
+                nano_cpus=int(eff_cpu * 1_000_000_000),
                 detach=True,
                 remove=False,
                 **extra,
             )
 
-            stdout_lines: list[str] = []
+            # Enforce stage_timeout with a watchdog.  The log-streaming loop
+            # below blocks until the container exits, so ``container.wait
+            # (timeout=…)`` (a docker-py HTTP read timeout, not a runtime
+            # cap) can never fire for a runaway container — a timer that
+            # kills the container is the only thing that actually bounds
+            # the runtime.
+            if timeout is not None and timeout > 0:
+                import threading  # noqa: PLC0415
+
+                def _kill() -> None:
+                    timed_out["flag"] = True
+                    try:
+                        container.kill()
+                    except Exception:
+                        pass
+
+                timer = threading.Timer(timeout, _kill)
+                timer.daemon = True
+                timer.start()
+
+            # Retain only the tail in memory: the returned stdout is used
+            # for error diagnosis, not as the artifact (tools write their
+            # real output to files in the workspace).  A chatty tool
+            # (Roary, IQ-TREE) can emit millions of lines, which would
+            # OOM the orchestrator if kept in full.  Every line is still
+            # streamed live to log_callback.
+            from collections import deque  # noqa: PLC0415
+
+            stdout_lines: "deque[str]" = deque(maxlen=_STDOUT_TAIL_LINES)
             for chunk in container.logs(stream=True, follow=True):
                 line = chunk.decode(errors="replace").rstrip("\n")
                 stdout_lines.append(line)
                 if log_callback:
                     log_callback(line)
 
-            result = container.wait(timeout=timeout)
+            result = container.wait()
+            if timer is not None:
+                timer.cancel()
             container.remove(force=True)
 
+            if timed_out["flag"]:
+                return CommandResult(
+                    exit_code=124,   # conventional timeout exit code
+                    stdout="\n".join(stdout_lines),
+                    stderr=f"stage exceeded timeout of {timeout}s and was killed",
+                )
             return CommandResult(
                 exit_code=int(result.get("StatusCode", 1)),
                 stdout="\n".join(stdout_lines),
             )
         except Exception as exc:
-            # If timeout expired, the container may still be running — remove it
+            if timer is not None:
+                timer.cancel()
+            # On any error the container may still be running — remove it.
             if container is not None:
                 try:
                     container.remove(force=True)

{bioflowkit-0.2.1 → bioflowkit-0.3.0}/bioflow/recipes/comparative_genomics/ani_matrix.py

@@ -10,7 +10,7 @@ from __future__ import annotations
 from pathlib import Path
 from typing import Iterable, Optional
 
-from bioflow import stage, pipeline
+from bioflow import pipeline, stage, stage_input
 from bioflow.io import write_text
 from bioflow.recipes import register
 
@@ -54,11 +54,15 @@ def ani_matrix(
     if not fnas:
         raise RuntimeError("No genomes given to ani_matrix")
 
+    # FastANI reads genome paths from the list FILE, not the command, so
+    # the SDK's command-path translator + auto-mount don't apply to them.
+    # stage_input() copies each genome into the workspace (always mounted
+    # at /work) and returns the container path to write into the list,
+    # which FastANI (working dir /work) can then open.
+    container_paths = [stage_input(g, subdir="ani_genomes") for g in fnas]
+
     list_path = out_dir / "genome_list.txt"
-    write_text(
-        list_path,
-        "\n".join(str(p) for p in fnas) + "\n",
-    )
+    write_text(list_path, "\n".join(container_paths) + "\n")
     return fastani_all_vs_all(list_path)
 
 

{bioflowkit-0.2.1 → bioflowkit-0.3.0}/bioflow/recipes/epigenomics/atac_seq.py

@@ -36,13 +36,15 @@ def trim(r1: Path, r2: Path, *, out_dir):
     return f"trim_galore --paired --cores 4 --output_dir {out_dir} {r1} {r2}"
 
 
-@stage(image="quay.io/biocontainers/bowtie2:2.5.4--he96a11b_7",
+@stage(image="staphb/bowtie2:2.5.4",
        cpu=8, ram_gb=16, depends_on=trim)
 def align(clean, bowtie2_index: Path, sample_id: str, *, out_dir):
     """Bowtie2 align with -X 2000 (ATAC-seq fragment max), sort + index.
 
-    Trimmed-read filenames are resolved at runtime via ``ls | head -1``
-    so the recipe is robust to TrimGalore's naming conventions.
+    Uses the StaPH-B bowtie2 image, which bundles samtools (the plain
+    ``biocontainers/bowtie2`` image does not).  Trimmed-read filenames
+    are resolved at runtime via ``ls | head -1`` so the recipe is robust
+    to TrimGalore's naming conventions.
     """
     return (
         f"bash -c '"

{bioflowkit-0.2.1 → bioflowkit-0.3.0}/bioflow/recipes/epigenomics/chip_seq.py

@@ -7,9 +7,10 @@ End-to-end workflow:
         → MACS3 callpeak (narrow peaks vs. input control)
         → HOMER annotatePeaks + findMotifsGenome (annotation + motifs)
 
-Optional input/IgG control: pass --ctrl-r1 / --ctrl-r2 to mirror the
-sample arm; control reads go through the same Bowtie2 + Picard chain
-and the resulting BAM is handed to MACS3 as ``-c``.
+Optional input/IgG control: pass ``--ctrl-bam`` pointing at a
+pre-aligned, deduplicated control BAM; MACS3 receives it as ``-c``.
+(Raw control reads are not aligned by this recipe — run the sample arm
+on the control FASTQs separately, or use the BAM you already have.)
 
 Researcher (Tier B) usage::
 
@@ -37,14 +38,16 @@ def trim(r1: Path, r2: Path, *, out_dir):
     return f"trim_galore --paired --cores 4 --output_dir {out_dir} {r1} {r2}"
 
 
-@stage(image="quay.io/biocontainers/bowtie2:2.5.4--he96a11b_7",
+@stage(image="staphb/bowtie2:2.5.4",
        cpu=8, ram_gb=16, depends_on=trim)
 def align(clean, bowtie2_index: Path, sample_id: str, *, out_dir):
-    """Bowtie2 alignment → sorted, indexed BAM (samtools chained in image).
+    """Bowtie2 alignment → sorted, indexed BAM.
 
-    The trimmed-read filenames are resolved at runtime (``ls | head -1``)
-    so the recipe survives variations in TrimGalore's naming and never
-    feeds a glob with multiple matches to ``bowtie2 -1``.
+    Uses the StaPH-B bowtie2 image, which bundles samtools (the plain
+    ``biocontainers/bowtie2`` image does not, so the sort/index chain
+    would fail there).  Trimmed-read filenames are resolved at runtime
+    (``ls | head -1``) so the recipe survives TrimGalore's naming and
+    never feeds a glob with multiple matches to ``bowtie2 -1``.
     """
     return (
         f"bash -c '"

{bioflowkit-0.2.1 → bioflowkit-0.3.0}/bioflow/recipes/genome_assembly/eukaryote_assembly.py

@@ -7,7 +7,7 @@ End-to-end ONT/HiFi workflow using the long-read tools added in 0.1.10:
         → compleasm + gfastats (assembly QC)
 
 For HiFi reads, polishing is usually unnecessary — pass
-``polish=False`` to skip the Medaka step.
+``--polish false`` (``polish=False``) to skip the Medaka step.
 
 Researcher (Tier B) usage::
 
@@ -46,7 +46,7 @@ def assemble(qc, long_reads: Path, *, out_dir, read_mode: str = "--nano-hq"):
 
 @stage(image="quay.io/biocontainers/medaka:1.11.3--py39h05d5c5e_0",
        cpu=8, ram_gb=32, depends_on=assemble)
-def polish(asm, long_reads: Path, *, out_dir, medaka_model: str = "r1041_e82_400bps_sup_v5.0.0"):
+def polish_consensus(asm, long_reads: Path, *, out_dir, medaka_model: str = "r1041_e82_400bps_sup_v5.0.0"):
     """Medaka ONT consensus polish of the Flye assembly."""
     return (
         f"medaka_consensus -i {long_reads} "
@@ -56,7 +56,7 @@ def polish(asm, long_reads: Path, *, out_dir, medaka_model: str = "r1041_e82_400
 
 
 @stage(image="quay.io/biocontainers/compleasm:0.2.6--pyh7cba7a3_0",
-       cpu=8, ram_gb=16, depends_on=polish)
+       cpu=8, ram_gb=16, depends_on=polish_consensus)
 def assess(polished, *, out_dir, busco_lineage: str = "eukaryota_odb10",
            busco_db: Path = Path("/refs/busco")):
     """compleasm completeness + gfastats contiguity (gfastats chained)."""
@@ -72,7 +72,7 @@ def assess(polished, *, out_dir, busco_lineage: str = "eukaryota_odb10",
 # ── Pipeline ────────────────────────────────────────────────────────────────
 
 @pipeline(
-    stages=[read_qc, assemble, polish, assess],
+    stages=[read_qc, assemble, polish_consensus, assess],
     description="Eukaryote long-read assembly: NanoPlot → Flye → Medaka → compleasm",
 )
 def eukaryote_assembly(
@@ -80,19 +80,26 @@ def eukaryote_assembly(
     *,
     out_dir: Path,
     read_mode: str = "--nano-hq",
+    polish: bool = True,
     medaka_model: str = "r1041_e82_400bps_sup_v5.0.0",
     busco_lineage: str = "eukaryota_odb10",
     busco_db: Path = Path("/refs/busco"),
 ):
-    """NanoPlot → Flye → Medaka → compleasm end-to-end."""
+    """NanoPlot → Flye → Medaka → compleasm end-to-end.
+
+    ``polish=False`` skips the Medaka step — appropriate for HiFi reads,
+    whose per-base accuracy makes ONT consensus polishing unnecessary.
+    ``assess`` then reads Flye's ``assembly.fasta`` directly (it already
+    falls back from ``consensus.fasta``).
+    """
     out_dir = Path(out_dir).resolve()
     out_dir.mkdir(parents=True, exist_ok=True)
 
     lr = Path(long_reads)
     qc = read_qc(lr)
     asm = assemble(qc, lr, read_mode=read_mode)
-    pol = polish(asm, lr, medaka_model=medaka_model)
-    return assess(pol, busco_lineage=busco_lineage, busco_db=Path(busco_db))
+    polished = polish_consensus(asm, lr, medaka_model=medaka_model) if polish else asm
+    return assess(polished, busco_lineage=busco_lineage, busco_db=Path(busco_db))
 
 
 register("eukaryote_assembly", eukaryote_assembly)

{bioflowkit-0.2.1 → bioflowkit-0.3.0}/bioflow/recipes/metagenomics/metagenome_assembly.py

@@ -48,12 +48,16 @@ def assemble(clean, *, out_dir):
     )
 
 
-@stage(image="quay.io/biocontainers/minimap2:2.28--he4a0461_0",
+@stage(image=("quay.io/biocontainers/mulled-v2-"
+              "66534bcbb7031a148b13e2ad42583020b9cd25c4:"
+              "b411340b52d82a9c276d87c7a3dcffc880be762f-0"),
        cpu=16, ram_gb=32, depends_on=assemble)
 def map_back(asm, clean, *, out_dir):
     """Map QC reads back to contigs for coverage (minimap2 + samtools).
 
-    samtools is bundled in the minimap2 BioContainer.
+    Uses a mulled minimap2 + samtools BioContainer (minimap2 2.31 +
+    samtools 1.23) — the plain ``biocontainers/minimap2`` image ships no
+    samtools, so the ``minimap2 | samtools sort`` chain fails on it.
     """
     contigs = f"{asm.out_dir}/megahit/final.contigs.fa"
     return (

{bioflowkit-0.2.1 → bioflowkit-0.3.0}/bioflow/recipes/metagenomics/metagenomics_profile.py

@@ -6,10 +6,11 @@ End-to-end short-read shotgun-metagenomic workflow:
 Requires a prebuilt Kraken2 database mounted into the workspace (e.g.
 the standard MiniKraken2 or PlusPF DB).  Use::
 
-    bioflow db fetch kraken2_standard --dest /refs
+    bioflow db fetch kraken2_standard_8gb --dest /refs
 
-…to install one (when the catalog has it), or pass --kraken2-db pointing
-to a pre-existing directory.
+…to install one, or pass --kraken2-db pointing to a pre-existing
+directory.  (Bracken also needs the ``databaseNNmers.kmer_distrib``
+files that ship inside the standard Kraken2 DB tarball.)
 
 Researcher (Tier B) usage::