jamofetch 3.7.7__tar.gz → 3.7.9__tar.gz

jamofetch-3.7.9/CHANGELOG.md

@@ -0,0 +1,53 @@
+# Changelog
+
+<!--next-version-placeholder-->
+
+## v3.7.9 (17/09/2026)
+
+### Fixed
+
+- **A library sequenced more than once no longer resolves to an arbitrary file.**
+  `_find_fastq_path()` sorted the link directory by `ST_CTIME` and kept the last
+  match. Every link comes from one `jamo link` call within the same second and
+  `ST_CTIME` is whole seconds, so the candidates all tie and Python's stable sort
+  returns whichever `os.listdir()` listed last - directory hash order on Linux,
+  so not alphabetical, not newest, and not necessarily the same on two hosts.
+  It now returns the single match, or raises and names every candidate.
+
+  Observed on library LBBCZRJ: four matching records across three pools - cell
+  37397 (run 3207, 1,296,169 reads), cell 37892 (run 3232, 1,965,608 reads,
+  registered twice under different `AUTO-` folders with identical md5s) and cell
+  37964 (run 3230, 948,348 reads). Any of the three could be analysed, silently,
+  with a 2x spread in read count.
+
+  Links pointing at the same target are not ambiguous and are still accepted.
+
+### Added
+
+- `get_cmd()` and `JamoFetcher.fetch_lib_seq()` take `smrt_cell_id` and
+  `physical_run_id`, which narrow the query to one sequencing of the library.
+  `smrt_cell_id` is the one to prefer: it identifies a library's membership in a
+  single pool, and equals the PacBio pipeline service's `libraries[].id`. For
+  LBBCZRJ it reduces four matches to one (or, for cell 37892, to two records
+  that are byte-identical and share a `file_name`).
+- CLI: `--smrt-cell-id` (single `--library` only) and `--physical-run-id`
+  (applies to every `--library`).
+
+### Changed
+
+- The CLI no longer swallows per-library fetch errors. `_fetch_seq()` caught
+  every exception with a bare `pass`, which hid the new ambiguity error whose
+  whole purpose is to be seen.
+- `JamoFetcher.fetch_lib_seq()` logs every candidate link when more than one
+  matches, before selection.
+
+### Compatibility
+
+- Calls that pass no id produce the byte-identical command 3.7.8 produced;
+  asserted in `tests/test_jamofetch.py`. Callers that relied on a library with
+  several matches silently resolving to one of them now get a `RuntimeError`
+  instead - that is the point of the release.
+
+## v0.1.0 (07/07/2023)
+
+- First release of `jamofetch`!

{jamofetch-3.7.7 → jamofetch-3.7.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: jamofetch
-Version: 3.7.7
+Version: 3.7.9
 Summary: A thin wrapper to retrieve sequence from JAMO at NERSC and on Dori.
 Author: Duncan Scott
 Requires-Python: >=3.9
@@ -89,15 +89,15 @@ options:
 (venv) [dnscott@ln005 jamofetch]$ jamofetch -d data -l NPUNN -l NOOHG -l HOGH -w --max -1
 fetching sequence:
 
-apptainer --silent run docker://doejgi/jamo-dori jamo link -s dori library NPUNN
+apptainer --silent exec docker://doejgi/jamo-dori jamo link -s dori custom '{"metadata.library_name":"NPUNN","metadata.fastq_type":"sdm_normal","group":"sdm"}'
 NPUNN /global/dna/dm_archive/sdm/pacbio/00/27/47/pbio-2747.27352.bc1001_BAK8A_OA--bc1001_BAK8A_OA.ccs.fastq.gz BACKUP_COMPLETE 6391936239a7711d789a9380
 NPUNN /clusterfs/jgi/groups/dsi/homes/dnscott/git/jamofetch/data/NPUNN.pbio-2747.27352.bc1001_BAK8A_OA--bc1001_BAK8A_OA.ccs.fastq.gz
 
-apptainer --silent run docker://doejgi/jamo-dori jamo link -s dori library NOOHG
+apptainer --silent exec docker://doejgi/jamo-dori jamo link -s dori custom '{"metadata.library_name":"NOOHG","metadata.fastq_type":"sdm_normal","group":"sdm"}'
 NOOHG /global/dna/dm_archive/sdm/pacbio/00/26/91/pbio-2691.26653.bc1001_BAK8A_OA--bc1001_BAK8A_OA.ccs.fastq.gz RESTORED 6347dbb35bc59487d7e768d6
 NOOHG /clusterfs/jgi/groups/dsi/homes/dnscott/git/jamofetch/data/NOOHG.pbio-2691.26653.bc1001_BAK8A_OA--bc1001_BAK8A_OA.ccs.fastq.gz
 
-apptainer --silent run docker://doejgi/jamo-dori jamo link -s dori library HOGH
+apptainer --silent exec docker://doejgi/jamo-dori jamo link -s dori custom '{"metadata.library_name":"HOGH","metadata.fastq_type":"sdm_normal","group":"sdm"}'
 HOGH /global/dna/dm_archive/sdm/illumina/00/63/97/6397.2.44053.GGCTAC.fastq.gz RESTORED 51d52a82067c014cd6ef4f6f
 HOGH /clusterfs/jgi/groups/dsi/homes/dnscott/git/jamofetch/data/HOGH.6397.2.44053.GGCTAC.fastq.gz
 

{jamofetch-3.7.7 → jamofetch-3.7.9}/README.md

@@ -78,15 +78,15 @@ options:
 (venv) [dnscott@ln005 jamofetch]$ jamofetch -d data -l NPUNN -l NOOHG -l HOGH -w --max -1
 fetching sequence:
 
-apptainer --silent run docker://doejgi/jamo-dori jamo link -s dori library NPUNN
+apptainer --silent exec docker://doejgi/jamo-dori jamo link -s dori custom '{"metadata.library_name":"NPUNN","metadata.fastq_type":"sdm_normal","group":"sdm"}'
 NPUNN /global/dna/dm_archive/sdm/pacbio/00/27/47/pbio-2747.27352.bc1001_BAK8A_OA--bc1001_BAK8A_OA.ccs.fastq.gz BACKUP_COMPLETE 6391936239a7711d789a9380
 NPUNN /clusterfs/jgi/groups/dsi/homes/dnscott/git/jamofetch/data/NPUNN.pbio-2747.27352.bc1001_BAK8A_OA--bc1001_BAK8A_OA.ccs.fastq.gz
 
-apptainer --silent run docker://doejgi/jamo-dori jamo link -s dori library NOOHG
+apptainer --silent exec docker://doejgi/jamo-dori jamo link -s dori custom '{"metadata.library_name":"NOOHG","metadata.fastq_type":"sdm_normal","group":"sdm"}'
 NOOHG /global/dna/dm_archive/sdm/pacbio/00/26/91/pbio-2691.26653.bc1001_BAK8A_OA--bc1001_BAK8A_OA.ccs.fastq.gz RESTORED 6347dbb35bc59487d7e768d6
 NOOHG /clusterfs/jgi/groups/dsi/homes/dnscott/git/jamofetch/data/NOOHG.pbio-2691.26653.bc1001_BAK8A_OA--bc1001_BAK8A_OA.ccs.fastq.gz
 
-apptainer --silent run docker://doejgi/jamo-dori jamo link -s dori library HOGH
+apptainer --silent exec docker://doejgi/jamo-dori jamo link -s dori custom '{"metadata.library_name":"HOGH","metadata.fastq_type":"sdm_normal","group":"sdm"}'
 HOGH /global/dna/dm_archive/sdm/illumina/00/63/97/6397.2.44053.GGCTAC.fastq.gz RESTORED 51d52a82067c014cd6ef4f6f
 HOGH /clusterfs/jgi/groups/dsi/homes/dnscott/git/jamofetch/data/HOGH.6397.2.44053.GGCTAC.fastq.gz
 

{jamofetch-3.7.7 → jamofetch-3.7.9}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "flit_core.buildapi"
 
 [project]
 name = "jamofetch"
-version = "3.7.7"
+version = "3.7.9"
 description = "A thin wrapper to retrieve sequence from JAMO at NERSC and on Dori."
 authors = [{ name = "Duncan Scott" }]
 readme = "README.md"

jamofetch-3.7.9/src/jamofetch/jamofetch.py

@@ -0,0 +1,427 @@
+#!/usr/bin/env python3
+
+import json
+import logging
+import os
+import pathlib
+import re
+import subprocess
+import sys
+import time
+from argparse import ArgumentParser
+
+# Base commands; get_cmd() appends the query.
+#
+# Dori uses `apptainer exec`, not `run`: the jamo-dori image's runscript strips the
+# double quotes out of the JSON query, and jamo then fails with a JSONDecodeError.
+JAMO_CMD_NERSC = 'module load jamo; jamo link'
+JAMO_CMD_DORI = 'apptainer --silent exec docker://doejgi/jamo-dori jamo link -s dori'
+
+TWO_HOURS = 7200  # seconds
+ONE_MINUTE = 60  # seconds
+
+
+def get_cmd(clean_lib_name, smrt_cell_id=None, physical_run_id=None) -> str:
+    """
+    Build the site-appropriate `jamo link` command for a library's SDM fastq.
+
+    Uses a custom query rather than `jamo link library <name>`. That form applies
+    jamo's default `raw_normal` filter, {'metadata.fastq_type': 'sdm_normal',
+    'user': 'sdm'}, and files from SDM's newer pipeline are owned by user
+    'sdm_pipeline' (group still 'sdm'), so it matched nothing for them. Filtering
+    on group instead of user matches both.
+
+    Key order matters: jamo names each symlink '<first string-valued query key's
+    value>.<file_name>', so metadata.library_name must stay first or
+    _find_fastq_path() will not find the link. The disambiguators below are
+    integers, so they cannot take that position, but they are appended last
+    anyway.
+
+    A library sequenced on more than one run matches one fastq per run, and
+    without a disambiguator NOTHING here chooses between them - the caller ends
+    up with several symlinks and _find_fastq_path() refuses to guess. Pass
+    smrt_cell_id (preferred) or physical_run_id to narrow the query to the
+    sequencing this analysis is actually about.
+
+    Observed case, library LBBCZRJ: four matching records across three pools -
+    cell 37397 (run 3207, 1,296,169 reads), cell 37892 (run 3232, 1,965,608
+    reads, registered twice under different AUTO- folders with identical md5s)
+    and cell 37964 (run 3230, 948,348 reads). Adding the cell id returns exactly
+    one record in the first and third cases, and in the second the two records
+    are byte-identical and share a file_name, so they collapse to one symlink.
+
+    smrt_cell_id is the finer key and is the one to prefer: it identifies a
+    library's membership in one specific pool. It is JAMO's
+    metadata.sdm_smrt_cell_id, and equals the PacBio pipeline service's
+    libraries[].id for the same library.
+
+    Args:
+        clean_lib_name: the library name, e.g. 'LBBCZRJ'.
+        smrt_cell_id: optional metadata.sdm_smrt_cell_id to narrow to.
+        physical_run_id: optional metadata.pacbio_physical_run_id to narrow to.
+            Coarser than smrt_cell_id; a run holds many cells.
+
+    Returns:
+        The full shell command, with the JSON query single-quoted.
+    """
+    # Validated because the name is interpolated into a shell command.
+    lib_name = _clean_library_name(clean_lib_name)
+    query = {
+        "metadata.library_name": lib_name,
+        "metadata.fastq_type": "sdm_normal",
+        "group": "sdm",
+    }
+
+    cell = _check_id(smrt_cell_id, 'smrt_cell_id')
+    if cell is not None:
+        query["metadata.sdm_smrt_cell_id"] = cell
+
+    run = _check_id(physical_run_id, 'physical_run_id')
+    if run is not None:
+        query["metadata.pacbio_physical_run_id"] = run
+
+    query = json.dumps(query, separators=(',', ':'))
+    base = JAMO_CMD_DORI if os.getenv('SLURM_PARTITION') == 'dori' else JAMO_CMD_NERSC
+    return f"{base} custom '{query}'"
+
+
+def _check_id(value, name):
+    """
+    Normalise an optional integer query value, or raise.
+
+    Accepts an int or a string of digits (ids arrive from JSON APIs and from
+    argparse as either) and returns an int, or None when nothing was given.
+    Anything else raises: these values are interpolated into a shell command, so
+    a stray string must never reach it.
+    """
+    if value is None:
+        return None
+    if isinstance(value, bool):
+        raise ValueError(f"invalid {name}: expecting a positive integer, got {value!r}")
+    if isinstance(value, str):
+        stripped = value.strip()
+        if not stripped.isdigit():
+            raise ValueError(f"invalid {name}: expecting a positive integer, got {value!r}")
+        value = int(stripped)
+    if not isinstance(value, int):
+        raise ValueError(f"invalid {name}: expecting a positive integer, got {value!r}")
+    if value <= 0:
+        raise ValueError(f"invalid {name}: expecting a positive integer, got {value!r}")
+    return value
+
+
+def _clean_library_name(library_name):
+    clean_name = f"{library_name}".strip()
+    if not re.match(r'^[A-Z]+$', clean_name):
+        raise ValueError(f"invalid library name {library_name}")
+    return clean_name
+
+
+def _check_int_not_negative(value_to_check, value_name, allow_minus_1=False) -> int:
+    """
+    Throw an error if value_to_check is not a positive integer.  Provide
+    value_name in error message.
+    """
+    if not value_to_check:
+        return 0
+    if not isinstance(value_to_check, int):
+        raise ValueError(f"invalid value for {value_name}: expecting integer")
+    if allow_minus_1 and value_to_check == -1:
+        return -1
+    if value_to_check <= 0:
+        raise ValueError(f"invalid value for {value_name}: expecting positive integer")
+    return value_to_check
+
+
+def _wait_for_seq(seq_path, wait_interval_secs, wait_max_secs):
+    """
+    Check if seq_path is a link.  If seq_path is a broken link, wait until it is valid.
+    Check link each wait_interval_secs seconds.  Exit when link is valid.  Throw
+    an error if wait_max_secs is exceeded.  Set wait_max_secs to None to wait
+    indefinitely.
+    """
+    if seq_path is None:
+        raise ValueError("null seq_path")
+
+    if not os.path.islink(seq_path):
+        raise ValueError("seq_path is not a link")
+
+    wait_interval = _check_int_not_negative(wait_interval_secs, 'wait_interval_secs')
+    wait_max = _check_int_not_negative(wait_max_secs, 'wait_max_secs', allow_minus_1=True)
+
+    total_wait = 0
+    if wait_max and wait_interval:
+        logging.debug(f"waiting for link {seq_path}; wait_interval: {wait_interval}; wait_max: {wait_max}")
+        while (not os.path.exists(seq_path)) and (wait_max == -1 or total_wait < wait_max):
+            time.sleep(wait_interval_secs)
+            if wait_interval > 0:
+                total_wait += wait_interval
+
+    if not os.path.exists(seq_path):
+        raise RuntimeError(f"sequence for link {seq_path} not provisioned within max wait seconds: {wait_max}")
+
+
+def _matching_links(lib_name, seq_dir):
+    """
+    Return every symlink in seq_dir that jamo created for this library, as a
+    list of (link_path, target) sorted by link name.
+
+    jamo names each link '<library>.<file_name>', so the prefix is what
+    identifies ownership. Targets are read with os.readlink rather than
+    os.path.realpath: a link whose target has not been copied to this host yet
+    is still a real candidate, and realpath would obscure that.
+    """
+    links = []
+    for file_name in sorted(os.listdir(os.path.realpath(seq_dir))):
+        file_path = os.path.join(seq_dir, file_name)
+        if not os.path.islink(file_path):
+            continue
+        if file_name.startswith(f"{lib_name}.") and file_name.endswith('.fastq.gz'):
+            links.append((file_path, os.readlink(file_path)))
+    return links
+
+
+def _find_fastq_path(lib_name, seq_dir):
+    """
+    Return the one sequence symlink jamo created for this library.
+
+    This used to sort the directory by ctime and keep the last match, which was
+    a coin flip whenever a library matched more than one fastq: every link comes
+    from a single `jamo link` call within the same second, ST_CTIME is whole
+    seconds, so all the candidates tie and Python's stable sort just hands back
+    whichever os.listdir() happened to return last. That is directory hash order
+    on Linux - not alphabetical, not newest, and not the same on two hosts. A
+    library sequenced three times could therefore be analysed against any of the
+    three, silently, with a 2x spread in read count and nothing in the log to
+    say which.
+
+    So it no longer guesses. One candidate is returned; several distinct targets
+    raise, naming every one of them. Narrow the query instead - pass
+    smrt_cell_id or physical_run_id to get_cmd()/fetch_lib_seq() - or clear out
+    stale links from an earlier fetch.
+
+    Several links pointing at the SAME target are not ambiguous and are accepted:
+    SDM registers the same file under more than one path (observed for LBBCZRJ,
+    two AUTO- folders, identical md5).
+    """
+    links = _matching_links(lib_name, seq_dir)
+
+    if not links:
+        raise RuntimeError(f"failed to find sequence file for library {lib_name}")
+
+    distinct_targets = {target for _, target in links}
+    if len(distinct_targets) > 1:
+        listing = "\n".join(
+            f"    {os.path.basename(link)} -> {target}" for link, target in links
+        )
+        raise RuntimeError(
+            f"library {lib_name} matches {len(distinct_targets)} different "
+            f"sequence files in {seq_dir}; refusing to pick one:\n{listing}\n"
+            "    Narrow the query with smrt_cell_id (preferred) or "
+            "physical_run_id, or remove links left over from an earlier fetch."
+        )
+
+    logging.debug(f"returning library seq file: {links[0][0]}")
+    return links[0][0]
+
+
+class LibSeq:
+
+    def __init__(self, lib_name, seq_path):
+        self._lib_name = _clean_library_name(lib_name)
+        self._seq_path = seq_path
+
+    def get_lib_name(self) -> str:
+        return self._lib_name
+
+    def get_seq_path(self):
+        return self._seq_path
+
+    def seq_exists(self) -> bool:
+        return os.path.exists(self._seq_path)
+
+    def get_real_path(self):
+        if self._seq_path is None:
+            return None
+        return os.path.realpath(self._seq_path)
+
+    def get_real_path_wait(self):
+        if self.seq_exists():
+            return self.get_real_path()
+        else:
+            raise RuntimeError(f"{self}: sequence file not available")
+
+    def __str__(self):
+        f"{self.__class__.__name__}(lib={self._lib_name};path={self._seq_path})"
+
+
+class JamoLibSeq(LibSeq):
+
+    def __init__(self, lib_name, seq_path, wait_interval_secs=10, wait_max_secs=-1):
+        LibSeq.__init__(self, lib_name, seq_path)
+        self._wait_interval_secs = _check_int_not_negative(wait_interval_secs, 'wait_interval_secs')
+        self._wait_max_secs = _check_int_not_negative(wait_max_secs, 'wait_max_secs', allow_minus_1=True)
+        self._sequence_ready = False
+
+    def get_real_path_wait(self):
+        if not self._sequence_ready:
+            _wait_for_seq(self._seq_path, self._wait_interval_secs, self._wait_max_secs)
+            self._sequence_ready = True
+        return os.path.realpath(self._seq_path)
+
+
+class JamoFetcher():
+    def __init__(self, link_dir='.', wait_interval_secs=10, wait_max_secs=-1):
+        self._wait_interval_secs = _check_int_not_negative(wait_interval_secs, 'wait_interval_secs')
+        self._wait_max_secs = _check_int_not_negative(wait_max_secs, 'wait_max_secs', allow_minus_1=True)
+        self._link_dir = os.path.realpath(link_dir)
+
+    def fetch_lib_seq(self, lib_name, out_file=sys.stderr, smrt_cell_id=None,
+                      physical_run_id=None) -> JamoLibSeq:
+        """
+        Execute JAMO command to link sequence for library
+        and return a LibSeq object containing the path to the sequence.
+
+        smrt_cell_id (preferred) or physical_run_id narrow the query to one
+        sequencing of this library. Without one, a library sequenced more than
+        once links several files and _find_fastq_path() raises rather than pick
+        between them - see get_cmd().
+
+        They are per-call rather than per-fetcher because one JamoFetcher is
+        commonly shared across a pool's libraries, and each library has its own
+        cell id.
+        """
+        pathlib.Path(self._link_dir).mkdir(parents=True, exist_ok=True, mode=0o775)
+        os.chdir(self._link_dir)
+        clean_lib_name = _clean_library_name(lib_name)
+
+        cmd = get_cmd(clean_lib_name, smrt_cell_id=smrt_cell_id,
+                      physical_run_id=physical_run_id)
+        print(f"\n{cmd}", file=out_file)
+        output = subprocess.check_output(cmd, shell=True)
+        for line in output.splitlines():
+            print(line.decode("utf-8"), file=out_file)
+
+        # Report what was linked before choosing, so an ambiguous fetch is
+        # visible in the log even though the next line raises on it.
+        links = _matching_links(clean_lib_name, self._link_dir)
+        if len(links) > 1:
+            print(f"{clean_lib_name}: {len(links)} candidate links in "
+                  f"{self._link_dir}", file=out_file)
+            for link, target in links:
+                print(f"    {os.path.basename(link)} -> {target}", file=out_file)
+
+        seq_path = _find_fastq_path(clean_lib_name, self._link_dir)
+        print(f"{clean_lib_name} {seq_path}", file=out_file)
+        return JamoLibSeq(clean_lib_name, seq_path, self._wait_interval_secs, self._wait_max_secs)
+
+
+def _fetch_seq(fetcher: JamoFetcher, libs, smrt_cell_id=None, physical_run_id=None):
+    lib_seq_dict = {}
+    for lib in libs:
+        try:
+            lib_seq: JamoLibSeq = fetcher.fetch_lib_seq(
+                lib, out_file=sys.stderr, smrt_cell_id=smrt_cell_id,
+                physical_run_id=physical_run_id)
+            lib_seq_dict[lib_seq.get_lib_name()] = lib_seq
+        except Exception as e:
+            # Was a bare `pass`, which hid every failure - including the
+            # ambiguous-match error, whose whole purpose is to be seen.
+            print(f"{lib}: {e}", file=sys.stderr)
+    return lib_seq_dict
+
+
+def main(args):
+    logging.debug(f"args: {args}")
+
+    if not args.library:
+        return
+
+    wait_interval_secs = _check_int_not_negative(args.interval, "interval")
+    wait_max_secs = _check_int_not_negative(args.max, "max", allow_minus_1=True)
+
+    # A cell id identifies one library's membership in one pool, so it cannot be
+    # shared across several -l libraries. A run id can: a run holds many cells.
+    if args.smrt_cell_id is not None and len(args.library) > 1:
+        raise ValueError(
+            "--smrt-cell-id applies to a single library; it was given with "
+            f"{len(args.library)} -l options. Use --physical-run-id, or fetch "
+            "one library at a time.")
+
+    link_dir = os.path.realpath(args.directory if args.directory else '.')
+    fetcher: JamoFetcher = JamoFetcher(link_dir=link_dir, wait_interval_secs=wait_interval_secs,
+                                       wait_max_secs=wait_max_secs)
+
+    print("fetching sequence:")
+    lib_seq_dict = _fetch_seq(fetcher, args.library,
+                              smrt_cell_id=args.smrt_cell_id,
+                              physical_run_id=args.physical_run_id)
+
+    if not lib_seq_dict:
+        return  # no sequence was fetched
+
+    print("\nsequence links:")
+    for lib_name, lib_seq in sorted(lib_seq_dict.items()):
+        print(f"{lib_seq.get_lib_name()} symlink: {lib_seq.get_seq_path()}")
+        print(f"{lib_seq.get_lib_name()} realpath: {lib_seq.get_real_path()}")
+
+    if args.wait:
+        total_wait = 0
+        seq_ready = set()
+        print("\nwaiting for JAMO to provision sequence . . . .")
+        while (len(seq_ready) < len(lib_seq_dict)) and (wait_max_secs == -1 or total_wait <= wait_max_secs):
+            for lib_name, lib_seq in sorted(lib_seq_dict.items()):
+                if not lib_name in seq_ready and lib_seq.seq_exists():
+                    print(f"{lib_name} sequence ready")
+                    seq_ready.add(lib_name)
+            if not wait_interval_secs or not wait_max_secs:
+                break  # do not wait
+            if len(seq_ready) < len(lib_seq_dict):
+                time.sleep(wait_interval_secs)
+                if wait_max_secs > 0:
+                    total_wait += wait_interval_secs
+        if len(seq_ready) < len(lib_seq_dict):
+            print("\nexiting, not all sequence ready")
+            if wait_max_secs != -1 and total_wait >= wait_max_secs:
+                print(f"max wait {wait_max_secs} exceeded")
+
+
+def cli():
+    try:
+        parser: ArgumentParser = ArgumentParser()
+        parser.add_argument('-l', '--library', action='append',
+                            help="library name(s) for which to retrieve sequence")
+        parser.add_argument('-d', '--directory', required=False, default='.',
+                            help="directory where to link sequence, defaults to current directory.  " +
+                                 "Directory will be created if it doesn't exit.")
+        parser.add_argument('--smrt-cell-id', required=False, type=int, default=None,
+                            help="metadata.sdm_smrt_cell_id to fetch, for a library sequenced " +
+                                 "more than once. Identifies one library in one pool, so it " +
+                                 "applies to a single --library.")
+        parser.add_argument('--physical-run-id', required=False, type=int, default=None,
+                            help="metadata.pacbio_physical_run_id to fetch, for a library " +
+                                 "sequenced more than once. Coarser than --smrt-cell-id; " +
+                                 "applies to every --library given.")
+        parser.add_argument('-i', '--interval', required=False, type=int, default=10,
+                            help="wait interval in seconds to check if sequence has been fetched, " +
+                                 "ignored if wait flag not set")
+        parser.add_argument('-m', '--max', required=False, type=int, default=TWO_HOURS,
+                            help="maximum time to wait for sequence in seconds, " +
+                                 "ignored if wait flag not set.  Specify -1 to wait indefinetely.")
+        parser.add_argument('-w', '--wait', action='store_true',
+                            help='wait for jamo to link sequence, then print "sequence ready"')
+        parser.add_argument('--logging', required=False, default='WARN',
+                            help="logging level (specify DEBUG for verbose logging)")
+
+        ARGS = parser.parse_args()
+        # logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p', level=log_level)
+        logging.basicConfig(format='%(message)s', level=ARGS.logging.upper())
+        main(ARGS)
+    except KeyboardInterrupt:
+        print('Interrupted', file=sys.stderr)
+        # https://unix.stackexchange.com/questions/251996/why-does-bash-set-exit-status-to-non-zero-on-ctrl-c-or-ctrl-z
+        sys.exit(130)
+
+
+if __name__ == "__main__":
+    cli()

jamofetch-3.7.9/tests/test_jamofetch.py

@@ -0,0 +1,184 @@
+import json
+import os
+import shlex
+
+import pytest
+
+from jamofetch import jamofetch
+
+
+def _query(cmd):
+    # The query is the single-quoted final argument; shlex undoes the shell quoting.
+    return json.loads(shlex.split(cmd)[-1])
+
+
+def test_nersc_cmd(monkeypatch):
+    monkeypatch.delenv('SLURM_PARTITION', raising=False)
+    cmd = jamofetch.get_cmd('LBBDHFZ')
+    assert cmd.startswith('module load jamo; jamo link custom ')
+    assert _query(cmd) == {
+        "metadata.library_name": "LBBDHFZ",
+        "metadata.fastq_type": "sdm_normal",
+        "group": "sdm",
+    }
+
+
+def test_dori_cmd_uses_exec(monkeypatch):
+    monkeypatch.setenv('SLURM_PARTITION', 'dori')
+    cmd = jamofetch.get_cmd('LBBDHFZ')
+    assert cmd.startswith('apptainer --silent exec docker://doejgi/jamo-dori jamo link -s dori custom ')
+    assert _query(cmd)["metadata.library_name"] == "LBBDHFZ"
+
+
+def test_library_name_is_first_query_key(monkeypatch):
+    # jamo names the symlink after the first string-valued key; _find_fastq_path
+    # relies on that being the library name.
+    monkeypatch.delenv('SLURM_PARTITION', raising=False)
+    assert next(iter(_query(jamofetch.get_cmd('LBBDHFZ')))) == "metadata.library_name"
+
+
+def test_query_does_not_filter_on_user(monkeypatch):
+    monkeypatch.delenv('SLURM_PARTITION', raising=False)
+    assert "user" not in _query(jamofetch.get_cmd('LBBDHFZ'))
+
+
+@pytest.mark.parametrize('bad', ["LBB'DHFZ", 'lbbdhfz', 'LBB DHFZ', 'LBB;rm'])
+def test_invalid_library_name_rejected(bad):
+    with pytest.raises(ValueError):
+        jamofetch.get_cmd(bad)
+
+
+# --- disambiguating a library sequenced more than once -----------------------
+#
+# Real case this exists for, library LBBCZRJ: four records matching the plain
+# query, across three pools -- cell 37397 (run 3207), cell 37892 (run 3232,
+# registered twice with identical md5s) and cell 37964 (run 3230). Read counts
+# 1,296,169 / 1,965,608 / 948,348. Before 3.7.9 the one analysed was whichever
+# os.listdir() returned last.
+
+
+def test_smrt_cell_id_narrows_query(monkeypatch):
+    monkeypatch.delenv('SLURM_PARTITION', raising=False)
+    query = _query(jamofetch.get_cmd('LBBCZRJ', smrt_cell_id=37892))
+    assert query == {
+        "metadata.library_name": "LBBCZRJ",
+        "metadata.fastq_type": "sdm_normal",
+        "group": "sdm",
+        "metadata.sdm_smrt_cell_id": 37892,
+    }
+
+
+def test_physical_run_id_narrows_query(monkeypatch):
+    monkeypatch.delenv('SLURM_PARTITION', raising=False)
+    query = _query(jamofetch.get_cmd('LBBCZRJ', physical_run_id=3232))
+    assert query["metadata.pacbio_physical_run_id"] == 3232
+
+
+def test_both_ids_may_be_given(monkeypatch):
+    monkeypatch.delenv('SLURM_PARTITION', raising=False)
+    query = _query(jamofetch.get_cmd('LBBCZRJ', smrt_cell_id=37892, physical_run_id=3232))
+    assert query["metadata.sdm_smrt_cell_id"] == 37892
+    assert query["metadata.pacbio_physical_run_id"] == 3232
+
+
+def test_library_name_still_first_key_with_ids(monkeypatch):
+    # jamo names the symlink after the first string-valued key. The ids are
+    # ints so they cannot take that slot, but the ordering is load-bearing
+    # enough to assert directly.
+    monkeypatch.delenv('SLURM_PARTITION', raising=False)
+    query = _query(jamofetch.get_cmd('LBBCZRJ', smrt_cell_id=37892))
+    assert next(iter(query)) == "metadata.library_name"
+
+
+def test_no_ids_produces_the_3_7_8_command(monkeypatch):
+    # Backward compatibility: callers that pass no id must get the byte-identical
+    # command 3.7.8 produced. synbioqc-pbj pins these strings in its own tests.
+    monkeypatch.delenv('SLURM_PARTITION', raising=False)
+    assert jamofetch.get_cmd('LBBDHFZ') == (
+        "module load jamo; jamo link custom "
+        '\'{"metadata.library_name":"LBBDHFZ","metadata.fastq_type":"sdm_normal","group":"sdm"}\''
+    )
+
+
+@pytest.mark.parametrize('bad', ['37892; rm -rf /', 'abc', '', -1, 0, 3.5, True, [37892]])
+def test_invalid_ids_rejected(bad):
+    with pytest.raises(ValueError):
+        jamofetch.get_cmd('LBBCZRJ', smrt_cell_id=bad)
+
+
+def test_numeric_string_id_accepted(monkeypatch):
+    # Ids arrive from JSON APIs and argparse as either int or str.
+    monkeypatch.delenv('SLURM_PARTITION', raising=False)
+    assert _query(jamofetch.get_cmd('LBBCZRJ', smrt_cell_id='37892'))[
+        "metadata.sdm_smrt_cell_id"] == 37892
+
+
+# --- _find_fastq_path no longer guesses --------------------------------------
+
+LBBCZRJ_TARGETS = {
+    "pbio-3207.37397.bc2080_OA--bc2080_OA.hifi_reads.bc2080_OA.ccs.fastq.gz":
+        "/global/dna/dm_archive/sdm/pacbio/00/32/07",
+    "pbr6184.3232.37663.37892.bc2080_OA--bc2080_OA.pacbio.processed_well.37663.37892.fastq.gz":
+        "/global/dna/dm_archive/sdm/analyses-242/AUTO-2421306",
+    "pbr6186.3230.37667.37964.bc2080_OA--bc2080_OA.pacbio.processed_well.37667.37964.fastq.gz":
+        "/global/dna/dm_archive/sdm/analyses-242/AUTO-2421406",
+}
+
+
+def _link(seq_dir, lib, file_name, folder):
+    link = os.path.join(seq_dir, f"{lib}.{file_name}")
+    os.symlink(os.path.join(folder, file_name), link)
+    return link
+
+
+def test_single_match_is_returned(tmp_path):
+    name, folder = next(iter(LBBCZRJ_TARGETS.items()))
+    _link(str(tmp_path), 'LBBCZRJ', name, folder)
+    assert jamofetch._find_fastq_path('LBBCZRJ', str(tmp_path)).endswith(name)
+
+
+def test_ambiguous_match_raises_and_names_every_candidate(tmp_path):
+    for name, folder in LBBCZRJ_TARGETS.items():
+        _link(str(tmp_path), 'LBBCZRJ', name, folder)
+
+    with pytest.raises(RuntimeError) as excinfo:
+        jamofetch._find_fastq_path('LBBCZRJ', str(tmp_path))
+
+    message = str(excinfo.value)
+    assert "3 different" in message
+    for name, folder in LBBCZRJ_TARGETS.items():
+        assert os.path.join(folder, name) in message
+    assert "smrt_cell_id" in message
+
+
+def test_duplicate_registrations_of_one_file_are_not_ambiguous(tmp_path):
+    # SDM registered the same file under two AUTO- folders with identical md5s.
+    # Same file_name means one link name, so only one link can exist -- but if
+    # two links ever point at the same target, that is not a real choice.
+    name = "pbr6184.3232.37663.37892.bc2080_OA--bc2080_OA.pacbio.processed_well.37663.37892.fastq.gz"
+    target = "/global/dna/dm_archive/sdm/analyses-242/AUTO-2421306/" + name
+    os.symlink(target, os.path.join(str(tmp_path), f"LBBCZRJ.{name}"))
+    os.symlink(target, os.path.join(str(tmp_path), f"LBBCZRJ.copy.{name}"))
+    assert jamofetch._find_fastq_path('LBBCZRJ', str(tmp_path)) is not None
+
+
+def test_other_libraries_links_are_ignored(tmp_path):
+    name, folder = next(iter(LBBCZRJ_TARGETS.items()))
+    _link(str(tmp_path), 'LBBCZRJ', name, folder)
+    for other_name, other_folder in list(LBBCZRJ_TARGETS.items())[1:]:
+        _link(str(tmp_path), 'LBBDHFZ', other_name, other_folder)
+    assert jamofetch._find_fastq_path('LBBCZRJ', str(tmp_path)).endswith(name)
+
+
+def test_no_match_raises(tmp_path):
+    with pytest.raises(RuntimeError, match="failed to find sequence file"):
+        jamofetch._find_fastq_path('LBBCZRJ', str(tmp_path))
+
+
+def test_broken_link_is_still_a_candidate(tmp_path):
+    # On dori the target is copied into a local cache and the link is broken
+    # until that lands; it is still the file this library resolved to.
+    name, folder = next(iter(LBBCZRJ_TARGETS.items()))
+    link = _link(str(tmp_path), 'LBBCZRJ', name, folder)
+    assert not os.path.exists(link)
+    assert jamofetch._find_fastq_path('LBBCZRJ', str(tmp_path)) == link

jamofetch-3.7.7/CHANGELOG.md

@@ -1,7 +0,0 @@
-# Changelog
-
-<!--next-version-placeholder-->
-
-## v0.1.0 (07/07/2023)
-
-- First release of `jamofetch`!

jamofetch-3.7.7/src/jamofetch/jamofetch.py

@@ -1,271 +0,0 @@
-#!/usr/bin/env python3
-
-import logging
-import os
-import pathlib
-import re
-import subprocess
-import sys
-import time
-from argparse import ArgumentParser
-from stat import ST_CTIME
-
-JAMO_CMD_NERSC = 'module load jamo; jamo link library'
-JAMO_CMD_DORI = 'apptainer --silent run docker://doejgi/jamo-dori jamo link -s dori library'
-
-TWO_HOURS = 7200  # seconds
-ONE_MINUTE = 60  # seconds
-
-
-def get_cmd(clean_lib_name) -> str:
-    if os.getenv('SLURM_PARTITION') == 'dori':
-        return f"{JAMO_CMD_DORI} {clean_lib_name}"
-    return f"{JAMO_CMD_NERSC} {clean_lib_name}"
-
-
-def _clean_library_name(library_name):
-    clean_name = f"{library_name}".strip()
-    if not re.match(r'^[A-Z]+$', clean_name):
-        raise ValueError(f"invalid library name {library_name}")
-    return clean_name
-
-
-def _check_int_not_negative(value_to_check, value_name, allow_minus_1=False) -> int:
-    """
-    Throw an error if value_to_check is not a positive integer.  Provide
-    value_name in error message.
-    """
-    if not value_to_check:
-        return 0
-    if not isinstance(value_to_check, int):
-        raise ValueError(f"invalid value for {value_name}: expecting integer")
-    if allow_minus_1 and value_to_check == -1:
-        return -1
-    if value_to_check <= 0:
-        raise ValueError(f"invalid value for {value_name}: expecting positive integer")
-    return value_to_check
-
-
-def _wait_for_seq(seq_path, wait_interval_secs, wait_max_secs):
-    """
-    Check if seq_path is a link.  If seq_path is a broken link, wait until it is valid.
-    Check link each wait_interval_secs seconds.  Exit when link is valid.  Throw
-    an error if wait_max_secs is exceeded.  Set wait_max_secs to None to wait
-    indefinitely.
-    """
-    if seq_path is None:
-        raise ValueError("null seq_path")
-
-    if not os.path.islink(seq_path):
-        raise ValueError("seq_path is not a link")
-
-    wait_interval = _check_int_not_negative(wait_interval_secs, 'wait_interval_secs')
-    wait_max = _check_int_not_negative(wait_max_secs, 'wait_max_secs', allow_minus_1=True)
-
-    total_wait = 0
-    if wait_max and wait_interval:
-        logging.debug(f"waiting for link {seq_path}; wait_interval: {wait_interval}; wait_max: {wait_max}")
-        while (not os.path.exists(seq_path)) and (wait_max == -1 or total_wait < wait_max):
-            time.sleep(wait_interval_secs)
-            if wait_interval > 0:
-                total_wait += wait_interval
-
-    if not os.path.exists(seq_path):
-        raise RuntimeError(f"sequence for link {seq_path} not provisioned within max wait seconds: {wait_max}")
-
-
-def _find_fastq_path(lib_name, seq_dir):
-    """
-    Find symlink created for library in sequence directory
-    by matching pattern similar to:
-    HHCHO.52485.1.359430.GTGCTTA-GTAAGCA.fastq.gz
-    """
-    # sort contents in directory by modification time (we want the last modified)
-    # adapted from:
-    # https://www.tutorialspoint.com/How-do-you-get-a-directory-listing-sorted-by-creation-date-in-Python
-    file_paths = [os.path.join(seq_dir, file_name) for file_name in os.listdir(os.path.realpath(seq_dir))]
-    logging.debug(f"file paths: {file_paths}")
-    # get file stats
-    path_stats = [(path, os.lstat(path)) for path in file_paths]
-    path_stats.sort(key=lambda x: x[1][ST_CTIME])
-    for path_stat in path_stats:
-        logging.debug(f"{path_stat[0]}:")
-        for stat in sorted(filter(lambda a: a.startswith('st_'), dir(path_stat[1]))):
-            logging.debug(f"    {stat}: {getattr(path_stat[1], stat)}")
-
-    matched_link = None
-    for path_stat in path_stats:
-        filepath = path_stat[0]
-        filetime = path_stat[1][ST_CTIME]
-        filename = os.path.basename(filepath)
-        if not os.path.islink(filepath):
-            continue
-        logging.debug(f"link candidate: {filename}; creation time: {filetime}")
-        if str(filename).startswith(f"{lib_name}.") and str(filename).endswith('.fastq.gz'):
-            matched_link = filepath
-            logging.debug(f"link match: {filename}")
-
-    if matched_link is not None:
-        logging.debug(f"returning library seq file: {matched_link}")
-        return matched_link
-
-    raise RuntimeError(f"failed to find sequence file for library {lib_name}")
-
-
-class LibSeq:
-
-    def __init__(self, lib_name, seq_path):
-        self._lib_name = _clean_library_name(lib_name)
-        self._seq_path = seq_path
-
-    def get_lib_name(self) -> str:
-        return self._lib_name
-
-    def get_seq_path(self):
-        return self._seq_path
-
-    def seq_exists(self) -> bool:
-        return os.path.exists(self._seq_path)
-
-    def get_real_path(self):
-        if self._seq_path is None:
-            return None
-        return os.path.realpath(self._seq_path)
-
-    def get_real_path_wait(self):
-        if self.seq_exists():
-            return self.get_real_path()
-        else:
-            raise RuntimeError(f"{self}: sequence file not available")
-
-    def __str__(self):
-        f"{self.__class__.__name__}(lib={self._lib_name};path={self._seq_path})"
-
-
-class JamoLibSeq(LibSeq):
-
-    def __init__(self, lib_name, seq_path, wait_interval_secs=10, wait_max_secs=-1):
-        LibSeq.__init__(self, lib_name, seq_path)
-        self._wait_interval_secs = _check_int_not_negative(wait_interval_secs, 'wait_interval_secs')
-        self._wait_max_secs = _check_int_not_negative(wait_max_secs, 'wait_max_secs', allow_minus_1=True)
-        self._sequence_ready = False
-
-    def get_real_path_wait(self):
-        if not self._sequence_ready:
-            _wait_for_seq(self._seq_path, self._wait_interval_secs, self._wait_max_secs)
-            self._sequence_ready = True
-        return os.path.realpath(self._seq_path)
-
-
-class JamoFetcher():
-    def __init__(self, link_dir='.', wait_interval_secs=10, wait_max_secs=-1):
-        self._wait_interval_secs = _check_int_not_negative(wait_interval_secs, 'wait_interval_secs')
-        self._wait_max_secs = _check_int_not_negative(wait_max_secs, 'wait_max_secs', allow_minus_1=True)
-        self._link_dir = os.path.realpath(link_dir)
-
-    def fetch_lib_seq(self, lib_name, out_file=sys.stderr) -> JamoLibSeq:
-        """
-        Execute JAMO command to link sequence for library
-        and return a LibSeq object containing the path to the sequence.
-        """
-        pathlib.Path(self._link_dir).mkdir(parents=True, exist_ok=True, mode=0o775)
-        os.chdir(self._link_dir)
-        clean_lib_name = _clean_library_name(lib_name)
-
-        cmd = get_cmd(clean_lib_name)
-        print(f"\n{cmd}", file=out_file)
-        output = subprocess.check_output(cmd, shell=True)
-        for line in output.splitlines():
-            print(line.decode("utf-8"), file=out_file)
-        seq_path = _find_fastq_path(clean_lib_name, self._link_dir)
-        print(f"{clean_lib_name} {seq_path}", file=out_file)
-        return JamoLibSeq(clean_lib_name, seq_path, self._wait_interval_secs, self._wait_max_secs)
-
-
-def _fetch_seq(fetcher: JamoFetcher, libs):
-    lib_seq_dict = {}
-    for lib in libs:
-        try:
-            lib_seq: JamoLibSeq = fetcher.fetch_lib_seq(lib, out_file=sys.stderr)
-            lib_seq_dict[lib_seq.get_lib_name()] = lib_seq
-        except Exception as e:
-            pass
-    return lib_seq_dict
-
-
-def main(args):
-    logging.debug(f"args: {args}")
-
-    if not args.library:
-        return
-
-    wait_interval_secs = _check_int_not_negative(args.interval, "interval")
-    wait_max_secs = _check_int_not_negative(args.max, "max", allow_minus_1=True)
-
-    link_dir = os.path.realpath(args.directory if args.directory else '.')
-    fetcher: JamoFetcher = JamoFetcher(link_dir=link_dir, wait_interval_secs=wait_interval_secs,
-                                       wait_max_secs=wait_max_secs)
-
-    print("fetching sequence:")
-    lib_seq_dict = _fetch_seq(fetcher, args.library)
-
-    if not lib_seq_dict:
-        return  # no sequence was fetched
-
-    print("\nsequence links:")
-    for lib_name, lib_seq in sorted(lib_seq_dict.items()):
-        print(f"{lib_seq.get_lib_name()} symlink: {lib_seq.get_seq_path()}")
-        print(f"{lib_seq.get_lib_name()} realpath: {lib_seq.get_real_path()}")
-
-    if args.wait:
-        total_wait = 0
-        seq_ready = set()
-        print("\nwaiting for JAMO to provision sequence . . . .")
-        while (len(seq_ready) < len(lib_seq_dict)) and (wait_max_secs == -1 or total_wait <= wait_max_secs):
-            for lib_name, lib_seq in sorted(lib_seq_dict.items()):
-                if not lib_name in seq_ready and lib_seq.seq_exists():
-                    print(f"{lib_name} sequence ready")
-                    seq_ready.add(lib_name)
-            if not wait_interval_secs or not wait_max_secs:
-                break  # do not wait
-            if len(seq_ready) < len(lib_seq_dict):
-                time.sleep(wait_interval_secs)
-                if wait_max_secs > 0:
-                    total_wait += wait_interval_secs
-        if len(seq_ready) < len(lib_seq_dict):
-            print("\nexiting, not all sequence ready")
-            if wait_max_secs != -1 and total_wait >= wait_max_secs:
-                print(f"max wait {wait_max_secs} exceeded")
-
-
-def cli():
-    try:
-        parser: ArgumentParser = ArgumentParser()
-        parser.add_argument('-l', '--library', action='append',
-                            help="library name(s) for which to retrieve sequence")
-        parser.add_argument('-d', '--directory', required=False, default='.',
-                            help="directory where to link sequence, defaults to current directory.  " +
-                                 "Directory will be created if it doesn't exit.")
-        parser.add_argument('-i', '--interval', required=False, type=int, default=10,
-                            help="wait interval in seconds to check if sequence has been fetched, " +
-                                 "ignored if wait flag not set")
-        parser.add_argument('-m', '--max', required=False, type=int, default=TWO_HOURS,
-                            help="maximum time to wait for sequence in seconds, " +
-                                 "ignored if wait flag not set.  Specify -1 to wait indefinetely.")
-        parser.add_argument('-w', '--wait', action='store_true',
-                            help='wait for jamo to link sequence, then print "sequence ready"')
-        parser.add_argument('--logging', required=False, default='WARN',
-                            help="logging level (specify DEBUG for verbose logging)")
-
-        ARGS = parser.parse_args()
-        # logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p', level=log_level)
-        logging.basicConfig(format='%(message)s', level=ARGS.logging.upper())
-        main(ARGS)
-    except KeyboardInterrupt:
-        print('Interrupted', file=sys.stderr)
-        # https://unix.stackexchange.com/questions/251996/why-does-bash-set-exit-status-to-non-zero-on-ctrl-c-or-ctrl-z
-        sys.exit(130)
-
-
-if __name__ == "__main__":
-    cli()

jamofetch-3.7.7/tests/test_jamofetch.py

@@ -1,2 +0,0 @@
-from jamofetch import jamofetch
-