orfdata 0.43.1__tar.gz → 0.45.0__tar.gz

{orfdata-0.43.1/orfdata.egg-info → orfdata-0.45.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: orfdata
-Version: 0.43.1
+Version: 0.45.0
 Summary: Reference proteomes with evidence-backed noncanonical ORFs on PyEnsembl genomes
 Author: OpenVax
 License-Expression: Apache-2.0
@@ -61,9 +61,11 @@ downloaded explicitly and checked against their pinned checksums. The
 database through its custom VFS; released dataset queries and rebuilds from
 normalized sources do not need it.
 
-List every bundled, downloadable or buildable resource and what is installed:
+Run `orfdata` for the command guide, or `orfdata help data` to discover data
+subcommands. List downloadable or buildable resources and what is installed:
 
 ```sh
+orfdata
 orfdata data list
 orfdata data status
 orfdata data info pfizer-atlas-dog-peptide-evidence
@@ -75,6 +77,8 @@ orfdata data path pfizer-atlas-dog-peptide-evidence
 when no artifact is distributed. The same command handles optional catalogues
 such as `openprot-mouse` and datasets such as `peptideatlas-dog-peptides`.
 `downloads` and `download` are aliases for `data`; `info` and `path` stay offline.
+Output is readable by default; use `--json` for scripts and `--limit 0` for full
+human tables. See the [CLI guide and migration notes](https://openvax.github.io/orfdata/cli/).
 
 `orfdata setup --species human` also installs `gdc-viruses`, the default
 200-reference GDC viral screen for human transcriptome analysis;

{orfdata-0.43.1 → orfdata-0.45.0}/README.md

@@ -29,9 +29,11 @@ downloaded explicitly and checked against their pinned checksums. The
 database through its custom VFS; released dataset queries and rebuilds from
 normalized sources do not need it.
 
-List every bundled, downloadable or buildable resource and what is installed:
+Run `orfdata` for the command guide, or `orfdata help data` to discover data
+subcommands. List downloadable or buildable resources and what is installed:
 
 ```sh
+orfdata
 orfdata data list
 orfdata data status
 orfdata data info pfizer-atlas-dog-peptide-evidence
@@ -43,6 +45,8 @@ orfdata data path pfizer-atlas-dog-peptide-evidence
 when no artifact is distributed. The same command handles optional catalogues
 such as `openprot-mouse` and datasets such as `peptideatlas-dog-peptides`.
 `downloads` and `download` are aliases for `data`; `info` and `path` stay offline.
+Output is readable by default; use `--json` for scripts and `--limit 0` for full
+human tables. See the [CLI guide and migration notes](https://openvax.github.io/orfdata/cli/).
 
 `orfdata setup --species human` also installs `gdc-viruses`, the default
 200-reference GDC viral screen for human transcriptome analysis;

orfdata-0.45.0/docs/cli-design.md

@@ -0,0 +1,53 @@
+# CLI design and audit
+
+ORFdata's command line is for discovering data, inspecting local state, acquiring
+explicitly selected resources, and exporting records for other tools. Running
+`orfdata` should explain these tasks without loading a catalogue or downloading
+anything.
+
+The design follows the [Command Line Interface Guidelines](https://clig.dev/#help):
+show help for bare command groups, make subcommands discoverable, keep successful
+output brief, and offer explicit machine formats. The output conventions also
+draw on OpenVax's OncoRef inventory summaries and MHCtools' shared table formatter.
+
+## Conventions
+
+- `orfdata`, `orfdata help`, and `orfdata --help` show the command index.
+- `orfdata help COMMAND` and `orfdata COMMAND --help` show command help.
+- Human summaries and tables are the default. `--json` selects a complete JSON
+  document. Export commands also offer explicit formats such as FASTA or JSONL.
+- Human tables preview 25 rows; `--limit 0` shows all rows. Machine exports are
+  complete regardless of the preview limit. Wrapping is only for human output.
+- Results go to stdout; progress and errors go to stderr. Proteome FASTA stays on
+  stdout and its build summary stays on stderr, as does the lift summary.
+- Discovery and help never acquire resources. Downloads and builds require an
+  explicit acquisition command and continue to enforce source terms and pins.
+- Errors identify the problem and return a status a script can branch on: 1 when
+  the command fails, 2 for a command-line mistake the parser or the command's own
+  checks catch (library code validates a few values itself, and those exit 1), and 3 when a
+  resource it needs isn't installed. Only command-line mistakes point to command
+  help; a missing installation names the command that installs it. Unexpected
+  internal errors keep their traceback rather than being reported as user errors.
+
+## Audit
+
+The command audit tests enumerate the parser tree, exercise help and bare groups,
+and run each leaf with offline fixtures. Acquisition tests use tiny files or
+replace the external acquisition boundary; they do not fetch production datasets
+or rebuild genome indexes. Export tests check structured payloads separately from
+human previews. This keeps the audit reproducible and small enough for CI.
+
+Every Tests workflow run attaches `cli-audit-python-VERSION` artifacts containing
+the actual help and command transcripts. Run the same audit locally with:
+
+```sh
+ORFDATA_CLI_AUDIT_DIR=/tmp/orfdata-cli-audit python -m pytest tests/test_cli_audit.py
+```
+
+The audit currently exercises 43 leaf commands in both human and JSON modes, plus
+the top-level guide, version, command aliases, nested help, errors, bounded
+previews, and complete machine exports. Its parser enumeration fails if a new
+command has no execution case. A tiny FASTA inventory file replaces the inventory
+download; other acquisitions use checksum-pinned fixtures and fixture viral
+records through the real handlers. These checks are not production dataset
+download or biological coverage validation.

orfdata-0.45.0/docs/cli.md

@@ -0,0 +1,142 @@
+# Command-line guide
+
+Run `orfdata` to see the command index and common starting points. It prints
+help and exits without reading a catalogue or acquiring data.
+
+```sh
+orfdata
+orfdata help data
+orfdata data list --species dog
+orfdata data status
+orfdata data info pfizer-atlas-dog-protein
+orfdata data fetch pfizer-atlas-dog-protein
+orfdata data path pfizer-atlas-dog-protein
+```
+
+`list` shows registered resources, including resources that need a local build.
+`status` shows resources with files in the selected local cache, including bundled
+data, damaged installs that `data fetch` will repair, and unreadable ones, whose
+permissions need fixing (or `--force` to replace them). Each damaged or unreadable
+row's exact fetch command is printed below the table. Neither command downloads
+anything. Use `--cache-root DIR` if your data
+lives outside the default cache, and `--verify` when you want to hash local files.
+
+## Finding commands
+
+`orfdata help COMMAND` and `orfdata COMMAND --help` are equivalent. Bare command
+groups such as `orfdata datasets` show their subcommands. `download` and
+`downloads` are aliases for `data`.
+
+| Command | Purpose | Subcommands |
+| --- | --- | --- |
+| `data` | Discover and acquire resources across all sources | `list`, `status`, `info`, `path`, `fetch`, `catalog`, `dataset`, `reference`, `genome`, `chains`, `viral` |
+| `query` | Preview or export ORF records | Use `--gene`, `--transcript`, `--orf-id`, `--species`, or `--catalog` |
+| `catalogs` | Manage optional ORF catalogues | `list`, `status`, `info`, `inspect`, `download`, `build`, `query` |
+| `datasets` | Manage quantitative measurements and their source references | `list`, `info`, `inspect`, `download`, `build`, `query`, `pairs`, `references`, `download-reference`, `inspect-reference`, `query-reference` |
+| `proteome` | Export a reference proteome as FASTA | Use `--species` and `--release`, or `--genome` |
+| `viral` | Inspect viral reference coverage and installed layers | `coverage`, `contexts`, `subsets`, `summary`, `completeness`, `inventory download` |
+| `setup` | Install a species' default catalogues, chains and collections | Requires `--species` |
+| `genomes` | Manage pinned annotations | `list`, `install` |
+| `chains` | Manage assembly mapping files | `list`, `download` |
+| `lift` | Write an ORF catalogue on another assembly | Requires `--to` and `--output` |
+
+`--cache-root DIR` works before or after a subcommand in every command group, so
+these are the same (`viral coverage`, `contexts` and `subsets` read packaged data
+only and refuse it):
+
+```sh
+orfdata datasets --cache-root /path/to/cache query pfizer-atlas-dog-protein
+orfdata datasets query pfizer-atlas-dog-protein --cache-root /path/to/cache
+```
+
+Every command that downloads takes `--timeout SECONDS`: how long a download may
+stall before it is retried, then fails (default 60; `--timeout none` waits
+indefinitely). The chromosome FASTA that `genomes install --genome-fasta` passes
+to PyEnsembl is downloaded without a timeout. The `--registry` option of
+`catalogs` and `datasets` stays before the subcommand:
+`orfdata catalogs --registry registry.json list`.
+
+## Human previews and complete exports
+
+Tables and short field summaries are the default. Tables show up to 25 entries;
+`--limit N` changes the preview and `--limit 0` shows every row. Tables wrap to
+terminal width. Metadata summaries omit bulky pins, processing details and long
+lists with a pointer to the full export.
+
+Use explicit formats in scripts:
+
+```sh
+orfdata data list --species dog --json
+orfdata catalogs info ncorf-mammals-mouse --json
+orfdata query --gene TPST1 --json
+orfdata query --species mouse --format fasta > mouse-ncorfs.fasta
+orfdata datasets query pfizer-atlas-dog-protein --unit iBAQ --format jsonl
+orfdata datasets pairs canine-mammary-stroma --basis same_specimen --json
+```
+
+`--json` emits one complete JSON document. Dataset `query` and `pairs` return
+JSON arrays; `--format jsonl` streams one original record per line. ORF queries
+also support `--format tsv`, `fasta`, and `gtf`. Viral coverage supports TSV.
+The preview limit never truncates a machine export.
+
+Measurement previews keep native units, scales and statuses, display values to
+six significant digits, and abbreviate long identifiers. Exports retain full
+identifiers and numeric precision. Reference
+labels distinguish scoped identifiers that happen to have the same spelling.
+Mapped measurements show the original observation and its mapping attribution;
+pair previews show their pairing basis, temporal relationship and target
+relationship. Full reference identities and provenance remain in JSON/JSONL.
+
+## Exit statuses
+
+| Status | Meaning |
+| --- | --- |
+| 0 | The command succeeded. |
+| 1 | The command failed: a download or checksum failed, local data or a file you passed is damaged or inconsistent, or a name the data is searched for isn't there (for example an unknown resource name or ORF ID). |
+| 2 | The command line is invalid, as the parser or the command's own checks find it: an unknown command or option, a missing argument, options that can't be combined, a restricted resource without `--accept-terms`, an out-of-range locus or length, or a value the command checks against the choices it lists (for example a dataset `--reference`). The message points to `--help`. |
+| 3 | A resource the command needs isn't installed. The message names the command that installs it, such as `orfdata data fetch NAME`. |
+
+Those checks run before anything is read, so the mistakes they catch exit 2
+whatever is installed. Some option values are checked only by the library code
+that uses them, and a bad one there exits 1 like any other failure: for example,
+an unknown `orfdata viral coverage --context` or `datasets pairs --basis`.
+
+In Python, a missing installation raises `orfdata.NotInstalledError`, which is
+also a `FileNotFoundError` and a `ValueError`. Up to 0.43, nearly every failure
+exited 2 (a partial `setup` exited 1); scripts that tested for 2 should test for
+a nonzero status, or for 3 before installing what is missing.
+
+## Output streams
+
+Results and exports go to stdout. Progress and errors go to stderr. `data path`
+and `viral inventory download` print a plain path by default. `proteome` writes
+FASTA to stdout unless `--output` is supplied; its summary goes to stderr.
+`lift` writes its catalogue to `--output` and also summarizes on stderr. For
+these two commands, `--json` selects the summary format on stderr.
+
+## Migrating scripts from 0.42 or 0.43
+
+Bare `orfdata` now shows help. Use `orfdata query --json` for the previous
+unfiltered JSON catalogue export. Existing root query flags (for example,
+`orfdata --gene TPST1`) still select the same records, with a human preview by
+default.
+
+Add `--json` to management commands or ORF queries whose JSON output your script
+parses. Add `--format jsonl` to dataset queries or paired-measurement exports to
+retain their previous streaming representation. Explicit FASTA, TSV and GTF
+exports retain their formats.
+
+Check exit statuses again: failures now exit 1, command-line mistakes 2 and a
+missing installation 3, where 0.42 and 0.43 exited 2 for nearly every failure (see
+[Exit statuses](#exit-statuses)). `orfdata data status` and
+`list_data(local=True)` now include damaged and unreadable installations, so a
+script that reads them as usable installations only should filter on
+`installed`.
+
+Stored data keeps its format, apart from viral collections: new installs go to a
+store that earlier versions don't read, and collections they installed still
+open. The [release notes](https://github.com/openvax/orfdata/blob/main/CHANGELOG.md)
+list every change, including those to Python APIs.
+
+The [CLI design and audit](cli-design.md) explains the conventions and the
+fixture-backed command audit that checks them in CI.

{orfdata-0.43.1 → orfdata-0.45.0}/docs/downloads.md

@@ -1,5 +1,6 @@
 # Discover and download data
 
+Run `orfdata` for the command guide, or `orfdata data` to see its subcommands.
 ORFdata has one common entry point for data acquisition:
 
 ```sh
@@ -24,12 +25,20 @@ metadata, including sizes and redistribution terms where declared. `path NAME`
 prints an installed file or directory for scripts; missing data is an error,
 and never triggers a download. Use `--verify` on reads to validate local bytes,
 `--cache-root DIR` to select a cache, and `--json` for machine output.
-`fetch --json` returns one result per distinct requested name.
+`fetch --json` returns one result per distinct requested name. Human tables
+preview 25 rows; `list` and `status` take those rows from each resource kind in
+turn, so every kind appears whenever there is room for one row of each (the
+default always has), and say what each kind left out. Use `--limit 0` for all
+rows. `--json` always returns complete
+results. See the [CLI guide](cli.md) for output formats and migration notes.
 
 Use `--kind` to disambiguate colliding resource names. For one resource,
 `--version VERSION` selects a historical release and `--source-path ASSET=PATH`
 provides already downloaded catalogue/dataset sources or genome files. Source
-overrides remain checksum-checked. `--force` installs a fresh generation.
+overrides remain checksum-checked. `fetch` reuses a valid installation and
+replaces one whose files fail verification, reporting it as `repaired`, as
+`setup` does; `--force` installs a fresh generation even over valid files, and is
+needed to replace files that can't be read.
 Restricted chain downloads require `--accept-terms` after reviewing `info`.
 
 The same operations are available in Python:
@@ -59,8 +68,12 @@ to every resource kind. Focused command groups such as `orfdata viral coverage`
 
 `list` shows every bundled, downloadable or locally buildable primary resource
 and its current state, including native protein search references declared by
-quantitative datasets. `status` shows resources already usable from local
-files. Filter either command with `--kind` or `--species`; use `--installed` or
+quantitative datasets. `status` shows resources with local files: usable ones
+and damaged or unreadable installs. `data fetch` replaces damaged files;
+unreadable ones are left alone until their permissions are fixed or `--force` is
+given. For catalogues, datasets, search references, genomes and chains, a repair
+installs a fresh generation and leaves the damaged one in place until the cache
+is pruned, as `--force` does; a viral collection is rebuilt in place. Filter either command with `--kind` or `--species`; use `--installed` or
 `--missing` with `list`, `--json` for programs, and `--verify` to hash installed
 files. Discovery and status commands never acquire or repair data.
 
@@ -94,6 +107,50 @@ command, and resolved cache path. Status values follow DataCache: `available`,
 an error rather than an empty listing, so a typo never reads as "nothing is
 available".
 
+## Interrupted downloads
+
+A published download restarts from the beginning by default. (A local build
+already keeps the source files it finished downloading, so rerunning it fetches
+only what is missing.) On an unreliable connection, add `--resume` to
+`orfdata data fetch` or `orfdata setup` (`resume=True` in `fetch_data` and
+`install_species`, or `resume="auto"` in a registry's download options).
+Rerunning the same command with `--resume` then requests only the bytes not yet
+received for published catalogues, genome and chain files, search references,
+a build's source files, and split archives such as the four parts of
+`pfizer-atlas-dog-peptide-evidence`.
+
+Resuming costs disk space, which is why it is off by default:
+
+- Partial downloads and their staging stay in the cache after a failure, in
+  owner-only `.staging-*` and `.datacache-resume-*` directories beside the
+  resource. Only a later successful `--resume` install of the same resource
+  removes them; an install without `--resume` uses its own staging and leaves
+  them. Delete them yourself to reclaim the space from a download you abandon.
+- An archive keeps every downloaded part until the dataset is installed. For
+  `pfizer-atlas-dog-peptide-evidence` that adds the 6.9 GB of parts to the peak
+  space the install needs, which is otherwise about 32 GB.
+- A finished resumed file is copied out of its partial before it is installed,
+  briefly needing twice that file's size.
+- An archive's resumable staging isn't named per user, so in a shared cache one
+  user's interrupted `--resume` install stops another user's `--resume` install
+  of the same dataset until it is removed. Installs without `--resume` aren't
+  affected.
+
+Datasets published as one compressed SQLite file can't resume: that is every
+published dataset except `pfizer-atlas-dog-peptide-evidence`, including the
+1.27 GB `pfizer-atlas-mouse-rna`. DataCache can't resume a transfer it
+decompresses while downloading, so with `--resume` these still restart, and
+publishing them in a resumable form is tracked in
+[#156](https://github.com/openvax/orfdata/issues/156). Sources that `setup`
+shares between catalogues are staged for one run and restart too. A viral
+collection is fetched record by record and doesn't resume. `--resume` is
+ignored when `--source-path` supplies the files.
+
+A resumed file is checked against the same pinned SHA-256 and size as a
+complete one. If it doesn't match, DataCache discards the partial and the
+install fails; rerun the same command, with `--resume`, to download that file
+again in full while keeping the other parts already received.
+
 ## Viral collections and contexts
 
 The Python API uses the same collection object for discovery, inventory reads,
@@ -294,3 +351,36 @@ the cached collection without downloading, and a damaged selection is rebuilt ra
 than reported, since nothing in it is unrecoverable. Adding
 `download_if_missing=True` to that call lets it repair the source collection too,
 which does download.
+
+A viral collection or selection is replaced the way DataCache replaces its own
+bundles. The refresh is built beside the installed copy and switched in with one
+rename while installs of the same collection wait their turn, so a reader sees
+the old files or the new ones, never a gap or a mix. Each refresh is a new
+directory: `orfdata data path gdc-viruses` prints the one holding the current
+files, so look it up again rather than saving it. Replaced copies are kept, as
+DataCache keeps its own, because a running program may still be reading them;
+nothing is pruned automatically. A collection installed by ORFdata 0.43 or
+earlier still opens, and is kept like any replaced copy after its first refresh.
+
+A collection is about 10 MB. To reclaim the space of replaced copies once no
+program uses them, run:
+
+```python
+from orfdata import prune_viral_references
+
+prune_viral_references(species="human", collection="gdc-viruses", keep=1)
+```
+
+This keeps the newest `keep` copies of the collection and of each of its
+selections, deletes the rest with the PyEnsembl indexes built for them under
+the same cache root, and returns `{"deleted": (...), "skipped": {path: reason}}`.
+A store that can't be read or doesn't validate is skipped untouched, with the
+reason, and the others are still pruned. A selection records the collection
+copy it was built from; after a refresh, asking for the same selection builds a
+new one from the current collection rather than reusing one built from the
+replaced copy.
+
+A forced install (`--force`, `force=True`) also repairs a store that lost its
+`.datacache-store.json` marker or whose newest copy isn't a directory. A path
+ORFdata didn't write in the store's place is never removed; the error names it
+so you can remove it yourself.

{orfdata-0.43.1 → orfdata-0.45.0}/docs/measurements.md

@@ -119,10 +119,11 @@ orfdata datasets inspect DATASET --version RELEASE
 orfdata datasets query DATASET --version RELEASE --quantity protein_abundance
 ```
 
-Queries stream JSON lines. `--reference ref:SHA256` selects an exact context;
+Queries show a human preview by default. Use `--format jsonl` to stream original
+JSON lines, or `--json` for one complete JSON array. `--reference ref:SHA256` selects an exact context;
 target queries also require `--target`, `--kind` and `--namespace`. Unqualified
-sample/assay ID queries across multiple references are rejected. Existing
-catalogue commands such as `orfdata --gene TPST1` retain their behavior.
+sample/assay ID queries across multiple references are rejected. ORF catalogue queries use `orfdata query --gene TPST1`; add `--json` for the
+complete structured export. See the [CLI migration guide](cli.md#migrating-scripts-from-042-or-043).
 
 ## What each value means
 

{orfdata-0.43.1 → orfdata-0.45.0}/docs/pairing.md

@@ -32,10 +32,11 @@ the same species. A protein FASTA does not need an invented genomic assembly:
 see [source references](source-references.md). Original targets and references
 remain attached to both sides, even when their local IDs happen to match.
 
-The CLI streams the same table rows:
+The CLI previews the pairing basis, reference scopes and target relationships.
+Export the same complete table rows with an explicit streaming format:
 
 ```sh
-orfdata datasets pairs DATASET --version RELEASE --basis same_material
+orfdata datasets pairs DATASET --version RELEASE --basis same_material --format jsonl
 ```
 
 Select exact targets with `left_target=...` and `right_target=...`, using the

{orfdata-0.43.1 → orfdata-0.45.0}/docs/proteome.md

@@ -857,8 +857,8 @@ option for several specific viruses.
 The FASTA goes to `--output` or
 standard output; `--snapshot FILE` also saves a snapshot, and `--cache`
 reuses or keeps the build in the cache; `--cache-root DIR` chooses the cache,
-for catalogues as well as builds. A
-JSON summary of the layers, identity and excluded ORFs goes to standard error.
+for catalogues as well as builds. A human summary goes to standard error; add
+`--json` for the complete JSON summary of layers, identity and excluded ORFs.
 
 ## Not yet included
 

{orfdata-0.43.1 → orfdata-0.45.0}/docs/viral.md

@@ -149,7 +149,10 @@ alongside the human catalogues and chains, and `--collections none` skips it.
 The command above installs only this checked collection under `ORFDATA_CACHE` (or
 the normal OpenVax cache), rather than asking for an export directory. Use
 `--cache-root DIRECTORY` to select another cache and `--force` to refresh or
-repair the installed files.
+repair the installed files. A refresh replaces the collection in one step and
+keeps the copies it replaced for programs still reading them, until
+`prune_viral_references()` removes them. Each refresh is a new directory, so
+look it up with `orfdata data path gdc-viruses` rather than saving it.
 
 Here `--species human` is the target analysis species. The current collection
 is for human transcriptome analysis; a future collection may target another
@@ -223,7 +226,7 @@ over every record in a downloaded collection:
 
 ```sh
 pip install 'orfdata[dev]'
-ORFDATA_GENBANK_CORPUS=<cache>/viral/references/homo_sapiens/gdc-viruses/<version>/genbank \
+ORFDATA_GENBANK_CORPUS="$(orfdata data path gdc-viruses)/records" \
   pytest tests/test_genbank_equivalence.py
 ```
 

{orfdata-0.43.1 → orfdata-0.45.0}/mkdocs.yml

@@ -19,6 +19,7 @@ extra_css:
   - stylesheets/docs.css
 nav:
   - Home: index.md
+  - Command-line guide: cli.md
   - Discover and download data: downloads.md
   - Downstream tools: downstream.md
   - Proteomes:
@@ -47,6 +48,7 @@ nav:
       - Dog MHC ligands and PeptideAtlas: dog-peptides.md
       - Dog paired-data and Ribo-seq source audit: dog-data-sources.md
   - Project:
+      - CLI design and audit: cli-design.md
       - Release notes: https://github.com/openvax/orfdata/blob/main/CHANGELOG.md
       - Data roadmap: https://github.com/openvax/orfdata/issues/3
 plugins:

{orfdata-0.43.1 → orfdata-0.45.0}/orfdata/__init__.py

@@ -21,6 +21,7 @@ from .data_resources import data_info, data_path, fetch_data
 from .dataset import QuantitativeDataset
 from .datasets import DatasetRegistry, build_dataset
 from .downloads import DataStatus, list_data
+from .errors import NotInstalledError
 from .evidence import MS, EvidenceRule, RiboSeq
 from .evidence_summary import EvidenceSummary
 from .genomes import PinnedGenome, install_genome, load_genome, pinned_genomes
@@ -96,6 +97,7 @@ from .viral_panel import (
     ViralReferenceFiles,
     download_viral_references,
     open_viral_references,
+    prune_viral_references,
     select_viral_references,
     viral_annotation_survey,
     viral_genome,
@@ -124,6 +126,7 @@ __all__ = [
     "ViralReferenceFiles",
     "download_viral_references",
     "open_viral_references",
+    "prune_viral_references",
     "select_viral_references",
     "viral_genome",
     "viral_annotation_survey",
@@ -180,6 +183,7 @@ __all__ = [
     "data_info",
     "data_path",
     "fetch_data",
+    "NotInstalledError",
     "DatasetManifest",
     "SourceAsset",
     "Sample",

{orfdata-0.43.1 → orfdata-0.45.0}/orfdata/catalog.py

@@ -9,6 +9,7 @@ import hashlib
 import json
 import os
 import re
+import zlib
 from collections import Counter
 from copy import copy, deepcopy
 from dataclasses import asdict, dataclass, field, replace
@@ -17,6 +18,7 @@ from pathlib import Path
 from typing import NamedTuple, Optional
 from urllib.parse import quote
 
+from .errors import NotInstalledError
 from .evidence import EVIDENCE_METHODS as EVIDENCE_METHODS  # Re-exported for existing imports.
 from .evidence import MS_METHODS, evidence_rule, stored_method
 from .evidence_summary import EvidenceSummary, unique_summaries
@@ -915,23 +917,52 @@ def _load_catalog(path=None, *, expected_sha256=None, resource="orf-evidence"):
             raise ValueError("Packaged uORF catalogue checksum mismatch")
     if expected_sha256 is not None and hashlib.sha256(raw).hexdigest() != expected_sha256:
         raise ValueError("uORF catalogue checksum mismatch")
+    try:
+        records, metadata = _parse_catalog(raw)
+    except _MALFORMED as error:
+        if path is None:
+            raise
+        raise _invalid_catalog(path, error) from error
+    try:
+        return ORFCatalog(records, metadata)
+    except (KeyError, ValueError) as error:  # Metadata the file lacks or gets wrong; anything else is a bug.
+        if path is None:
+            raise
+        raise _invalid_catalog(path, error) from error
+
+
+# What reading a damaged or malformed catalogue file raises: truncated or corrupt
+# gzip, undecodable bytes, invalid JSON, and fields that are missing or of the wrong type.
+_MALFORMED = (ValueError, KeyError, TypeError, EOFError, zlib.error, gzip.BadGzipFile)
+
+
+def _invalid_catalog(path, error):
+    detail = str(error) if isinstance(error, ValueError) else f"{type(error).__name__}: {error}"
+    return ValueError(f"{path} is not a valid ORF catalogue: {detail}")
+
+
+def _parse_catalog(raw):
+    """A catalogue file's records and metadata, checking its shape."""
     if raw.startswith(b"\x1f\x8b"):
         raw = gzip.decompress(raw)
     data = json.loads(raw)
+    if not isinstance(data, dict):
+        raise ValueError("the file is not a JSON object")
     if data.get("schema") not in ("isovar.uorf_evidence.v1", "orfdata.catalog.v1"):
         raise ValueError("Unsupported uORF evidence schema")
-    if data["metadata"].get("coordinates") != "1-based-inclusive":
+    metadata = data["metadata"]
+    if not isinstance(metadata, dict) or not isinstance(data["records"], list):
+        raise ValueError("metadata must be an object and records a list")
+    if metadata.get("coordinates") != "1-based-inclusive":
         raise ValueError("Unsupported uORF coordinate convention")
     legacy = data["schema"] == "isovar.uorf_evidence.v1"
-    metadata = data["metadata"]
     if legacy:
         if metadata.get("genome_build") != "GRCh38":
             raise ValueError("Legacy Isovar data are explicitly human GRCh38 only")
         metadata = dict(metadata, taxon_id=9606, species="homo_sapiens")
     elif not metadata.get("sources"):
         raise ValueError("Generic catalogues require source provenance")
-    records = [_record(row, legacy) for row in data["records"]]
-    return ORFCatalog(records, metadata)
+    return [_record(row, legacy) for row in data["records"]], metadata
 
 
 def _record(row, legacy=False):
@@ -1079,7 +1110,8 @@ def load_catalog(path=None, *, species=None, genome_build=None, registry=None, e
                          "lifted or merged from several")
     # Everything missing is reported at once: catalogues to install, then chains to download.
     root = getattr(registry, "_cache_root", None)
-    needs = [_install_hint(c) for c, _ in plan if c.registry is not None and _installed(c) in (None, "damaged")]
+    states = [(c, _installed(c)) for c, _ in plan if c.registry is not None]  # Each inspected once.
+    needs = [_install_hint(c, state) for c, state in states if state in (None, "damaged")]
     chains = list(dict.fromkeys(c for _, route in plan for c in route))
     missing = [c for c in chains if not chain_downloaded(c, cache_root=root)]
     cli_root = "" if root is None else f" --cache-root {root}"
@@ -1093,9 +1125,11 @@ def load_catalog(path=None, *, species=None, genome_build=None, registry=None, e
         accept = " --accept-terms" if any(c.restricted for c in missing) else ""
         chosen = ("" if catalogs is None else " --catalogs " + (
             catalogs if isinstance(catalogs, str) else ",".join(catalogs)))
-        raise ValueError(f"{species} {genome_build} ORFs need " + "; and ".join(needs)
-                         + f". Or install everything at once with 'orfdata setup --species {species}{chosen}{accept}"
-                         f"{cli_root}'")
+        # Missing catalogues or chains are a missing installation; a damaged one is a failure to repair.
+        error = ValueError if any(state == "damaged" for _, state in states) else NotInstalledError
+        raise error(f"{species} {genome_build} ORFs need " + "; and ".join(needs)
+                    + f". Or install everything at once with 'orfdata setup --species {species}{chosen}{accept}"
+                    f"{cli_root}'")
     # Lifting and merging take seconds; one process reuses the result for exactly the same inputs.
     key = (species, genome_build, expected_sha256, tuple((c.name, c.version, c.resource, _installed_at(c), tuple(
         x.sha256 for x in route)) for c, route in plan))
@@ -1205,18 +1239,23 @@ def _plan(found, species, genome_build):
                                           item[0].priority, item[0].name))
 
 
-def _install_hint(choice):
-    """The command that installs a registered release, or repairs a damaged one, in the registry's cache."""
+def _how_to_install(choice, state):
+    """How to install a registered release, or repair a damaged one, in the registry's cache."""
+    from .downloads import install_hint
+
     registry, name, version = choice.registry, choice.name, choice.version
-    root = getattr(registry, "_cache_root", None)
-    damaged = _installed(choice) == "damaged"
+    damaged = state == "damaged"
     verb = "download" if registry.published(name, version) else "build"
-    how = " again" if damaged else (" from its pinned sources" if verb == "build" else "")
-    return (f"the {name} {version} catalogue, which {'is damaged' if damaged else 'is not installed'}: {verb} it"
-            f"{how} with CatalogRegistry({'' if root is None else f'cache_root={str(root)!r}'}).{verb}({name!r}, "
-            f"{version!r}{', force=True' if damaged else ''}) or 'orfdata catalogs"
-            f"{'' if root is None else f' --cache-root {root}'} {verb} {name} --version {version}"
-            f"{' --force' if damaged else ''}'")
+    how = " again" if damaged else ("" if verb == "download" else " from its pinned sources")
+    call = f"{verb}({name!r}, {version!r}{', force=True' if damaged else ''})"
+    return f"{verb} it{how} with " + install_hint(registry, call, name, "catalog", version, force=damaged)
+
+
+def _install_hint(choice, state):
+    """The catalogue a species' ORFs need, and how to install or repair it."""
+    damaged = state == "damaged"
+    return (f"the {choice.name} {choice.version} catalogue, which {'is damaged' if damaged else 'is not installed'}: "
+            + _how_to_install(choice, state))
 
 
 def _open_choice(choice, expected_sha256):
@@ -1283,13 +1322,8 @@ def _open_registered(choice, expected_sha256):
     registry, name, version = choice.registry, choice.name, choice.version
     installed = _installed(choice)
     if installed in (None, "damaged"):
-        verb = "download" if registry.published(name, version) else "build"
-        when = " again" if installed else (" first" if verb == "download" else " from its pinned sources")
-        how = (f"{verb} it{when} with CatalogRegistry().{verb}({name!r}, {version!r}"
-               f"{', force=True' if installed else ''}) or 'orfdata catalogs {verb} {name} --version {version}"
-               f"{' --force' if installed else ''}'")
-        state = "damaged" if installed else "not installed"
-        raise ValueError(f"The {name} {version} ORF catalogue is {state}; {how}")
+        state, error = ("damaged", ValueError) if installed else ("not installed", NotInstalledError)
+        raise error(f"The {name} {version} ORF catalogue is {state}; {_how_to_install(choice, installed)}")
     if installed == "built":
         inspected = registry.inspect_build(name, version)
         if inspected.status != "available":