orfdata 0.16.0__tar.gz → 0.18.0__tar.gz

{orfdata-0.16.0/orfdata.egg-info → orfdata-0.18.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: orfdata
-Version: 0.16.0
+Version: 0.18.0
 Summary: Reference open reading frames and experimental translation evidence on annotated transcripts
 Author: OpenVax
 License-Expression: Apache-2.0
@@ -99,6 +99,9 @@ checks. **The bundled experimental ORF catalogue is currently human-only**. Offl
 bind real human, mouse, dog and yeast reference slices; those fixtures are not
 new experimental ORF catalogues. [Species and reference identity](docs/species.md).
 
+[Optional catalogue management](docs/catalogs.md) provides explicit, versioned
+downloads and source builds while returning the same ORF objects and queries.
+
 Install with `pip install orfdata[pyensembl]`. See the
 [measurement API](docs/measurements.md) for native units, explicit samples and
 assays, and DataCache-managed downloads. Downloadable

{orfdata-0.16.0 → orfdata-0.18.0}/README.md

@@ -72,6 +72,9 @@ checks. **The bundled experimental ORF catalogue is currently human-only**. Offl
 bind real human, mouse, dog and yeast reference slices; those fixtures are not
 new experimental ORF catalogues. [Species and reference identity](docs/species.md).
 
+[Optional catalogue management](docs/catalogs.md) provides explicit, versioned
+downloads and source builds while returning the same ORF objects and queries.
+
 Install with `pip install orfdata[pyensembl]`. See the
 [measurement API](docs/measurements.md) for native units, explicit samples and
 assays, and DataCache-managed downloads. Downloadable

orfdata-0.18.0/docs/catalogs.md

@@ -0,0 +1,89 @@
+# Optional ORF catalogues
+
+`CatalogRegistry` manages versioned ORF catalogues with the same explicit
+discovery, download, build and offline-open workflow as `DatasetRegistry`.
+It returns the existing `ORFCatalog`; query results still bind to the original
+PyEnsembl transcript through `ReferenceORF`.
+
+The default optional registry currently has no released entries. The bundled
+human catalogue remains available through `load_catalog()` without downloads.
+The examples below use a caller-supplied registry; they do not imply a published
+experimental catalogue for another species.
+
+```python
+import json
+from orfdata import CatalogRegistry
+
+with open("catalog-registry.json") as handle:
+    registry = CatalogRegistry(json.load(handle))
+releases = registry.list(species="yeast")
+for manifest in releases:
+    print(manifest.catalog_id, manifest.version, manifest.coverage)
+
+# Select an entry from your populated registry.
+name = releases[0].catalog_id
+registry.download(name)             # Explicit, checksum-verified acquisition.
+catalog = registry.open(name)       # Offline; no implicit download or repair.
+records = catalog.query(evidence="ms", require_resolved=True)
+orfs = catalog.for_transcript(transcript)  # An already loaded PyEnsembl transcript.
+```
+
+Use `info(name)` for immutable provenance and `download_metadata(name)` for
+artifact URLs, checksums and access terms. `inspect(name)` checks the local
+installation. Species, strain, assembly and annotation contexts remain attached
+to accepted records; a matching accession alone cannot transfer evidence.
+
+```sh
+orfdata catalogs --registry catalog-registry.json list --species yeast
+orfdata catalogs --registry catalog-registry.json download NAME
+orfdata catalogs --registry catalog-registry.json query NAME --evidence ms --require-resolved
+```
+
+Queries share the ordinary ORF CLI's gene, transcript, genomic-position and
+evidence filters, and JSON/TSV/FASTA exports. They cannot override the selected
+release with a local `--catalog` argument. Use `orfdata --catalog FILE` for the
+existing standalone local-file workflow.
+
+## Source builds
+
+`registry.build(name, source_paths={"source.json": path})` explicitly rebuilds
+an allowlisted source adapter. `registry.open_build(name)` opens that local
+result, and `inspect_build(name)` checks its input pins, transform identity and
+output receipt without reopening original input files. The CLI equivalents are
+`catalogs build NAME --source-path source.json=PATH` and
+`catalogs query NAME --built`. A local receipt's checksum is distinct from an
+independently pinned published artifact; `inspect_build().verified` remains false.
+
+Builds are atomic. `force=True` creates a fresh bundle while existing readers'
+files survive. Cache roots follow `ORFDATA_CACHE`, `OPENVAX_DATA_CACHE`, or the
+explicit `cache_root`; catalogues, quantitative datasets and local source builds
+use separate locations.
+
+## Defining a release
+
+A registry definition maps a name to `default_version` and `versions`. Each
+version contains a `CatalogManifest`, `distribution` access terms and an
+`assets` entry for raw `catalog.json.gz` bytes, pinned by URL, SHA-256 and size.
+The manifest declares references, original `SourceAsset` inputs, citations,
+processing and coverage. `coverage.records` is required; source importers also
+report original assignment counts and unresolved categories. Distinct protein
+sequences and evidence summaries are separate counts.
+
+Managed files use the existing `orfdata.catalog.v1` JSON representation and embed
+the full manifest in `metadata.catalog_manifest`. Every accepted record must
+declare a complete genomic reference included in that manifest. Opening validates
+the whole manifest, species/assembly, references and declared output counts.
+
+The initial `catalog-copy-v1` adapter wraps an independently prepared, unmanaged, pinned
+portable catalogue named by `processing.catalog_asset`; it preserves native
+metadata and adds the managed manifest. It rejects an input that already has a
+managed manifest instead of replacing its release provenance. It does not infer missing references or
+validate new biological models on behalf of a source importer. Python developers
+can use `build_catalog(manifest, builder)`, where `builder(inputs, output_path)`
+writes the managed file. DataCache verifies inputs and validates the completed
+output before publishing the local build. Importer paths in metadata are never
+executed as Python code.
+
+[OpenProt source qualification](https://github.com/openvax/orfdata/issues/18)
+and its [coordinate discrepancy](https://github.com/openvax/orfdata/issues/55)
+remain separate from this data-management API.

{orfdata-0.16.0 → orfdata-0.18.0}/docs/reference.md

@@ -12,7 +12,7 @@ PyEnsembl. Binding uses a transcript you have already loaded.
 ORF. Each has:
 
 - `model`: source identity, ORF type, coordinates, species and protein sequence.
-- `evidence`: ribosome observations and peptide-level MS observations.
+- `evidence`: ribosome observations, peptide-level MS and native aggregate support.
 - `transcript`: the original PyEnsembl transcript, including its original biotype.
 - `coding_sequence` and `protein_sequence`: checked against that transcript.
 
@@ -46,10 +46,44 @@ possible protein. Inspect `orf.evidence.peptides` for peptide sequences, source
 accessions, matching intervals, mapping counts and review quality. Missing counts
 remain unknown; low or mixed spectrum reviews do not count as supporting MS.
 
+Some sources provide only aggregate support counts. `EvidenceSummary` keeps the
+native metric, definition, value, source/release, source checksum and explicitly
+scoped entity. For example, [OpenProt's native metrics](https://api.openprot.org/api/2.0/SC/downloads/yeast_s288c-openprot-2_2-altprots+isoforms-min_2_pep-R64-1-1.106.tsv.readme.html)
+include an MS score that sums study-specific unique-peptide
+counts; it is neither global peptide uniqueness nor protein abundance. Its
+translation score counts studies with significant translation detection.
+`None` means unknown and remains distinct from a reported zero.
+
+Positive summaries in `orf.evidence.summaries` can satisfy general evidence
+queries. To require actual peptide records or named ribosome studies, use:
+
+```python
+orf.evidence.has_evidence("ms", require_resolved=True)
+catalog.query(evidence="ribosome", require_resolved=True)
+```
+
+The CLI uses `--evidence ms --require-resolved`. Uniqueness and review filters
+always require resolved peptide observations. Summaries never create peptide
+sequences, residue coverage, retained variant peptides or mutant-translation
+evidence. Their protein checksum and species/reference checks prevent attaching
+them to a different protein or incompatible source context.
+
+A sequence-level summary may have several transcript assignments. Those records
+share the summary's `key`; `catalog.evidence_summaries` returns each native metric
+once and rejects conflicting facts for the same identity. Native accession
+aliases must be resolved explicitly by the source adapter. Counts from different
+metrics and sources are not summed, and a later peptide-resolved snapshot is not
+an additional independent experiment merely because it has more detail.
+
+This API includes source-backed OpenProt human, mouse and yeast test slices.
+It does not yet publish those species' OpenProt ORF catalogues; coordinate and
+sequence qualification remains tracked in [#18](https://github.com/openvax/orfdata/issues/18).
+
 The distinction between translation, peptide detection and functional protein
 evidence follows the [TransCODE study](https://www.nature.com/articles/s41586-026-10459-x).
-Source evidence belongs to the exact reference ORF. It is not automatically
-evidence for a different isoform, a mutant protein, a fusion or another species.
+Source evidence stays at its declared ORF or protein-sequence scope. It is not
+automatically evidence for another isoform, a mutant protein, a fusion or
+another species.
 
 ## Variants and RNA products
 

{orfdata-0.16.0 → orfdata-0.18.0}/orfdata/__init__.py

@@ -12,8 +12,11 @@ from .catalog import (
     load_catalog,
     load_uorf_catalog,
 )
+from .catalog_manifest import CatalogManifest
+from .catalogs import CatalogRegistry, build_catalog
 from .dataset import QuantitativeDataset
 from .datasets import DatasetRegistry, build_dataset
+from .evidence_summary import EvidenceSummary
 from .initiation_context import KOZAK_PREFERENCES, InitiationContext, annotate_initiation_context
 from .measurements import (
     Assay,
@@ -44,6 +47,10 @@ __all__ = [
     "ORFCatalog",
     "ORFRecord",
     "ORFEvidence",
+    "CatalogManifest",
+    "CatalogRegistry",
+    "build_catalog",
+    "EvidenceSummary",
     "PeptideEvidence",
     "ORFModel",
     "ReferenceORF",

{orfdata-0.16.0 → orfdata-0.18.0}/orfdata/catalog.py

@@ -13,6 +13,7 @@ from importlib.resources import files
 from pathlib import Path
 from typing import Optional
 
+from .evidence_summary import EvidenceSummary, unique_summaries
 from .initiation_context import KOZAK_PREFERENCES, InitiationContext
 from .model import ORFModel, ReferenceORF, _species
 from .reference import ReferenceContext
@@ -73,25 +74,41 @@ class ORFEvidence:
 
     ribosome_studies: tuple = ()
     peptides: tuple = ()
+    summaries: tuple = ()
+
+    def __post_init__(self):
+        object.__setattr__(self, "summaries", unique_summaries(self.summaries))
 
     def to_dict(self):
-        return asdict(self)
+        result = asdict(self)
+        if not self.summaries:
+            result.pop("summaries")
+        return result
 
-    def has_evidence(self, method, *, require_unique=False, require_reviewed=False):
-        """Select reference observations by assay, without clinical promotion."""
+    def has_evidence(self, method, *, require_unique=False, require_reviewed=False, require_resolved=False):
+        """Select reported support, or require named studies/resolved peptides.
+
+        Aggregate counts never satisfy uniqueness or spectrum-review filters.
+        No selection establishes evidence for a modified protein product.
+        """
         if method not in EVIDENCE_METHODS:
             raise ValueError(f"Unknown evidence method: {method}")
         if method == "ribosome":
             if require_unique or require_reviewed:
                 raise ValueError("MS filters do not apply to ribosome evidence")
-            return bool(self.ribosome_studies)
-        return any(
+            return bool(self.ribosome_studies) or (not require_resolved and any(
+                s.method == "ribosome" and s.is_supporting for s in self.summaries))
+        resolved = any(
             (method == "ms" or p.method == method or (method == "mhc" and p.method == "hla"))
             and p.is_supporting
             and (not require_unique or p.unique_in_source)
             and (not require_reviewed or p.quality == "reviewed_support")
             for p in self.peptides
         )
+        return resolved or (not (require_resolved or require_unique or require_reviewed) and any(
+            (method == "ms" and s.method != "ribosome"
+             or s.method == method or method == "mhc" and s.method == "hla")
+            and s.is_supporting for s in self.summaries))
 
 
 @dataclass(frozen=True)
@@ -116,6 +133,14 @@ class ORFRecord:
     translation_table: int = 1
     source: str = ""
     reference_context: Optional[ReferenceContext] = None
+    summaries: tuple = ()
+
+    def __post_init__(self):
+        object.__setattr__(self, "summaries", unique_summaries(self.summaries))
+        if self.summaries:
+            model = self.model
+            for summary in self.summaries:
+                summary.validate_orf(model)
 
     @property
     def model(self):
@@ -140,7 +165,7 @@ class ORFRecord:
 
     @property
     def evidence(self):
-        return ORFEvidence(self.ribosome_studies, self.peptides)
+        return ORFEvidence(self.ribosome_studies, self.peptides, self.summaries)
 
     @classmethod
     def from_model(cls, model, evidence=None, *, initiation_context=None):
@@ -154,6 +179,7 @@ class ORFRecord:
             **fields,
             ribosome_studies=evidence.ribosome_studies,
             peptides=evidence.peptides,
+            summaries=evidence.summaries,
             initiation_context=initiation_context,
         )
 
@@ -164,6 +190,8 @@ class ORFRecord:
     def to_dict(self):
         """Return a serializable copy of this record."""
         result = asdict(self)
+        if not self.summaries:
+            result.pop("summaries")
         if self.reference_context is None:
             result.pop("reference_context")
         if self.initiation_context is not None:
@@ -284,9 +312,10 @@ class ORFRecord:
             translation_change="not_measured",
         )
 
-    def has_evidence(self, method, *, require_unique=False, require_reviewed=False):
+    def has_evidence(self, method, *, require_unique=False, require_reviewed=False, require_resolved=False):
         return self.evidence.has_evidence(
-            method, require_unique=require_unique, require_reviewed=require_reviewed
+            method, require_unique=require_unique, require_reviewed=require_reviewed,
+            require_resolved=require_resolved,
         )
 
     def _coding_offset(self, position):
@@ -423,6 +452,13 @@ class ORFCatalog:
                     ):
                         raise ValueError(f"Peptide sequence does not match {r.orf_id}")
 
+        self._evidence_summaries = unique_summaries(s for r in self.records for s in r.summaries)
+
+    @property
+    def evidence_summaries(self):
+        """Distinct source counts, without multiplying transcript assignments."""
+        return self._evidence_summaries
+
     @property
     def metadata(self):
         """Return a copy of dataset version, sources, scope and provenance."""
@@ -472,6 +508,7 @@ class ORFCatalog:
         evidence=None,
         require_unique=False,
         require_reviewed=False,
+        require_resolved=False,
         biotype=None,
         include_initiation_context=False,
     ):
@@ -479,10 +516,12 @@ class ORFCatalog:
 
         A locus query requires an explicit assembly. Gene and transcript lookup
         is a discovery aid, not evidence transfer between transcript isoforms.
-        Transcript identifiers match exactly as supplied in the source; these
-        are unversioned identifiers. Empty results do not prove no translation.
+        Transcript identifiers match exactly as supplied in the source,
+        including versions when present. Empty results do not prove no translation.
         ``require_unique`` means unique within the evidence source's mapping
         scope, not unique in every normal human proteome.
+        ``require_resolved`` excludes aggregate-only support: MS needs a peptide
+        observation and ribosome evidence needs a named study.
         ``include_initiation_context`` also intersects placed spliced start
         flanks, for example an upstream Kozak SNV. The default remains coding
         blocks only. It does not include introns between context bases.
@@ -504,6 +543,8 @@ class ORFCatalog:
             raise ValueError("require_unique applies to MS evidence only")
         if require_reviewed and evidence not in EVIDENCE_METHODS[1:]:
             raise ValueError("require_reviewed applies to MS evidence only")
+        if require_resolved and evidence is None:
+            raise ValueError("require_resolved requires an evidence method")
 
         return tuple(
             r
@@ -530,7 +571,8 @@ class ORFCatalog:
             and (
                 evidence is None
                 or r.has_evidence(
-                    evidence, require_unique=require_unique, require_reviewed=require_reviewed
+                    evidence, require_unique=require_unique, require_reviewed=require_reviewed,
+                    require_resolved=require_resolved,
                 )
             )
         )
@@ -604,6 +646,7 @@ def _load_catalog(path=None, *, expected_sha256=None, resource="orf-evidence"):
             p["spectrum_ids"] = tuple(p.get("spectrum_ids", ()))
             peptides.append(PeptideEvidence(**p))
         row["peptides"] = tuple(peptides)
+        row["summaries"] = tuple(EvidenceSummary.from_dict(s) for s in row.get("summaries", ()))
         records.append(ORFRecord(**row))
     return ORFCatalog(records, metadata)
 

orfdata-0.18.0/orfdata/catalog_cli.py

@@ -0,0 +1,63 @@
+"""Explicit optional catalogue discovery, acquisition and offline queries."""
+
+import argparse
+import json
+from pathlib import Path
+
+from .catalogs import CatalogRegistry
+from .cli import _add_query_arguments, _query
+
+
+def run(argv=None, prog=None):
+    parser = argparse.ArgumentParser(description=__doc__, prog=prog)
+    parser.add_argument("--registry", help="Local JSON catalogue registry definitions")
+    parser.add_argument("--cache-root", help="Override the OpenVax cache root")
+    commands = parser.add_subparsers(dest="command", required=True)
+    listing = commands.add_parser("list", help="Discover release manifests without downloading")
+    listing.add_argument("--species")
+    listing.add_argument("--taxon-id", type=int)
+    query_parser = None
+    for command in ("info", "inspect", "download", "build", "query"):
+        subparser = commands.add_parser(command)
+        subparser.add_argument("name")
+        subparser.add_argument("--version")
+        if command in ("inspect", "query"):
+            subparser.add_argument("--built", action="store_true", help="Select a local source build explicitly")
+        if command in ("download", "build"):
+            subparser.add_argument("--force", action="store_true")
+            subparser.add_argument("--source-path", action="append", default=[], metavar="ASSET=PATH")
+        if command == "query":
+            query_parser = subparser
+            _add_query_arguments(subparser, catalog_path=False)
+    args = parser.parse_args(argv)
+    try:
+        definitions = json.loads(Path(args.registry).read_text()) if args.registry else None
+        registry = CatalogRegistry(definitions, cache_root=args.cache_root)
+        if args.command == "list":
+            output = [m.to_dict() for m in registry.list(species=args.species, taxon_id=args.taxon_id)]
+        elif args.command == "info":
+            output = registry.info(args.name, args.version).to_dict()
+        elif args.command == "inspect":
+            method = registry.inspect_build if args.built else registry.inspect
+            inspection = method(args.name, args.version)
+            output = dict(status=inspection.status, path=str(inspection.path), verified=inspection.verified)
+            if inspection.error is not None:
+                output["error"] = str(inspection.error)
+        elif args.command in ("download", "build"):
+            paths = {}
+            for item in args.source_path:
+                asset, separator, path = item.partition("=")
+                if not separator or not asset or not path or asset in paths:
+                    raise ValueError("Source overrides require unique ASSET=PATH entries")
+                paths[asset] = path
+            method = registry.build if args.command == "build" else registry.download
+            method(args.name, args.version, force=args.force, source_paths=paths or None)
+            inspect = registry.inspect_build if args.command == "build" else registry.inspect
+            result = inspect(args.name, args.version)
+            output = dict(status=result.status, path=str(result.path), verified=result.verified)
+        else:
+            method = registry.open_build if args.built else registry.open
+            return _query(args, query_parser, lambda _: method(args.name, args.version))
+    except (ValueError, TypeError, KeyError, OSError) as error:
+        parser.error(str(error))
+    print(json.dumps(output, indent=2, sort_keys=True))

orfdata-0.18.0/orfdata/catalog_manifest.py

@@ -0,0 +1,101 @@
+"""Pinned provenance and coverage for an optional ORF catalogue."""
+
+import re
+from collections.abc import Mapping
+from dataclasses import dataclass
+
+from .measurements import SourceAsset, _identifier, _metadata, _Record
+from .reference import ReferenceContext
+
+
+@dataclass(frozen=True)
+class CatalogManifest(_Record):
+    """One species/assembly catalogue release, independent of abundance data."""
+
+    catalog_id: str
+    version: str
+    title: str
+    references: tuple
+    sources: tuple
+    citations: tuple
+    coverage: Mapping
+    processing: Mapping
+    schema: str = "orfdata.catalog-manifest.v1"
+
+    @classmethod
+    def from_dict(cls, data):
+        values = dict(data)
+        values["references"] = tuple(ReferenceContext(**c) for c in values["references"])
+        values["sources"] = tuple(SourceAsset.from_dict(s) for s in values["sources"])
+        return cls(**values)
+
+    def __post_init__(self):
+        for name in ("catalog_id", "version", "title"):
+            _identifier(getattr(self, name), name)
+        for name in ("catalog_id", "version"):
+            if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]*", getattr(self, name)):
+                raise ValueError(f"{name} must be a portable catalogue/path identifier")
+        if self.schema != "orfdata.catalog-manifest.v1":
+            raise ValueError("Unsupported catalogue manifest schema")
+        for name in ("references", "sources", "citations"):
+            values = getattr(self, name)
+            if isinstance(values, str):
+                raise TypeError(f"Catalogue {name} must be a sequence of entries")
+            values = tuple(values)
+            if not values:
+                raise ValueError(f"Catalogue {name} cannot be empty")
+            object.__setattr__(self, name, values)
+        if any(not isinstance(r, ReferenceContext) for r in self.references):
+            raise TypeError("Catalogue references must be ReferenceContexts")
+        if len({r.key for r in self.references}) != len(self.references):
+            raise ValueError("Duplicate catalogue reference contexts")
+        if len({r.taxon_id for r in self.references}) != 1:
+            raise ValueError("A catalogue must have one species")
+        builds = {r.genome_build for r in self.references if r.genome_build is not None}
+        if len(builds) != 1 or not any(r.is_genomic for r in self.references):
+            raise ValueError("A catalogue must have one assembly and a complete genomic reference")
+        if any(not isinstance(s, SourceAsset) for s in self.sources):
+            raise TypeError("Catalogue sources must be SourceAssets")
+        if len({s.asset_id for s in self.sources}) != len(self.sources):
+            raise ValueError("Duplicate catalogue source assets")
+        for citation in self.citations:
+            _identifier(citation, "citation")
+        _metadata(self, "coverage", "processing")
+        count = self.coverage.get("records")
+        if type(count) is not int or count < 0:
+            raise ValueError("Catalogue coverage must declare a nonnegative records count")
+        for key in ("distinct_protein_sequences", "evidence_summaries"):
+            if key in self.coverage and (type(self.coverage[key]) is not int or self.coverage[key] < 0):
+                raise ValueError(f"Catalogue {key} must be a nonnegative integer")
+        if not self.processing:
+            raise ValueError("Catalogue processing provenance is required")
+
+    @property
+    def taxon_id(self):
+        return self.references[0].taxon_id
+
+    @property
+    def genome_build(self):
+        return next(r.genome_build for r in self.references if r.is_genomic)
+
+    def validate_catalog(self, catalog):
+        """Check content identity, accepted records and declared reference scope."""
+        if catalog.metadata.get("catalog_manifest") != self.to_dict():
+            raise ValueError("Catalogue manifest disagrees with the registered release")
+        if catalog.taxon_id != self.taxon_id or catalog.genome_build != self.genome_build:
+            raise ValueError("Catalogue species or assembly disagrees with manifest")
+        if len(catalog.records) != self.coverage["records"]:
+            raise ValueError("Catalogue record count disagrees with manifest")
+        references = {r.key for r in self.references}
+        for record in catalog.records:
+            reference = record.reference_context
+            if reference is None or not reference.is_genomic or reference.key not in references:
+                raise ValueError("Managed ORF records require a declared complete genomic reference")
+        for summary in catalog.evidence_summaries:
+            if summary.target.reference.key not in references:
+                raise ValueError("Evidence summary reference is absent from catalogue manifest")
+        counts = dict(distinct_protein_sequences=len({r.protein_sequence for r in catalog.records}),
+                      evidence_summaries=len(catalog.evidence_summaries))
+        for key, actual in counts.items():
+            if key in self.coverage and self.coverage[key] != actual:
+                raise ValueError(f"Catalogue {key} count disagrees with manifest")