orfdata 0.42.2__tar.gz → 0.42.4__tar.gz

{orfdata-0.42.2 → orfdata-0.42.4}/MANIFEST.in

@@ -15,3 +15,4 @@ recursive-include tests/data/ncorf_mammals *.bed *.fa *.xlsx *.gz *.md
 recursive-include tests/data/riborf *.xlsx *.gz *.md
 recursive-include tests/data/kaabinejadian *.zip *.tsv
 recursive-include tests/data/peptideatlas *.zip *.gz
+recursive-include tests/data/mammary_stroma *.gz *.txt *.xml

{orfdata-0.42.2/orfdata.egg-info → orfdata-0.42.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: orfdata
-Version: 0.42.2
+Version: 0.42.4
 Summary: Reference proteomes with evidence-backed noncanonical ORFs on PyEnsembl genomes
 Author: OpenVax
 License-Expression: Apache-2.0

{orfdata-0.42.2 → orfdata-0.42.4}/docs/catalogs.md

@@ -82,29 +82,19 @@ pip install "orfdata"
 orfdata setup --species mouse --accept-terms   # Three catalogues and the chains
 ```
 
-`orfdata setup` downloads the published Chang et al. catalogue, builds the other
-registered mouse catalogues from their pinned sources, and downloads the chain files (`--accept-terms` for UCSC's non-commercial
+`orfdata setup` downloads the published Chang et al. and RibORF catalogues, builds OpenProt from their pinned sources, and downloads the chain files (`--accept-terms` for UCSC's non-commercial
 GRCm38/GRCm39 chains). It prints what it did with each: built, downloaded,
 already installed, repaired or skipped. If one fails, the rest are still
 installed and it exits with status 1, naming each failure; in Python,
 `install_species` raises `InstallError`, whose `result` holds the same
-report. `load_catalog(species="mouse")` then merges the three,
-in this order, into 60,707 ORFs: 2,976 MS-supported OpenProt ORFs, 16,484
-Ribo-seq ORFs of Chang et al. and 51,991 of Yang et al. 10,448 of Chang's and
-Yang's ORFs are the same ORF (28 of them also in OpenProt), and 296 OpenProt
-ORFs have Ribo-seq support from one or both, 242 from Yang et al. alone. On
-Ensembl 115 (GRCm39):
-
-| Evidence | Selected | Added to the proteome |
-| --- | ---: | ---: |
-| `"ms"` (default) | 2,976 | 2,854 |
-| `RiboSeq()` | 58,109 | 57,992 |
-| `RiboSeq() & MS()` | 378 | 371 |
-| `None` | 60,707 | 60,475 |
-
-Loading the merged catalogue takes about 10 seconds and 1.1 GB of memory, once
-per process, whichever evidence is selected. On GRCm38 and NCBIM37 all three
-are [lifted automatically](#every-assembly-of-a-species).
+report. `load_catalog(species="mouse")` merges OpenProt, Chang et al. and
+Yang et al. in that order. Current inputs contain 2,976 MS-supported OpenProt
+ORFs, 16,484 Chang et al. ORFs and 51,831 validated Yang et al. target placements.
+Overlapping source models merge by coding path and protein while keeping their
+original evidence. Merged counts and target-genome placement counts depend on
+these versions; inspect the resulting catalogue and proteome layer receipts.
+On GRCm38 and NCBIM37 the catalogues are
+[lifted automatically](#every-assembly-of-a-species).
 
 ### Mouse Ribo-seq ORFs (Chang et al.)
 
@@ -143,48 +133,55 @@ ATG, CTG, GTG or TTG. Its types are uORF (8,294), lncORF (3,576), uoORF
 
 ### Mouse RibORF ORFs (Yang et al.)
 
-`riborf-mouse` holds the mouse ORFs that RibORF 2.0 identified as translated in
-the Ribo-seq data of Yang et al. (Nat Commun 2024,
-[doi:10.1038/s41467-024-46240-9](https://doi.org/10.1038/s41467-024-46240-9);
-224 samples in 26 GEO series): uORF (26,052), lncORF (8,149), iORF (6,155),
-dORF (5,174), uoORF (3,157), N-terminal truncation (2,059), doORF (670) and
-extension (575). The source's ouORF, odORF and
-noncoding types take Chang et al.'s names, so `--biotype uoORF` finds both
-sources' overlapping uORFs. Annotated CDSs, likely sequencing errors,
-and stop-codon readthrough are not imported. Release 2 retains 11,902 valid
-ORFs shorter than five residues. Release 1, which applied a five-residue import
-floor, remains available explicitly as `--version 1`.
-
-- **Placement.** The source gives mm10 / GENCODE vM25 transcript coordinates
-  (a 1-based start, an exclusive end) and no proteins. Rather than lift mm10
-  through a restricted chain, each ORF is placed on the same transcript in
-  Ensembl 107 GRCm39 and translated from that transcript's sequence. Of the
-  54,128 non-skipped source rows, 51,991 resolve. Listed as unresolved, with the
-  reason: 1,642 rows on transcripts that changed length since GENCODE
-  vM25, 399 on transcripts that Ensembl 107 has in its GTF but in
-  neither FASTA, 95 on transcripts that are gone, and one whose
-  transcript's sequence changed at the same length. Placing through the
-  transcript gives the same coding path as lifting the source's mm10 blocks
-  through UCSC's chain for all but 2, in a duplicated region of chr14 where
-  the chain is ambiguous. Of the 12,222 source rows below five residues,
-  11,902 resolve and 320 carry one of the same explicit failures. In all,
-  51,881 place on Ensembl 115.
-- **Sources.** Supplementary Data 3 (CC BY 4.0) and Ensembl 107's GTF, cDNA
-  and ncRNA FASTAs; about 115 MB, of which the 32 MB GTF is the one Chang et
-  al.'s catalogue uses, so `orfdata setup` downloads it once.
-- **Evidence.** Presence in the source's set is aggregate Ribo-seq support:
-  `RiboSeq()` selects every ORF, `RiboSeq(min_studies=2)` none.
-- **Length policy.** `catalog.query(min_protein_length=5)` and
-  `Proteome.build(..., min_protein_length=5)` reproduce a downstream
-  five-residue selection without removing short models from the catalogue.
-  Proteome layers record the threshold and number filtered.
-- **IDs.** `riborf_` and the GRCm39 start and stop, such as
-  `riborf_4_120262600_120262620_+`, never a name Chang et al. use. Two paths
-  with the same start and stop add their transcript (`…@ENSMUST00000226495.2`),
-  and a lookup by the shared name lists both, as for OpenProt's split
-  accessions. An ORF in both sets merges by its coding path and protein,
-  whatever its name, and keeps Chang et al.'s ID; its `riborf_` ID still
-  finds it.
+`riborf-mouse` generation 3 reconstructs the ORFs reported by Yang et al.
+(Nat Commun 2024,
+[doi:10.1038/s41467-024-46240-9](https://doi.org/10.1038/s41467-024-46240-9))
+against their original GENCODE M25/GRCm38 reference before any placement on
+Ensembl 107/GRCm39. The original table supplies RNA intervals and BED blocks,
+not protein sequences. All 54,128 non-skipped rows reconstruct against the
+pinned native RNA, with the full transcript version, length, start/stop and
+every BED block checked.
+
+- **Download.** `orfdata data fetch riborf-mouse` installs the verified 24.9 MB
+  catalogue. Rebuilding remains available from its pinned source files.
+- **Target placement.** Exact full RNA, coding nucleotide sequence and translated
+  protein must agree. All target cDNA and ncRNA sequences are searched, including
+  duplicate loci and renamed transcripts. A surviving stable ID, unchanged RNA
+  length or matching start/stop alone cannot transfer evidence.
+- **Coverage.** 51,831 rows have a unique validated target placement, representing
+  41,882 distinct proteins. Another 118 have multiple exact target placements;
+  2,179 lack an unchanged complete target RNA. These 2,297 retain their native
+  models, sequences, evidence and explicit mapping reasons in
+  `catalog.metadata["unresolved"]`. `extra_orfs(...).unlifted` also reports them
+  under the requested evidence rule. A changed RNA with an unchanged small CDS
+  remains unresolved under this conservative placement method.
+- **Provenance.** Placed records retain their native `ORFRecord` in
+  `record.lifted_from` and a typed `record.reference_mapping`. The latter records
+  source/target references, original row ID, RNA interval, native CDS, source
+  pins, coordinate checks and exact sequence comparisons. Evidence summaries
+  continue to name the native reference and protein. No chain file is invented;
+  `record.liftover` is `None`. JSON and TSV exports retain both objects.
+  Managed manifests keep target annotation contexts in `references` and native
+  evidence contexts in `source_references`.
+- **Sources.** Supplementary Data 3 (CC BY 4.0), native GENCODE M25 GTF/RNA and
+  Ensembl 107 GTF/cDNA/ncRNA: about 198 MB in total. The 32 MB target GTF is also
+  used by Chang et al., so species setup acquires it once.
+- **Evidence and length.** The 224 samples in 26 GEO series provide aggregate
+  RibORF presence, not per-ORF named studies. `RiboSeq()` selects the target
+  records; `MS()` and `RiboSeq(min_studies=2)` select none. The catalogue retains
+  11,858 target ORFs shorter than five residues. Consumers choose length limits
+  when querying or building proteomes.
+- **IDs and earlier generations.** IDs are `riborf_` followed by the original
+  source row ID, including its native transcript version and RNA interval.
+  They do not relabel a current coordinate as an original observation.
+  Generations 1/2 reconstructed proteins directly from the target annotation;
+  new builds of those unvalidated generations fail before downloads and direct
+  callers to generation 3. Archived files remain historical data, with their
+  saved provenance; they are not retroactively sequence-validated.
+
+The source's ouORF, odORF and noncoding categories use the common uoORF, doORF
+and lncORF names. Canonical CDSs, likely sequencing errors and stop-codon
+readthrough rows remain excluded.
 
 ## Mouse ORFs from OpenProt
 

{orfdata-0.42.2 → orfdata-0.42.4}/docs/coverage.md

@@ -8,13 +8,14 @@ data coverage, independently of what the API can represent.
 | --- | --- | --- | --- |
 | Human | Wang: 29 specimens, gene/transcript FPKM and estimated copies/cell | Wang: 15,210 groups, iBAQ and estimated copies/cell; Pfizer: 10,352 groups, 37 channels, iBAQ/rIBAQ | Wang: adjacent cryosections; Pfizer: native RNA missing |
 | Mouse | Pfizer: 606 libraries, 56,051 gene targets, counts | Pfizer: 12,124 groups, 63 channels, iBAQ/rIBAQ | Unresolved library-to-pool correspondence |
-| Dog | Pfizer: 434 libraries, 30,941 gene targets, counts | Pfizer: 9,929 groups, 38 channels, iBAQ/rIBAQ | Unresolved library-to-pool correspondence |
+| Dog | Pfizer: 434 libraries, 30,941 gene targets, counts; mammary stroma: 18,582 genes × 30 samples | Pfizer: 9,929 groups, 38 channels, iBAQ/rIBAQ; mammary stroma: 1,583 native groups × 28 samples | Mammary stroma: 14 shared specimens, 28 paired compartments and two RNA samples without protein mates; Pfizer crosswalk remains unresolved |
 | Budding yeast | CoPY: four RNA libraries plus a source aggregate, FPKM and estimated copies/cell; Ingolia: four RNA libraries, counts and densities | CoPY: calibrated SRM and separately calibrated label-free estimates | CoPY: explicit source-cohort pairing; individual cross-assay replicate matches unknown |
 
 ## Peptides and MHC ligands
 
-Dog has three peptide-level datasets:
+Dog has four peptide-level datasets:
 
+- **Mammary stroma** ([guide](mammary-stroma.md)): 5,413 native peptide rows, intensity and LFQ columns, reported group memberships and original control/decoy flags across 28 samples.
 - **Pfizer native peptide evidence** ([guide](pfizer-peptides.md)): 953,980 target
   peptide sequences, native peptide/PSM reporter quantities and reported protein-group
   memberships. The 96 runs retain 38 tissue pools, four bridge channels and two
@@ -28,7 +29,7 @@ Dog has three peptide-level datasets:
   dog tumours, each searched against four protein databases, and from three
   human HCT 116 lines that each express one DLA-88 allele.
 
-All three searched known proteins only. Other species have no peptide-level
+All four searched known proteins only. Other species have no peptide-level
 measurement dataset yet; the human and mouse ORF catalogues carry pooled
 peptide evidence for their noncanonical ORFs.
 

{orfdata-0.42.2 → orfdata-0.42.4}/docs/dog-data-sources.md

@@ -1,7 +1,8 @@
 # Canine paired measurements and translation-evidence audit
 
 Source check: **28 October 2026**. This audit identifies importable measurement
-files and their limitations. It does not add a dataset to the registry.
+files and their limitations. The subsequent [mammary-stroma importer](mammary-stroma.md)
+now exposes the paired study through the common quantitative registry.
 The [machine-readable audit](data/dog-source-audit-20261028.json) contains source
 URLs, hashes, counts, search queries and a candidate sample crosswalk.
 
@@ -37,7 +38,8 @@ deposited search used a dated canonical UniProt reference, but its FASTA is not
 among the 31 PRIDE files. Sequence/reference qualification remains necessary;
 these data do not establish ncORF translation. Author-derived contrasts in the
 supplementary workbooks must stay distinct from sample-level measurements.
-Import work remains tracked in [#20](https://github.com/openvax/orfdata/issues/20).
+The paired study is now imported as `canine-mammary-stroma`; the remaining dog
+data requests stay tracked in [#20](https://github.com/openvax/orfdata/issues/20).
 
 ## Dog Ribo-seq and ncORF translation
 

orfdata-0.42.4/docs/mammary-stroma.md

@@ -0,0 +1,104 @@
+# Paired canine mammary-stroma RNA and protein
+
+`canine-mammary-stroma` imports the processed measurements of
+[Amini et al. (2021)](https://doi.org/10.1016/j.neo.2021.03.001),
+[GSE135183](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE135183) and
+[PXD023023](https://www.ebi.ac.uk/pride/archive/projects/PXD023023).
+It uses the same data commands and `QuantitativeDataset` API as other species
+and sources:
+
+```sh
+orfdata data info canine-mammary-stroma
+orfdata data fetch canine-mammary-stroma
+orfdata data path canine-mammary-stroma
+```
+
+```python
+from orfdata import DatasetRegistry
+
+registry = DatasetRegistry()
+dataset = registry.open("canine-mammary-stroma")  # Offline; no implicit download.
+rna_reference, protein_reference = dataset.manifest.references
+pairs = list(dataset.pairings())
+complete_pairs = [pair for pair in pairs if pair.right_assay_id is not None]
+pair = complete_pairs[0]
+rna = dataset.measurements(
+    reference=pair.reference, assay_id=pair.left_assay_id, unit="estimated_count"
+)
+protein = dataset.measurements(
+    reference=pair.right_reference, assay_id=pair.right_assay_id, unit="LFQ intensity"
+)
+```
+
+The release contains 993,560 native measurements in a 52 MB download and a
+514 MB installed SQLite file. Rebuilding uses approximately 8 MB of pinned processed inputs; no raw MS
+files, sequencing reads, alignment index or vendor database is required.
+
+| Layer | Native scope | Quantities |
+| --- | --- | --- |
+| RNA | 18,582 Ensembl gene IDs × 30 samples, 15 cases | Integer estimated counts from Kallisto/tximport |
+| Protein | 1,583 native MaxQuant groups × 28 samples, 14 cases | Intensity, LFQ intensity, iBAQ |
+| Peptide | 5,413 native peptide rows × 28 samples | Intensity and the separately reported LFQ intensity columns |
+
+Different quantitative columns do not add biological replicates. The 58
+assays retain their original sample and run identities. Native MS rows include
+77 reversed protein groups and 52 contaminant groups; these counts are not
+counts of confidently detected canine proteins.
+
+## Specimen pairing
+
+The authors reused archived FFPE mammary specimens for separate laser-capture
+RNA and protein preparations. The importer verifies a unique match between
+clinical Table 1 and GEO by breed, age and histological subtype, recording
+spelling normalizations and both original sets of labels. This is a
+**metadata-derived crosswalk within an author-declared shared cohort**, not an
+author-provided case-ID mapping.
+
+Fourteen shared cases give 28 complete assay pairs: normal stroma and
+cancer-associated stroma separately. Two RNA samples from GEO case 127 have
+explicitly missing protein mates; the paper reports exclusion of proteomics
+case 7 because its paraffin block was unavailable. No protein sample is invented.
+
+These pairs use `same_specimen`. This basis means separate preparations from the
+same original specimen, without claiming identical aliquots or adjacent
+sections. Temporal labels refer to the original biological specimen, not the
+dates of RNA sequencing and mass-spectrometry acquisition. Missing mates retain
+an unknown temporal relationship. Pairing does not itself establish a
+relationship between RNA genes and protein groups: this release has no
+cross-modal target relations or current-reference mappings.
+
+The normal compartment belongs to a cancer-bearing mammary specimen. It is not
+a healthy-animal normal-tissue atlas. Source case IDs identify individual
+specimens, and repeated compartments or quantities never imply additional dogs.
+
+## Missingness, native records and reference limits
+
+- RNA zeros remain measured zero counts. Counts are estimated gene-level values,
+  with no conversion to TPM, copies per cell or translation efficiency.
+- MaxQuant MS zeros denote unquantified cells. Their numeric values are `None`
+  with status `missing`; the original `"0"` survives in provenance. The 2,156
+  native `"NaN"` iBAQ cells also remain missing, with a separate reason.
+- Every native MS field survives once in the corresponding mapping's
+  `provenance["native_record"]`. Measurements retain source row/column, original
+  numeric text, identification type and the original control/decoy flags.
+- Peptide mappings retain every reported native protein-group ID. A unique
+  assignment to one *group* is not unique protein or isoform detection. Group
+  records retain reported protein labels, including contaminants, without
+  assigning them to a canine biological protein reference.
+- No source QC threshold, aggregation, imputation, differential-expression model
+  or RNA/protein ratio is applied. Consumers can inspect the native flags and
+  scores explicitly.
+
+RNA was quantified against Ensembl transcripts on CanFam3.1; its exact release
+and transcript sequence bytes are not reported. The protein search used a
+canonical UniProt dog reference dated 2019-07-29 with decoys and contaminants;
+the searched FASTA is absent from the deposited processed files. Both references
+remain source-bound and incomplete. Source membership and uniqueness flags are
+preserved, but no full-background sequence uniqueness is asserted. These data
+provide neither ncORF translation evidence nor MHC-presentation evidence.
+
+The [earlier source audit](dog-data-sources.md) records the acquisition and
+crosswalk evidence. The PRIDE project declares CC0; GEO numerical data and
+factual metadata use [GEO's public-data terms](https://www.ncbi.nlm.nih.gov/geo/info/disclaimer.html).
+Factual clinical curation cites Table 1 of the CC BY-NC-ND article. Its original
+text and figures are excluded from the dataset artifact.

{orfdata-0.42.2 → orfdata-0.42.4}/docs/pairing.md

@@ -17,7 +17,7 @@ a Cartesian product of technical replicates. Each row retains its assay pairing
 and target relationship. A three-assay experiment can declare several pairwise
 comparisons; sharing an RNA measurement does not create independent replicates.
 
-Pairing bases are `same_material`, `adjacent_donor_tissue`,
+Pairing bases are `same_material`, `same_specimen`, `adjacent_donor_tissue`,
 `biological_replicate`, `independent_cultures`, `tissue_aggregate` and
 `condition_aggregate`. The caller must choose the allowed bases. By default,
 only explicitly concurrent assays are returned; opt into `longitudinal` or
@@ -116,3 +116,8 @@ identities nor transfers abundance or protein-existence evidence. ORF comparison
 additionally require their own sequence, frame and positional evidence and
 `allow_orfs=True`; a gene ortholog cannot establish an orthologous ncORF.
 Human, mouse, dog, yeast and arbitrary other taxa use this same API.
+
+`same_specimen` records separate preparations from the same original specimen.
+It does not assert identical aliquots (`same_material`) or adjacent sections.
+The [canine mammary-stroma dataset](mammary-stroma.md) uses this distinction and
+retains missing mates explicitly.

{orfdata-0.42.2 → orfdata-0.42.4}/mkdocs.yml

@@ -41,6 +41,7 @@ nav:
       - Human, mouse and dog protein: pfizer-protein.md
       - Pfizer dog peptide and group evidence: pfizer-peptides.md
       - Dog and mouse RNA: pfizer-rna.md
+      - Paired canine mammary stroma: mammary-stroma.md
       - Yeast protein and RNA: copy-yeast.md
       - Yeast RNA and ribosome occupancy: ingolia-yeast.md
       - Dog MHC ligands and PeptideAtlas: dog-peptides.md

{orfdata-0.42.2 → orfdata-0.42.4}/orfdata/catalog.py

@@ -11,7 +11,7 @@ import os
 import re
 from collections import Counter
 from copy import copy, deepcopy
-from dataclasses import asdict, dataclass, replace
+from dataclasses import asdict, dataclass, field, replace
 from importlib.resources import files
 from pathlib import Path
 from typing import NamedTuple, Optional
@@ -22,8 +22,10 @@ from .evidence import MS_METHODS, evidence_rule, stored_method
 from .evidence_summary import EvidenceSummary, unique_summaries
 from .initiation_context import KOZAK_PREFERENCES, InitiationContext
 from .liftover import Liftover
+from .measurements import _plain
 from .model import ORFModel, TranscriptORF, _contig, _id_forms, _species, _transcript_match
-from .reference import ReferenceContext
+from .reference import ReferenceContext, ScopedTarget
+from .reference_mapping import ReferenceMapping
 from .sequence import genetic_code, reverse_complement_dna
 
 
@@ -142,17 +144,35 @@ class ORFRecord:
     members: tuple = ()
     lifted_from: Optional["ORFRecord"] = None
     liftover: Optional[Liftover] = None
+    reference_mapping: Optional[ReferenceMapping] = field(default=None, hash=False)
 
     def __post_init__(self):
         object.__setattr__(self, "summaries", unique_summaries(self.summaries))
         object.__setattr__(self, "members", tuple(self.members))
-        if (self.lifted_from is None) != (self.liftover is None):
-            raise ValueError("A lifted record names both its original record and the chain")
         original = self.lifted_from
+        methods = int(self.liftover is not None) + int(self.reference_mapping is not None)
+        if methods != int(original is not None):
+            raise ValueError("A placed record names its original and exactly one chain or reference mapping")
         if original is not None and any(getattr(self, name) != getattr(original, name) for name in (
                 "orf_id", "protein_sequence", "translation_table", "start_codon", "source",
                 "ribosome_studies", "peptides", "summaries")):
             raise ValueError(f"A lifted record keeps its original's ID, protein and evidence: {self.orf_id}")
+        mapping = self.reference_mapping
+        if mapping is not None:
+            if not isinstance(mapping, ReferenceMapping):
+                raise TypeError("ORF reference placements require a ReferenceMapping")
+            if original.reference_context is None or self.reference_context is None:
+                raise ValueError("ORF reference placements require both reference identities")
+            source = ScopedTarget(original.reference_context, "orf", original.orf_id, original.source)
+            target = ScopedTarget(self.reference_context, "orf", self.orf_id, self.source)
+            if mapping.source != source or mapping.targets != (target,) or mapping.status != "unique":
+                raise ValueError("ORF reference mapping endpoints disagree with the records")
+            digest = hashlib.sha256(self.protein_sequence.encode("ascii")).hexdigest()
+            if not mapping.sequence_validated or mapping.validation["sequence"]["source_sha256"] != digest:
+                raise ValueError("ORF reference mapping protein identity disagrees with the records")
+            expected = {"source": _placement_position(original), "target": _placement_position(self)}
+            if _plain(mapping.validation.get("position")) != expected:
+                raise ValueError("ORF reference mapping positions disagree with the records")
         if self.members:
             # Each member validated its own summaries against its own identity.
             key = _merge_key(self)
@@ -229,7 +249,7 @@ class ORFRecord:
 
     def to_dict(self):
         """Return a serializable copy of this record."""
-        result = asdict(self)
+        result = _plain(self)
         if not self.summaries:
             result.pop("summaries")
         if self.reference_context is None:
@@ -246,7 +266,10 @@ class ORFRecord:
             result["lifted_from"] = self.lifted_from.to_dict()
         else:
             result.pop("lifted_from")
+        if self.liftover is None:
             result.pop("liftover")
+        if self.reference_mapping is None:
+            result.pop("reference_mapping")
         return result
 
     def map_initiation_position(self, position, *, genome_build):
@@ -417,6 +440,12 @@ class ORFRecord:
         }
 
 
+def _placement_position(record):
+    """The coding placement bound by a typed source-to-target reference mapping."""
+    return dict(transcript_id=record.transcript_id, contig=record.contig, strand=record.strand,
+                coding_blocks=[list(block) for block in record.coding_blocks])
+
+
 def _qualify_colliding_ids(records):
     """Give colliding source records unique catalogue IDs, retaining their native records."""
     records = tuple(records)
@@ -436,7 +465,7 @@ def _qualify_colliding_ids(records):
         names = [f"{quote(r.source, safe='')}:{quote(r.orf_id, safe='')}" if i in qualified
                  else r.orf_id for i, r in enumerate(records)]
     return tuple(r if name == r.orf_id else replace(
-        r, orf_id=name, members=r.members or (r,), lifted_from=None, liftover=None)
+        r, orf_id=name, members=r.members or (r,), lifted_from=None, liftover=None, reference_mapping=None)
         for r, name in zip(records, names))
 
 
@@ -449,6 +478,15 @@ class ORFCatalog:
         self.genome_build = metadata["genome_build"]
         self.taxon_id = metadata["taxon_id"]
         self.species = metadata["species"]
+        for entry in metadata.get("unresolved", ()):
+            if entry.get("native_record") is not None:
+                native = _record(entry["native_record"])
+                mapping = ReferenceMapping.from_dict(entry["reference_mapping"])
+                expected = ScopedTarget(native.reference_context, "orf", native.orf_id, native.source)
+                if (mapping.source != expected or mapping.status == "unique"
+                        or mapping.target_reference.taxon_id != self.taxon_id
+                        or mapping.target_reference.genome_build != self.genome_build):
+                    raise ValueError("Unresolved reference mapping disagrees with its native record or catalogue")
         self._by_id = {r.orf_id: r for r in self.records}
         self._aliases = {}
         self._by_source_id = {}
@@ -834,7 +872,7 @@ def merge_catalogs(catalogs):
         pinned = {None if m.reference_context is None else m.reference_context.key for m in members}
         records.append(replace(
             # The merged ORF is no one source's lift; each member keeps its own lift provenance.
-            primary, members=tuple(members), lifted_from=None, liftover=None,
+            primary, members=tuple(members), lifted_from=None, liftover=None, reference_mapping=None,
             initiation_context=_shared_context(primary, members),
             # Sources pinned to different annotations of one assembly leave the ORF unpinned.
             reference_context=primary.reference_context if len(pinned) == 1 else None, **_union(members, study)))
@@ -927,7 +965,10 @@ def _record(row, legacy=False):
     row["members"] = tuple(_record(m) for m in row.get("members", ()))
     if row.get("lifted_from") is not None:
         row["lifted_from"] = _record(row["lifted_from"])
+    if row.get("liftover") is not None:
         row["liftover"] = Liftover(**row["liftover"])
+    if row.get("reference_mapping") is not None:
+        row["reference_mapping"] = ReferenceMapping.from_dict(row["reference_mapping"])
     return ORFRecord(**row)
 
 

{orfdata-0.42.2 → orfdata-0.42.4}/orfdata/catalog_manifest.py

@@ -21,14 +21,22 @@ class CatalogManifest(_Record):
     coverage: Mapping
     processing: Mapping
     schema: str = "orfdata.catalog-manifest.v1"
+    source_references: tuple = ()
 
     @classmethod
     def from_dict(cls, data):
         values = dict(data)
         values["references"] = tuple(ReferenceContext(**c) for c in values["references"])
         values["sources"] = tuple(SourceAsset.from_dict(s) for s in values["sources"])
+        values["source_references"] = tuple(ReferenceContext(**c) for c in values.get("source_references", ()))
         return cls(**values)
 
+    def to_dict(self):
+        result = super().to_dict()
+        if not self.source_references:
+            result.pop("source_references")
+        return result
+
     def __post_init__(self):
         for name in ("catalog_id", "version", "title"):
             _identifier(getattr(self, name), name)
@@ -54,6 +62,14 @@ class CatalogManifest(_Record):
         builds = {r.genome_build for r in self.references if r.genome_build is not None}
         if len(builds) != 1 or not any(r.is_genomic for r in self.references):
             raise ValueError("A catalogue must have one assembly and a complete genomic reference")
+        if isinstance(self.source_references, str):
+            raise TypeError("Source references must be a sequence of contexts")
+        object.__setattr__(self, "source_references", tuple(self.source_references))
+        if any(not isinstance(r, ReferenceContext) or not r.is_genomic
+               or r.taxon_id != self.taxon_id for r in self.source_references):
+            raise ValueError("Native source references must be genomic contexts of the catalogue species")
+        if len({r.key for r in self.source_references}) != len(self.source_references):
+            raise ValueError("Duplicate native source reference contexts")
         if any(not isinstance(s, SourceAsset) for s in self.sources):
             raise TypeError("Catalogue sources must be SourceAssets")
         if len({s.asset_id for s in self.sources}) != len(self.sources):
@@ -91,9 +107,16 @@ class CatalogManifest(_Record):
             reference = record.reference_context
             if reference is None or not reference.is_genomic or reference.key not in references:
                 raise ValueError("Managed ORF records require a declared complete genomic reference")
+        evidence_references = references | {r.key for r in self.source_references}
         for summary in catalog.evidence_summaries:
-            if summary.target.reference.key not in references:
+            if summary.target.reference.key not in evidence_references:
                 raise ValueError("Evidence summary reference is absent from catalogue manifest")
+        for entry in catalog.metadata.get("unresolved", ()):
+            if entry.get("native_record") is not None:
+                native = ReferenceContext(**entry["native_record"]["reference_context"])
+                target = ReferenceContext(**entry["reference_mapping"]["target_reference"])
+                if native.key not in evidence_references or target.key not in references:
+                    raise ValueError("Unresolved placement reference is absent from catalogue manifest")
         counts = dict(distinct_protein_sequences=len({r.protein_sequence for r in catalog.records}),
                       evidence_summaries=len(catalog.evidence_summaries))
         for key, actual in counts.items():

{orfdata-0.42.2 → orfdata-0.42.4}/orfdata/catalogs.py

@@ -40,7 +40,7 @@ def _copy_catalog(inputs, output, manifest):
         handle.write(json.dumps(catalog.to_dict(), sort_keys=True, separators=(",", ":"), allow_nan=False).encode())
 
 
-def _builder(manifest):
+def _builder(manifest, *, for_build=False):
     """The packaged importer a release names; import paths in metadata are never executed."""
     importer = manifest.processing.get("importer")
     if importer == "catalog-copy-v1":
@@ -54,7 +54,13 @@ def _builder(manifest):
 
         return lambda inputs, output: import_orfs(inputs, output, manifest)
     if importer in ("riborf-v1", "riborf-v2"):
-        from .sources.riborf import import_orfs
+        from .sources.riborf import import_orfs, validate_transcript_manifest
+
+        if for_build:
+            validate_transcript_manifest(manifest)
+        return lambda inputs, output: import_orfs(inputs, output, manifest)
+    if importer == "riborf-native-v3":
+        from .sources.riborf_native import import_orfs
 
         return lambda inputs, output: import_orfs(inputs, output, manifest)
     if importer in ("riborf-genome-v1", "riborf-genome-v2"):
@@ -198,7 +204,7 @@ class CatalogRegistry:
     def build(self, name, version=None, *, force=False, source_paths=None, download_options=None):
         """Explicitly materialize an allowlisted catalogue source adapter."""
         manifest = self.info(name, version)
-        return build_catalog(manifest, _builder(manifest), cache_root=self._cache_root,
+        return build_catalog(manifest, _builder(manifest, for_build=True), cache_root=self._cache_root,
                              source_paths=source_paths, force=force, download_options=download_options)
 
     def inspect_build(self, name, version=None, *, verify_files=True):

{orfdata-0.42.2 → orfdata-0.42.4}/orfdata/cli.py

@@ -320,14 +320,14 @@ def _query(args, parser, catalog_loader):
             "initiation_context_json",
         )
         scoped = not isinstance(catalog, UORFCatalog)
-        provenance = ("members", "liftover", "lifted_from")
+        provenance = ("members", "liftover", "lifted_from", "reference_mapping")
         if scoped:
             fields += ("start_codon", "reference_context_json", "summaries_json") + tuple(
                 f + "_json" for f in provenance)
         elif any(r.summaries for r in records):
             # The released upstream-only columns have no place for aggregate counts.
             parser.error("The legacy upstream-ORF TSV cannot export aggregate evidence; use --format json")
-        elif any(r.members or r.liftover for r in records):
+        elif any(r.members or r.lifted_from for r in records):
             parser.error("The legacy upstream-ORF TSV cannot export merged or lifted records; use --format json")
         elif any(getattr(r, "start_codon", None) for r in records):
             parser.error("The legacy upstream-ORF TSV cannot export declared start codons; use --format json")