orfdata 0.3.0__tar.gz → 0.5.0__tar.gz

{orfdata-0.3.0/orfdata.egg-info → orfdata-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: orfdata
-Version: 0.3.0
+Version: 0.5.0
 Summary: Reference open reading frames and experimental translation evidence on annotated transcripts
 Author: OpenVax
 License-Expression: Apache-2.0

{orfdata-0.3.0 → orfdata-0.5.0}/docs/measurements.md

@@ -46,6 +46,10 @@ mapping table. The default PyEnsembl target uses its unversioned `id`; request
 mapping across annotation releases, gene isoforms, ORFs or species. A variant or
 fusion product does not inherit a native reference protein measurement.
 
+For native sources with unreported genomic metadata, use explicit
+[source references and mapping views](source-references.md) to retain the original
+observation alongside its mapping to a PyEnsembl target.
+
 The CLI uses the same operations:
 
 ```sh
@@ -80,9 +84,9 @@ records technical replicates, runs, fractions, TMT plex/channel/bridge,
 normalization and calibration. A sample's strain/breed belongs in its attributes;
 it need not be the strain used to assemble its reference genome. `TargetMapping`
 retains all candidate assignments or an unmapped reason. A protein-group
-representative does not become a uniquely measured protein. Explicit sample
-pairing and orthology comparisons follow in
-[issue #6](https://github.com/openvax/orfdata/issues/6).
+representative does not become a uniquely measured protein. See
+[pairing and orthology comparisons](pairing.md) for explicit experimental
+relationships and derived RNA/protein ratios.
 
 ## Importing a source
 
@@ -96,7 +100,10 @@ An importer calls `QuantitativeDataset.create(path, manifest, samples=...,
 assays=..., measurements=..., mappings=...)`. These inputs may be generators;
 measurements are written incrementally. Every observation names a source asset
 or earlier input measurements. Derived inputs precede outputs in the same
-reference context. Duplicate IDs, broken foreign keys, wrong dataset releases
+dataset release. `input_measurements` names IDs within the output reference;
+`input_references` instead uses `MeasurementReference` records with each input's
+dataset, version, reference and ID. The two forms cannot be mixed. Cross-species
+derived abundance is rejected. Duplicate IDs, broken foreign keys, wrong dataset releases
 and cross-reference links fail the import. Failed builds do not replace data.
 
 Use `build_dataset(manifest, builder, ...)` for a managed local import.

orfdata-0.5.0/docs/pairing.md

@@ -0,0 +1,97 @@
+# Comparing RNA, ribosome and protein abundance
+
+Use declared experimental relationships to compare assays. A tissue label alone
+does not make an RNA sample and a protein sample a matched pair.
+
+```python
+for match in dataset.matched_measurements(allowed_bases=["same_material"]):
+    row = match.to_table_row()
+    print(row["rna_abundance"], row["protein_abundance"])
+```
+
+Rows include named RNA, ribosome occupancy, translation-efficiency, protein,
+peptide and spectrum-count columns, each containing original measurement
+records with units, missingness and provenance. Missing columns are empty lists.
+Multiple values remain multiple values; the API does not average them or create
+a Cartesian product of technical replicates. Each row retains its assay pairing
+and target relationship. A three-assay experiment can declare several pairwise
+comparisons; sharing an RNA measurement does not create independent replicates.
+
+Pairing bases are `same_material`, `adjacent_donor_tissue`,
+`biological_replicate`, `independent_cultures`, `tissue_aggregate` and
+`condition_aggregate`. The caller must choose the allowed bases. By default,
+only explicitly concurrent assays are returned; opt into `longitudinal` or
+`unknown` with `allowed_temporal`. Missing mates stay explicit. Technical runs,
+TMT channels and bridges remain in the assays and do not become biological
+sample counts.
+
+RNA annotation and protein search references can differ. `AssayPair.reference`
+is the left assay's context; `right_reference` explicitly identifies the right
+assay's context and defaults to the left for older records. Both must belong to
+the same species. A protein FASTA does not need an invented genomic assembly:
+see [source references](source-references.md). Original targets and references
+remain attached to both sides, even when their local IDs happen to match.
+
+The CLI streams the same table rows:
+
+```sh
+orfdata datasets pairs DATASET --version RELEASE --basis same_material
+```
+
+## Which targets are being compared?
+
+`TargetRelation` links exact, reference-scoped targets with mapping evidence
+and cardinality. `same_target` compares one target across assays/samples.
+`canonical_translation` links a gene/transcript to its annotated protein.
+`parent_rna_context` links parent RNA to an ORF but keeps the RNA value attached
+to its original gene/transcript. `shared_group_member` records an ambiguous
+protein-group assignment; the group value stays a group value.
+`gene_protein_group_context` links RNA to an ambiguous source protein group
+without assigning that group's abundance to an individual protein or ORF.
+
+Importers pass `AssayPair` and `TargetRelation` iterators as `pairings=` and
+`relations=` to `QuantitativeDataset.create`. IDs belong to the dataset release
+and left reference context. The right assay/target keeps its own context.
+Invalid assay references, contradictory known donor
+identities and conflicting one-to-one assignments fail the import. Original
+storage-version-1 datasets remain readable and have no declared relationships.
+
+## Ratios are derived measurements
+
+```python
+ratio = match.ratio(
+    selected_protein_measurement, selected_rna_measurement,
+    measurement_id="ratio-1", method="native protein/RNA ratio",
+    normalization={"protein": "source bridge normalization", "rna": "source TPM"},
+)
+```
+
+Select actual measurements from opposite sides of the match. Ratios require
+one-to-one target attribution and linear inputs. Parent-RNA/ORF and shared-group
+links cannot produce target-specific ratios. There is no implicit log conversion,
+unit conversion, replicate averaging or uncertainty propagation.
+
+An iBAQ/TPM ratio keeps `(iBAQ)/(TPM)` units. Copies-per-cell values also retain
+their native composite units and the calibration recorded on their source
+assays. Missing, censored or nonpositive denominators produce an unresolved
+`not_measured` result with its reason, never infinity or a measured zero. RPF/RNA
+ratios are source-defined translation-efficiency measures, not measured flux.
+All derived values retain their input IDs, pairing, target attribution and
+normalization assumptions. When input reference contexts differ, the ratio
+stores `MeasurementReference` records in `input_references`, rather than
+unqualified local IDs. This preserves source identity across annotation and
+search-FASTA releases.
+
+## Across species
+
+`compare_orthologs(left_dataset, right_dataset, mappings)` is separate and
+opt-in. Each `OrthologyMapping` pins its source file, release, both reference
+contexts, cardinality, confidence definition and evidence. The default accepts
+one-to-one canonical gene mappings; one-to-many relationships require explicit
+permission through `allowed_cardinalities`.
+
+The result contains each species' original observations. It neither merges
+identities nor transfers abundance or protein-existence evidence. ORF comparisons
+additionally require their own sequence, frame and positional evidence and
+`allow_orfs=True`; a gene ortholog cannot establish an orthologous ncORF.
+Human, mouse, dog, yeast and arbitrary other taxa use this same API.

{orfdata-0.3.0 → orfdata-0.5.0}/docs/reference.md

@@ -75,9 +75,8 @@ reconstructed RNA products and preserves each candidate's RNA support. For
 example, the native TPST1 peptide KIILSKISV is absent from the Sid TPST1–CRCP
 alternatives, so its MS evidence does not support those altered products.
 
-The reference integration is available in Varcode 10.12.0; the Isovar 1.50.0
-integration is in review. The stricter reference-context contract is coordinated
-with Varcode 10.13.0 and Isovar 1.51.0.
+The reference integration, including strict reference-context validation, is
+available in Varcode 10.13.0 and Isovar 1.51.0.
 
 ## Data and limits
 

orfdata-0.5.0/docs/source-references.md

@@ -0,0 +1,90 @@
+# Source references and PyEnsembl mappings
+
+A proteomics experiment may search a known protein FASTA without naming a
+genomic assembly. An older RNA study may report its annotation provider but
+omit the release. ORFdata keeps these source identities and their unknown fields.
+
+```python
+from orfdata import ReferenceContext
+
+source = ReferenceContext(
+    taxon_id=9615, species="dog", genome_build=None,
+    annotation_name="uniprot", annotation_version=None,
+    sequence_sha256=searched_fasta_sha256,
+)
+```
+
+Unknown assembly/provider fields require a pinned `sequence_sha256` or
+`source_sha256`. The former identifies a sequence set; the latter can pin an
+input table when its reference release is unreported. These source contexts
+cannot bind a genomic transcript. A `ReferenceORF` still requires a complete
+genomic model validated against its actual PyEnsembl transcript. Existing
+complete-reference keys are unchanged.
+
+## Query through an explicit mapping
+
+`ReferenceMapping` connects a native `ScopedTarget` to all candidate targets in
+one destination reference. It records the dataset/version, mapping method,
+validation, source provenance, status and unresolved/unmapped reason. Importers
+must perform the stated checks against the pinned inputs. Unique identifier
+resolution is separate from sequence validation.
+
+```python
+from orfdata import ScopedTarget
+
+target = ScopedTarget.from_pyensembl(gene, kind="gene", namespace="ensembl:110")
+for view in dataset.mapped_measurements(target):
+    print(view.measurement.value, view.measurement.unit)
+    print(view.measurement.target, view.requested_target)
+    print(view.mapping.method, view.mapping.validation)
+```
+
+The view contains the **original observation**, including its source reference,
+target, sample, assay, unit and provenance. Nothing is relabeled or copied onto
+the requested target. Direct `dataset.measurements(target=...)` still returns
+only observations native to that exact target/reference.
+
+By default, mapped queries return uniquely resolved assignments. Use
+`mapping_statuses=["ambiguous", "unresolved"]` to inspect other candidates, and
+`require_sequence=True` to require recorded exact sequence validation. A group
+value returned for a candidate member has `attribution="group_membership_context"`;
+it does not independently quantify that protein. Parent gene/transcript RNA
+cannot map directly onto an ORF. Use a separate parent-context relationship.
+
+The CLI exposes the same query:
+
+```sh
+orfdata datasets query DATASET --mapped --reference ref:SHA256 \
+    --kind protein --namespace ensembl:110 --target PROTEIN_ID --require-sequence
+```
+
+## Importer contract
+
+Pass `reference_mappings=` to `QuantitativeDataset.create`. Declare source and
+destination contexts in the manifest. Each source/destination-reference pair
+has one mapping containing all candidates, preventing competing unique claims.
+Unmapped records remain visible through `dataset.reference_mappings()`.
+
+The status is `unique`, `ambiguous`, `unmapped` or `unresolved`. A single reported
+protein representative with incomplete group membership is unresolved; its
+measurement cannot become a uniquely measured member. Identifier-based mappings
+must say what was checked and leave sequence validation absent when unavailable.
+For an exact sequence search, record:
+
+```python
+validation = {"sequence": {
+    "source_sha256": source_protein_sequence_sha256,
+    "target_sha256": matching_protein_sequence_sha256,
+    "all_candidates_checked": True,
+}}
+```
+
+These are hashes of the compared sequences, distinct from the whole searched
+FASTA's hash in the reference context. Both hashes must agree to count as an
+exact match. ORF assignments additionally require their own `frame` and
+`position` evidence. Known reference-strain differences need explicit
+`strain_relationship` evidence. Cross-species mappings use the separate
+[orthology API](pairing.md#across-species).
+
+Storage version 3 preserves these mappings. Earlier versions remain readable
+and report no declared reference mappings; no mapping is inferred on load.

{orfdata-0.3.0 → orfdata-0.5.0}/orfdata/__init__.py

@@ -15,9 +15,26 @@ from .catalog import (
 from .dataset import QuantitativeDataset
 from .datasets import DatasetRegistry, build_dataset
 from .initiation_context import KOZAK_PREFERENCES, InitiationContext, annotate_initiation_context
-from .measurements import Assay, DatasetManifest, Measurement, Sample, SourceAsset, TargetMapping
+from .measurements import (
+    Assay,
+    DatasetManifest,
+    Measurement,
+    MeasurementReference,
+    Sample,
+    SourceAsset,
+    TargetMapping,
+)
 from .model import ORFModel, ReferenceORF
+from .pairing import (
+    AssayPair,
+    MatchedMeasurements,
+    OrthologyComparison,
+    OrthologyMapping,
+    TargetRelation,
+    compare_orthologs,
+)
 from .reference import ReferenceContext, ScopedTarget
+from .reference_mapping import MappedMeasurement, ReferenceMapping
 from .version import __version__
 
 __all__ = [
@@ -39,7 +56,16 @@ __all__ = [
     "Sample",
     "Assay",
     "Measurement",
+    "MeasurementReference",
     "TargetMapping",
+    "AssayPair",
+    "TargetRelation",
+    "MatchedMeasurements",
+    "OrthologyMapping",
+    "OrthologyComparison",
+    "compare_orthologs",
+    "ReferenceMapping",
+    "MappedMeasurement",
     "load_catalog",
     "UORFCatalog",
     "UORFRecord",

{orfdata-0.3.0 → orfdata-0.5.0}/orfdata/dataset.py

@@ -13,10 +13,20 @@ from .measurements import (
     Assay,
     DatasetManifest,
     Measurement,
+    MeasurementReference,
     Sample,
     TargetMapping,
 )
+from .pairing import (
+    PAIRING_BASES,
+    TEMPORAL_RELATIONS,
+    AssayPair,
+    MatchedMeasurements,
+    TargetRelation,
+    _check_one_to_one,
+)
 from .reference import ReferenceContext, ScopedTarget
+from .reference_mapping import MappedMeasurement, ReferenceMapping
 
 _SCHEMA = """
 CREATE TABLE metadata (name TEXT PRIMARY KEY, value TEXT NOT NULL);
@@ -44,13 +54,41 @@ CREATE TABLE measurement (
     FOREIGN KEY(reference_key, assay_id, sample_id)
         REFERENCES assay(reference_key, id, sample_id));
 CREATE TABLE input_measurement (
-    reference_key TEXT NOT NULL, output_id TEXT NOT NULL, input_id TEXT NOT NULL,
-    PRIMARY KEY(reference_key, output_id, input_id),
+    reference_key TEXT NOT NULL, output_id TEXT NOT NULL,
+    input_reference_key TEXT NOT NULL, input_id TEXT NOT NULL,
+    PRIMARY KEY(reference_key, output_id, input_reference_key, input_id),
     FOREIGN KEY(reference_key, output_id) REFERENCES measurement(reference_key, id),
-    FOREIGN KEY(reference_key, input_id) REFERENCES measurement(reference_key, id));
+    FOREIGN KEY(input_reference_key, input_id) REFERENCES measurement(reference_key, id));
 CREATE INDEX target_lookup ON target(reference_key, namespace, kind, identifier);
 CREATE INDEX observation_lookup ON measurement(reference_key, target_key, sample_id, assay_id);
 CREATE INDEX assay_lookup ON measurement(reference_key, assay_id, quantity, status);
+CREATE TABLE assay_pair (
+    reference_key TEXT NOT NULL REFERENCES reference(key), id TEXT NOT NULL,
+    right_reference_key TEXT NOT NULL REFERENCES reference(key),
+    left_assay_id TEXT, right_assay_id TEXT, record TEXT NOT NULL,
+    PRIMARY KEY(reference_key, id),
+    FOREIGN KEY(reference_key, left_assay_id) REFERENCES assay(reference_key, id),
+    FOREIGN KEY(right_reference_key, right_assay_id) REFERENCES assay(reference_key, id));
+CREATE TABLE target_relation (
+    reference_key TEXT NOT NULL, id TEXT NOT NULL,
+    right_reference_key TEXT NOT NULL,
+    left_target_key TEXT NOT NULL, right_target_key TEXT NOT NULL, record TEXT NOT NULL,
+    PRIMARY KEY(reference_key, id),
+    FOREIGN KEY(reference_key, left_target_key) REFERENCES target(reference_key, key),
+    FOREIGN KEY(right_reference_key, right_target_key) REFERENCES target(reference_key, key));
+CREATE TABLE reference_mapping (
+    reference_key TEXT NOT NULL, id TEXT NOT NULL, source_target_key TEXT NOT NULL,
+    target_reference_key TEXT NOT NULL REFERENCES reference(key), record TEXT NOT NULL,
+    PRIMARY KEY(reference_key, id),
+    UNIQUE(reference_key, source_target_key, target_reference_key),
+    FOREIGN KEY(reference_key, source_target_key) REFERENCES target(reference_key, key));
+CREATE TABLE reference_assignment (
+    source_reference_key TEXT NOT NULL, mapping_id TEXT NOT NULL,
+    target_reference_key TEXT NOT NULL, target_key TEXT NOT NULL,
+    PRIMARY KEY(source_reference_key, mapping_id, target_reference_key, target_key),
+    FOREIGN KEY(source_reference_key, mapping_id) REFERENCES reference_mapping(reference_key, id),
+    FOREIGN KEY(target_reference_key, target_key) REFERENCES target(reference_key, key));
+CREATE INDEX reference_assignment_lookup ON reference_assignment(target_reference_key, target_key);
 """
 
 
@@ -73,7 +111,8 @@ class QuantitativeDataset:
     def __init__(self, path):
         self.path = Path(path).resolve()
         with closing(self._connect()) as connection:
-            if connection.execute("PRAGMA user_version").fetchone()[0] != 1:
+            self._storage_version = connection.execute("PRAGMA user_version").fetchone()[0]
+            if self._storage_version not in (1, 2, 3):
                 raise ValueError("Unsupported quantitative database schema")
             row = connection.execute("SELECT value FROM metadata WHERE name='manifest'").fetchone()
             if row is None:
@@ -88,7 +127,8 @@ class QuantitativeDataset:
         return connection
 
     @classmethod
-    def create(cls, path, manifest, *, samples, assays, measurements, mappings=()):
+    def create(cls, path, manifest, *, samples, assays, measurements, mappings=(),
+               pairings=(), relations=(), reference_mappings=()):
         """Write a new dataset from iterators, validating scope and foreign keys.
 
         Never overwrite an existing file. An interrupted/invalid build removes
@@ -106,7 +146,7 @@ class QuantitativeDataset:
             connection = sqlite3.connect(path)
             connection.execute("PRAGMA foreign_keys=ON")
             connection.executescript(_SCHEMA)
-            connection.execute("PRAGMA user_version=1")
+            connection.execute("PRAGMA user_version=3")
             connection.execute("INSERT INTO metadata VALUES ('manifest', ?)",
                                (_json(manifest.to_dict()),))
             for reference in manifest.references:
@@ -125,12 +165,68 @@ class QuantitativeDataset:
                 connection.execute("INSERT INTO assay VALUES (?, ?, ?, ?)",
                                    (assay.reference.key, assay.assay_id, assay.sample_id,
                                     _json(assay.to_dict())))
+            for pair in pairings:
+                if not isinstance(pair, AssayPair):
+                    raise TypeError("pairings must contain AssayPairs")
+                if (pair.dataset_id, pair.dataset_version) != (manifest.dataset_id, manifest.version):
+                    raise ValueError("Pairing belongs to a different dataset release")
+                if pair.basis in ("same_material", "adjacent_donor_tissue"):
+                    units = []
+                    for reference, identifier in ((pair.reference, pair.left_assay_id),
+                                                  (pair.right_reference, pair.right_assay_id)):
+                        row = connection.execute(
+                            "SELECT s.record FROM assay a JOIN sample s ON a.reference_key=s.reference_key "
+                            "AND a.sample_id=s.id WHERE a.reference_key=? AND a.id=?",
+                            (reference.key, identifier)).fetchone()
+                        if row:
+                            unit = json.loads(row[0]).get("biological_unit_id")
+                            if unit is not None:
+                                units.append(unit)
+                    if len(set(units)) > 1:
+                        raise ValueError("Declared pairing contradicts biological-unit identities")
+                connection.execute("INSERT INTO assay_pair VALUES (?, ?, ?, ?, ?, ?)",
+                                   (pair.reference.key, pair.pair_id, pair.right_reference.key, pair.left_assay_id,
+                                    pair.right_assay_id, _json(pair.to_dict())))
+            relation_links = ({}, {})
+            for relation in relations:
+                if not isinstance(relation, TargetRelation):
+                    raise TypeError("relations must contain TargetRelations")
+                if (relation.dataset_id, relation.dataset_version) != (manifest.dataset_id, manifest.version):
+                    raise ValueError("Target relation belongs to a different dataset release")
+                for target in (relation.left, relation.right):
+                    connection.execute("INSERT OR IGNORE INTO target VALUES (?, ?, ?, ?, ?)",
+                                       (target.reference.key, _target_key(target), target.kind,
+                                        target.namespace, target.identifier))
+                for index, (first, second) in enumerate(((relation.left, relation.right), (relation.right, relation.left))):
+                    _check_one_to_one(relation_links[index],
+                                      (relation.relation, first.key, second.reference.key, second.kind, second.namespace), second.key,
+                                      relation.cardinality == "one_to_one")
+                connection.execute("INSERT INTO target_relation VALUES (?, ?, ?, ?, ?, ?)",
+                                   (relation.left.reference.key, relation.relation_id,
+                                    relation.right.reference.key,
+                                    _target_key(relation.left), _target_key(relation.right),
+                                    _json(relation.to_dict())))
             for mapping in mappings:
                 if not isinstance(mapping, TargetMapping):
                     raise TypeError("mappings must contain TargetMappings")
                 connection.execute("INSERT INTO mapping VALUES (?, ?, ?)",
                                    (mapping.reference.key, mapping.mapping_id,
                                     _json(mapping.to_dict())))
+            for mapping in reference_mappings:
+                if not isinstance(mapping, ReferenceMapping):
+                    raise TypeError("reference_mappings must contain ReferenceMappings")
+                if (mapping.dataset_id, mapping.dataset_version) != (manifest.dataset_id, manifest.version):
+                    raise ValueError("Reference mapping belongs to a different dataset release")
+                for target in (mapping.source, *mapping.targets):
+                    connection.execute("INSERT OR IGNORE INTO target VALUES (?, ?, ?, ?, ?)",
+                                       (target.reference.key, _target_key(target), target.kind,
+                                        target.namespace, target.identifier))
+                connection.execute("INSERT INTO reference_mapping VALUES (?, ?, ?, ?, ?)",
+                                   (mapping.source.reference.key, mapping.mapping_id, _target_key(mapping.source),
+                                    mapping.target_reference.key, _json(mapping.to_dict())))
+                connection.executemany("INSERT INTO reference_assignment VALUES (?, ?, ?, ?)",
+                                       ((mapping.source.reference.key, mapping.mapping_id,
+                                         target.reference.key, _target_key(target)) for target in mapping.targets))
             source_ids = {s.asset_id for s in manifest.sources}
             for observation in measurements:
                 if not isinstance(observation, Measurement):
@@ -140,15 +236,17 @@ class QuantitativeDataset:
                 source = observation.provenance.get("source_asset")
                 if source is not None and source not in source_ids:
                     raise ValueError("Measurement provenance names an unknown source asset")
-                if source is None and not observation.input_measurements:
+                if source is None and not (observation.input_measurements or observation.input_references):
                     raise ValueError("Measurement requires a source_asset or input measurements")
                 reference_key = observation.target.reference.key
-                for identifier in observation.input_measurements:
+                inputs = [(reference_key, identifier) for identifier in observation.input_measurements]
+                inputs.extend((value.reference.key, value.measurement_id) for value in observation.input_references)
+                for input_reference, identifier in inputs:
                     if connection.execute(
                         "SELECT 1 FROM measurement WHERE reference_key=? AND id=?",
-                        (reference_key, identifier),
+                        (input_reference, identifier),
                     ).fetchone() is None:
-                        raise ValueError("Derived inputs must precede outputs in the same reference context")
+                        raise ValueError("Derived inputs must precede outputs in their declared reference contexts")
                 target = observation.target
                 target_key = _target_key(target)
                 connection.execute("INSERT OR IGNORE INTO target VALUES (?, ?, ?, ?, ?)",
@@ -160,11 +258,12 @@ class QuantitativeDataset:
                                    (reference_key, observation.measurement_id, target_key,
                                     observation.sample_id, observation.assay_id,
                                     observation.quantity, observation.status, _json(record)))
-                connection.executemany("INSERT INTO input_measurement VALUES (?, ?, ?)",
-                                       ((reference_key, observation.measurement_id, identifier)
-                                        for identifier in observation.input_measurements))
+                connection.executemany("INSERT INTO input_measurement VALUES (?, ?, ?, ?)",
+                                       ((reference_key, observation.measurement_id, input_reference, identifier)
+                                        for input_reference, identifier in inputs))
             counts = {table: connection.execute(f"SELECT count(*) FROM {table}").fetchone()[0]
-                      for table in ("sample", "assay", "mapping", "target", "measurement")}
+                      for table in ("sample", "assay", "mapping", "target", "measurement",
+                                    "assay_pair", "target_relation", "reference_mapping")}
             connection.execute("INSERT INTO metadata VALUES ('counts', ?)", (_json(counts),))
             connection.commit()
             connection.close()
@@ -205,6 +304,108 @@ class QuantitativeDataset:
     def mappings(self, *, reference=None):
         return self._records("mapping", TargetMapping, reference=reference)
 
+    def reference_mappings(self, *, source_reference=None, target=None):
+        """Iterate original source-to-reference mappings, including unresolved IDs."""
+        scope = self._scope(source_reference)
+        if target is not None:
+            if not isinstance(target, ScopedTarget):
+                raise TypeError("target must be a ScopedTarget")
+            self._scope(target.reference)
+        if self._storage_version < 3:
+            return
+        query = "SELECT m.record FROM reference_mapping m"
+        conditions, params = [], []
+        if target is not None:
+            query += (" JOIN reference_assignment a ON m.reference_key=a.source_reference_key "
+                      "AND m.id=a.mapping_id")
+            conditions.extend(("a.target_reference_key=?", "a.target_key=?"))
+            params.extend((target.reference.key, _target_key(target)))
+        if scope is not None:
+            conditions.append("m.reference_key=?")
+            params.append(scope)
+        if conditions:
+            query += " WHERE " + " AND ".join(conditions)
+        query += " ORDER BY m.reference_key, m.id"
+        with closing(self._connect()) as connection:
+            for row in connection.execute(query, params):
+                yield ReferenceMapping.from_dict(json.loads(row[0]))
+
+    def mapped_measurements(self, target, *, mapping_statuses=("unique",),
+                            require_sequence=False, **observation_filters):
+        """Return original values plus mappings, never relabeled measurements.
+
+        Ambiguous or unresolved candidate membership requires explicit status
+        opt-in. Gene RNA cannot map to ORF abundance. require_sequence excludes
+        identifier-only mappings; ORF assignments always require their own
+        sequence/frame/position validation when the mapping is constructed.
+        """
+        allowed = set(mapping_statuses)
+        if not allowed or allowed - {"unique", "ambiguous", "unresolved"}:
+            raise ValueError("Choose explicit usable mapping statuses")
+        if type(require_sequence) is not bool:
+            raise TypeError("require_sequence must be a boolean")
+        if set(observation_filters) - {"sample_id", "assay_id", "quantity", "status"}:
+            raise ValueError("Mapped observation filters cannot override source/target reference identity")
+        for mapping in self.reference_mappings(target=target):
+            if mapping.status not in allowed or (require_sequence and not mapping.sequence_validated):
+                continue
+            for observation in self.measurements(target=mapping.source, **observation_filters):
+                yield MappedMeasurement(observation, mapping, target)
+
+    def pairings(self, *, reference=None):
+        """Iterate source-declared pairings; legacy releases have none."""
+        if self._storage_version == 1:
+            self._scope(reference)
+            return iter(())
+        return self._records("assay_pair", AssayPair, reference=reference)
+
+    def relations(self, *, reference=None):
+        """Iterate same-species relationships, optionally scoped by their left reference."""
+        if self._storage_version == 1:
+            self._scope(reference)
+            return iter(())
+        return self._records("target_relation", TargetRelation, reference=reference)
+
+    def matched_measurements(self, *, allowed_bases, allowed_temporal=("concurrent",),
+                             reference=None, pair_ids=None, relation_ids=None):
+        """Iterate explicit assay/target matches, preserving all values and missing mates.
+
+        Defaults require concurrent assays; opting into longitudinal or unknown
+        timing is explicit. No matching by tissue names or inferred orthology.
+        Each side contains its own original observations, never a Cartesian
+        multiplication or an aggregate over technical replicates/quantities.
+        """
+        bases, temporal = set(allowed_bases), set(allowed_temporal)
+        if isinstance(pair_ids, str) or isinstance(relation_ids, str):
+            raise TypeError("Relationship ID selections must be sequences, not a single string")
+        if not bases or bases - set(PAIRING_BASES):
+            raise ValueError("Specify valid allowed pairing bases explicitly")
+        if not temporal or temporal - set(TEMPORAL_RELATIONS):
+            raise ValueError("Specify valid allowed temporal relationships")
+        if (pair_ids or relation_ids) and reference is None and len(self._references) > 1:
+            raise ValueError("Relationship ID queries across species require a reference")
+        relations = tuple(relation for relation in self.relations(reference=reference)
+                          if relation_ids is None or relation.relation_id in relation_ids)
+        for pair in self.pairings(reference=reference):
+            if pair.basis not in bases or pair.temporal_relation not in temporal:
+                continue
+            if pair_ids is not None and pair.pair_id not in pair_ids:
+                continue
+            by_target = []
+            for assay_reference, assay in ((pair.reference, pair.left_assay_id),
+                                           (pair.right_reference, pair.right_assay_id)):
+                values = {}
+                if assay is not None:
+                    for value in self.measurements(reference=assay_reference, assay_id=assay):
+                        values.setdefault(value.target.key, []).append(value)
+                by_target.append(values)
+            for relation in relations:
+                if (relation.left.reference.key == pair.reference.key
+                        and relation.right.reference.key == pair.right_reference.key):
+                    yield MatchedMeasurements(pair, relation,
+                                              tuple(by_target[0].get(relation.left.key, ())),
+                                              tuple(by_target[1].get(relation.right.key, ())))
+
     def measurements(self, *, target=None, reference=None, sample_id=None,
                      assay_id=None, quantity=None, status=None):
         """Iterate native observations; a target match is exact, never orthology.
@@ -239,6 +440,9 @@ class QuantitativeDataset:
             for row in connection.execute(query, [value for _, value in selected]):
                 values = json.loads(row[0])
                 values["target"] = ScopedTarget(self._references[row[1]], row[2], row[4], row[3])
+                if "input_references" in values:
+                    values["input_references"] = tuple(MeasurementReference.from_dict(value)
+                                                       for value in values["input_references"])
                 yield Measurement(**values)
 
     @property