orfdata 0.4.0__tar.gz → 0.5.0__tar.gz

{orfdata-0.4.0/orfdata.egg-info → orfdata-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: orfdata
-Version: 0.4.0
+Version: 0.5.0
 Summary: Reference open reading frames and experimental translation evidence on annotated transcripts
 Author: OpenVax
 License-Expression: Apache-2.0

{orfdata-0.4.0 → orfdata-0.5.0}/docs/measurements.md

@@ -46,6 +46,10 @@ mapping table. The default PyEnsembl target uses its unversioned `id`; request
 mapping across annotation releases, gene isoforms, ORFs or species. A variant or
 fusion product does not inherit a native reference protein measurement.
 
+For native sources with unreported genomic metadata, use explicit
+[source references and mapping views](source-references.md) to retain the original
+observation alongside its mapping to a PyEnsembl target.
+
 The CLI uses the same operations:
 
 ```sh
@@ -96,7 +100,10 @@ An importer calls `QuantitativeDataset.create(path, manifest, samples=...,
 assays=..., measurements=..., mappings=...)`. These inputs may be generators;
 measurements are written incrementally. Every observation names a source asset
 or earlier input measurements. Derived inputs precede outputs in the same
-reference context. Duplicate IDs, broken foreign keys, wrong dataset releases
+dataset release. `input_measurements` names IDs within the output reference;
+`input_references` instead uses `MeasurementReference` records with each input's
+dataset, version, reference and ID. The two forms cannot be mixed. Cross-species
+derived abundance is rejected. Duplicate IDs, broken foreign keys, wrong dataset releases
 and cross-reference links fail the import. Failed builds do not replace data.
 
 Use `build_dataset(manifest, builder, ...)` for a managed local import.

{orfdata-0.4.0 → orfdata-0.5.0}/docs/pairing.md

@@ -25,6 +25,13 @@ only explicitly concurrent assays are returned; opt into `longitudinal` or
 TMT channels and bridges remain in the assays and do not become biological
 sample counts.
 
+RNA annotation and protein search references can differ. `AssayPair.reference`
+is the left assay's context; `right_reference` explicitly identifies the right
+assay's context and defaults to the left for older records. Both must belong to
+the same species. A protein FASTA does not need an invented genomic assembly:
+see [source references](source-references.md). Original targets and references
+remain attached to both sides, even when their local IDs happen to match.
+
 The CLI streams the same table rows:
 
 ```sh
@@ -39,10 +46,13 @@ and cardinality. `same_target` compares one target across assays/samples.
 `parent_rna_context` links parent RNA to an ORF but keeps the RNA value attached
 to its original gene/transcript. `shared_group_member` records an ambiguous
 protein-group assignment; the group value stays a group value.
+`gene_protein_group_context` links RNA to an ambiguous source protein group
+without assigning that group's abundance to an individual protein or ORF.
 
 Importers pass `AssayPair` and `TargetRelation` iterators as `pairings=` and
 `relations=` to `QuantitativeDataset.create`. IDs belong to the dataset release
-and reference context. Invalid assay references, contradictory known donor
+and left reference context. The right assay/target keeps its own context.
+Invalid assay references, contradictory known donor
 identities and conflicting one-to-one assignments fail the import. Original
 storage-version-1 datasets remain readable and have no declared relationships.
 
@@ -67,7 +77,10 @@ assays. Missing, censored or nonpositive denominators produce an unresolved
 `not_measured` result with its reason, never infinity or a measured zero. RPF/RNA
 ratios are source-defined translation-efficiency measures, not measured flux.
 All derived values retain their input IDs, pairing, target attribution and
-normalization assumptions.
+normalization assumptions. When input reference contexts differ, the ratio
+stores `MeasurementReference` records in `input_references`, rather than
+unqualified local IDs. This preserves source identity across annotation and
+search-FASTA releases.
 
 ## Across species
 

{orfdata-0.4.0 → orfdata-0.5.0}/docs/reference.md

@@ -75,9 +75,8 @@ reconstructed RNA products and preserves each candidate's RNA support. For
 example, the native TPST1 peptide KIILSKISV is absent from the Sid TPST1–CRCP
 alternatives, so its MS evidence does not support those altered products.
 
-The reference integration is available in Varcode 10.12.0; the Isovar 1.50.0
-integration is in review. The stricter reference-context contract is coordinated
-with Varcode 10.13.0 and Isovar 1.51.0.
+The reference integration, including strict reference-context validation, is
+available in Varcode 10.13.0 and Isovar 1.51.0.
 
 ## Data and limits
 

orfdata-0.5.0/docs/source-references.md

@@ -0,0 +1,90 @@
+# Source references and PyEnsembl mappings
+
+A proteomics experiment may search a known protein FASTA without naming a
+genomic assembly. An older RNA study may report its annotation provider but
+omit the release. ORFdata keeps these source identities and their unknown fields.
+
+```python
+from orfdata import ReferenceContext
+
+source = ReferenceContext(
+    taxon_id=9615, species="dog", genome_build=None,
+    annotation_name="uniprot", annotation_version=None,
+    sequence_sha256=searched_fasta_sha256,
+)
+```
+
+Unknown assembly/provider fields require a pinned `sequence_sha256` or
+`source_sha256`. The former identifies a sequence set; the latter can pin an
+input table when its reference release is unreported. These source contexts
+cannot bind a genomic transcript. A `ReferenceORF` still requires a complete
+genomic model validated against its actual PyEnsembl transcript. Existing
+complete-reference keys are unchanged.
+
+## Query through an explicit mapping
+
+`ReferenceMapping` connects a native `ScopedTarget` to all candidate targets in
+one destination reference. It records the dataset/version, mapping method,
+validation, source provenance, status and unresolved/unmapped reason. Importers
+must perform the stated checks against the pinned inputs. Unique identifier
+resolution is separate from sequence validation.
+
+```python
+from orfdata import ScopedTarget
+
+target = ScopedTarget.from_pyensembl(gene, kind="gene", namespace="ensembl:110")
+for view in dataset.mapped_measurements(target):
+    print(view.measurement.value, view.measurement.unit)
+    print(view.measurement.target, view.requested_target)
+    print(view.mapping.method, view.mapping.validation)
+```
+
+The view contains the **original observation**, including its source reference,
+target, sample, assay, unit and provenance. Nothing is relabeled or copied onto
+the requested target. Direct `dataset.measurements(target=...)` still returns
+only observations native to that exact target/reference.
+
+By default, mapped queries return uniquely resolved assignments. Use
+`mapping_statuses=["ambiguous", "unresolved"]` to inspect other candidates, and
+`require_sequence=True` to require recorded exact sequence validation. A group
+value returned for a candidate member has `attribution="group_membership_context"`;
+it does not independently quantify that protein. Parent gene/transcript RNA
+cannot map directly onto an ORF. Use a separate parent-context relationship.
+
+The CLI exposes the same query:
+
+```sh
+orfdata datasets query DATASET --mapped --reference ref:SHA256 \
+    --kind protein --namespace ensembl:110 --target PROTEIN_ID --require-sequence
+```
+
+## Importer contract
+
+Pass `reference_mappings=` to `QuantitativeDataset.create`. Declare source and
+destination contexts in the manifest. Each source/destination-reference pair
+has one mapping containing all candidates, preventing competing unique claims.
+Unmapped records remain visible through `dataset.reference_mappings()`.
+
+The status is `unique`, `ambiguous`, `unmapped` or `unresolved`. A single reported
+protein representative with incomplete group membership is unresolved; its
+measurement cannot become a uniquely measured member. Identifier-based mappings
+must say what was checked and leave sequence validation absent when unavailable.
+For an exact sequence search, record:
+
+```python
+validation = {"sequence": {
+    "source_sha256": source_protein_sequence_sha256,
+    "target_sha256": matching_protein_sequence_sha256,
+    "all_candidates_checked": True,
+}}
+```
+
+These are hashes of the compared sequences, distinct from the whole searched
+FASTA's hash in the reference context. Both hashes must agree to count as an
+exact match. ORF assignments additionally require their own `frame` and
+`position` evidence. Known reference-strain differences need explicit
+`strain_relationship` evidence. Cross-species mappings use the separate
+[orthology API](pairing.md#across-species).
+
+Storage version 3 preserves these mappings. Earlier versions remain readable
+and report no declared reference mappings; no mapping is inferred on load.

{orfdata-0.4.0 → orfdata-0.5.0}/orfdata/__init__.py

@@ -15,7 +15,15 @@ from .catalog import (
 from .dataset import QuantitativeDataset
 from .datasets import DatasetRegistry, build_dataset
 from .initiation_context import KOZAK_PREFERENCES, InitiationContext, annotate_initiation_context
-from .measurements import Assay, DatasetManifest, Measurement, Sample, SourceAsset, TargetMapping
+from .measurements import (
+    Assay,
+    DatasetManifest,
+    Measurement,
+    MeasurementReference,
+    Sample,
+    SourceAsset,
+    TargetMapping,
+)
 from .model import ORFModel, ReferenceORF
 from .pairing import (
     AssayPair,
@@ -26,6 +34,7 @@ from .pairing import (
     compare_orthologs,
 )
 from .reference import ReferenceContext, ScopedTarget
+from .reference_mapping import MappedMeasurement, ReferenceMapping
 from .version import __version__
 
 __all__ = [
@@ -47,6 +56,7 @@ __all__ = [
     "Sample",
     "Assay",
     "Measurement",
+    "MeasurementReference",
     "TargetMapping",
     "AssayPair",
     "TargetRelation",
@@ -54,6 +64,8 @@ __all__ = [
     "OrthologyMapping",
     "OrthologyComparison",
     "compare_orthologs",
+    "ReferenceMapping",
+    "MappedMeasurement",
     "load_catalog",
     "UORFCatalog",
     "UORFRecord",

{orfdata-0.4.0 → orfdata-0.5.0}/orfdata/dataset.py

@@ -13,6 +13,7 @@ from .measurements import (
     Assay,
     DatasetManifest,
     Measurement,
+    MeasurementReference,
     Sample,
     TargetMapping,
 )
@@ -25,6 +26,7 @@ from .pairing import (
     _check_one_to_one,
 )
 from .reference import ReferenceContext, ScopedTarget
+from .reference_mapping import MappedMeasurement, ReferenceMapping
 
 _SCHEMA = """
 CREATE TABLE metadata (name TEXT PRIMARY KEY, value TEXT NOT NULL);
@@ -52,25 +54,41 @@ CREATE TABLE measurement (
     FOREIGN KEY(reference_key, assay_id, sample_id)
         REFERENCES assay(reference_key, id, sample_id));
 CREATE TABLE input_measurement (
-    reference_key TEXT NOT NULL, output_id TEXT NOT NULL, input_id TEXT NOT NULL,
-    PRIMARY KEY(reference_key, output_id, input_id),
+    reference_key TEXT NOT NULL, output_id TEXT NOT NULL,
+    input_reference_key TEXT NOT NULL, input_id TEXT NOT NULL,
+    PRIMARY KEY(reference_key, output_id, input_reference_key, input_id),
     FOREIGN KEY(reference_key, output_id) REFERENCES measurement(reference_key, id),
-    FOREIGN KEY(reference_key, input_id) REFERENCES measurement(reference_key, id));
+    FOREIGN KEY(input_reference_key, input_id) REFERENCES measurement(reference_key, id));
 CREATE INDEX target_lookup ON target(reference_key, namespace, kind, identifier);
 CREATE INDEX observation_lookup ON measurement(reference_key, target_key, sample_id, assay_id);
 CREATE INDEX assay_lookup ON measurement(reference_key, assay_id, quantity, status);
 CREATE TABLE assay_pair (
     reference_key TEXT NOT NULL REFERENCES reference(key), id TEXT NOT NULL,
+    right_reference_key TEXT NOT NULL REFERENCES reference(key),
     left_assay_id TEXT, right_assay_id TEXT, record TEXT NOT NULL,
     PRIMARY KEY(reference_key, id),
     FOREIGN KEY(reference_key, left_assay_id) REFERENCES assay(reference_key, id),
-    FOREIGN KEY(reference_key, right_assay_id) REFERENCES assay(reference_key, id));
+    FOREIGN KEY(right_reference_key, right_assay_id) REFERENCES assay(reference_key, id));
 CREATE TABLE target_relation (
     reference_key TEXT NOT NULL, id TEXT NOT NULL,
+    right_reference_key TEXT NOT NULL,
     left_target_key TEXT NOT NULL, right_target_key TEXT NOT NULL, record TEXT NOT NULL,
     PRIMARY KEY(reference_key, id),
     FOREIGN KEY(reference_key, left_target_key) REFERENCES target(reference_key, key),
-    FOREIGN KEY(reference_key, right_target_key) REFERENCES target(reference_key, key));
+    FOREIGN KEY(right_reference_key, right_target_key) REFERENCES target(reference_key, key));
+CREATE TABLE reference_mapping (
+    reference_key TEXT NOT NULL, id TEXT NOT NULL, source_target_key TEXT NOT NULL,
+    target_reference_key TEXT NOT NULL REFERENCES reference(key), record TEXT NOT NULL,
+    PRIMARY KEY(reference_key, id),
+    UNIQUE(reference_key, source_target_key, target_reference_key),
+    FOREIGN KEY(reference_key, source_target_key) REFERENCES target(reference_key, key));
+CREATE TABLE reference_assignment (
+    source_reference_key TEXT NOT NULL, mapping_id TEXT NOT NULL,
+    target_reference_key TEXT NOT NULL, target_key TEXT NOT NULL,
+    PRIMARY KEY(source_reference_key, mapping_id, target_reference_key, target_key),
+    FOREIGN KEY(source_reference_key, mapping_id) REFERENCES reference_mapping(reference_key, id),
+    FOREIGN KEY(target_reference_key, target_key) REFERENCES target(reference_key, key));
+CREATE INDEX reference_assignment_lookup ON reference_assignment(target_reference_key, target_key);
 """
 
 
@@ -94,7 +112,7 @@ class QuantitativeDataset:
         self.path = Path(path).resolve()
         with closing(self._connect()) as connection:
             self._storage_version = connection.execute("PRAGMA user_version").fetchone()[0]
-            if self._storage_version not in (1, 2):
+            if self._storage_version not in (1, 2, 3):
                 raise ValueError("Unsupported quantitative database schema")
             row = connection.execute("SELECT value FROM metadata WHERE name='manifest'").fetchone()
             if row is None:
@@ -110,7 +128,7 @@ class QuantitativeDataset:
 
     @classmethod
     def create(cls, path, manifest, *, samples, assays, measurements, mappings=(),
-               pairings=(), relations=()):
+               pairings=(), relations=(), reference_mappings=()):
         """Write a new dataset from iterators, validating scope and foreign keys.
 
         Never overwrite an existing file. An interrupted/invalid build removes
@@ -128,7 +146,7 @@ class QuantitativeDataset:
             connection = sqlite3.connect(path)
             connection.execute("PRAGMA foreign_keys=ON")
             connection.executescript(_SCHEMA)
-            connection.execute("PRAGMA user_version=2")
+            connection.execute("PRAGMA user_version=3")
             connection.execute("INSERT INTO metadata VALUES ('manifest', ?)",
                                (_json(manifest.to_dict()),))
             for reference in manifest.references:
@@ -154,19 +172,20 @@ class QuantitativeDataset:
                     raise ValueError("Pairing belongs to a different dataset release")
                 if pair.basis in ("same_material", "adjacent_donor_tissue"):
                     units = []
-                    for identifier in (pair.left_assay_id, pair.right_assay_id):
+                    for reference, identifier in ((pair.reference, pair.left_assay_id),
+                                                  (pair.right_reference, pair.right_assay_id)):
                         row = connection.execute(
                             "SELECT s.record FROM assay a JOIN sample s ON a.reference_key=s.reference_key "
                             "AND a.sample_id=s.id WHERE a.reference_key=? AND a.id=?",
-                            (pair.reference.key, identifier)).fetchone()
+                            (reference.key, identifier)).fetchone()
                         if row:
                             unit = json.loads(row[0]).get("biological_unit_id")
                             if unit is not None:
                                 units.append(unit)
                     if len(set(units)) > 1:
                         raise ValueError("Declared pairing contradicts biological-unit identities")
-                connection.execute("INSERT INTO assay_pair VALUES (?, ?, ?, ?, ?)",
-                                   (pair.reference.key, pair.pair_id, pair.left_assay_id,
+                connection.execute("INSERT INTO assay_pair VALUES (?, ?, ?, ?, ?, ?)",
+                                   (pair.reference.key, pair.pair_id, pair.right_reference.key, pair.left_assay_id,
                                     pair.right_assay_id, _json(pair.to_dict())))
             relation_links = ({}, {})
             for relation in relations:
@@ -180,10 +199,11 @@ class QuantitativeDataset:
                                         target.namespace, target.identifier))
                 for index, (first, second) in enumerate(((relation.left, relation.right), (relation.right, relation.left))):
                     _check_one_to_one(relation_links[index],
-                                      (relation.relation, first.key, second.kind, second.namespace), second.key,
+                                      (relation.relation, first.key, second.reference.key, second.kind, second.namespace), second.key,
                                       relation.cardinality == "one_to_one")
-                connection.execute("INSERT INTO target_relation VALUES (?, ?, ?, ?, ?)",
+                connection.execute("INSERT INTO target_relation VALUES (?, ?, ?, ?, ?, ?)",
                                    (relation.left.reference.key, relation.relation_id,
+                                    relation.right.reference.key,
                                     _target_key(relation.left), _target_key(relation.right),
                                     _json(relation.to_dict())))
             for mapping in mappings:
@@ -192,6 +212,21 @@ class QuantitativeDataset:
                 connection.execute("INSERT INTO mapping VALUES (?, ?, ?)",
                                    (mapping.reference.key, mapping.mapping_id,
                                     _json(mapping.to_dict())))
+            for mapping in reference_mappings:
+                if not isinstance(mapping, ReferenceMapping):
+                    raise TypeError("reference_mappings must contain ReferenceMappings")
+                if (mapping.dataset_id, mapping.dataset_version) != (manifest.dataset_id, manifest.version):
+                    raise ValueError("Reference mapping belongs to a different dataset release")
+                for target in (mapping.source, *mapping.targets):
+                    connection.execute("INSERT OR IGNORE INTO target VALUES (?, ?, ?, ?, ?)",
+                                       (target.reference.key, _target_key(target), target.kind,
+                                        target.namespace, target.identifier))
+                connection.execute("INSERT INTO reference_mapping VALUES (?, ?, ?, ?, ?)",
+                                   (mapping.source.reference.key, mapping.mapping_id, _target_key(mapping.source),
+                                    mapping.target_reference.key, _json(mapping.to_dict())))
+                connection.executemany("INSERT INTO reference_assignment VALUES (?, ?, ?, ?)",
+                                       ((mapping.source.reference.key, mapping.mapping_id,
+                                         target.reference.key, _target_key(target)) for target in mapping.targets))
             source_ids = {s.asset_id for s in manifest.sources}
             for observation in measurements:
                 if not isinstance(observation, Measurement):
@@ -201,15 +236,17 @@ class QuantitativeDataset:
                 source = observation.provenance.get("source_asset")
                 if source is not None and source not in source_ids:
                     raise ValueError("Measurement provenance names an unknown source asset")
-                if source is None and not observation.input_measurements:
+                if source is None and not (observation.input_measurements or observation.input_references):
                     raise ValueError("Measurement requires a source_asset or input measurements")
                 reference_key = observation.target.reference.key
-                for identifier in observation.input_measurements:
+                inputs = [(reference_key, identifier) for identifier in observation.input_measurements]
+                inputs.extend((value.reference.key, value.measurement_id) for value in observation.input_references)
+                for input_reference, identifier in inputs:
                     if connection.execute(
                         "SELECT 1 FROM measurement WHERE reference_key=? AND id=?",
-                        (reference_key, identifier),
+                        (input_reference, identifier),
                     ).fetchone() is None:
-                        raise ValueError("Derived inputs must precede outputs in the same reference context")
+                        raise ValueError("Derived inputs must precede outputs in their declared reference contexts")
                 target = observation.target
                 target_key = _target_key(target)
                 connection.execute("INSERT OR IGNORE INTO target VALUES (?, ?, ?, ?, ?)",
@@ -221,12 +258,12 @@ class QuantitativeDataset:
                                    (reference_key, observation.measurement_id, target_key,
                                     observation.sample_id, observation.assay_id,
                                     observation.quantity, observation.status, _json(record)))
-                connection.executemany("INSERT INTO input_measurement VALUES (?, ?, ?)",
-                                       ((reference_key, observation.measurement_id, identifier)
-                                        for identifier in observation.input_measurements))
+                connection.executemany("INSERT INTO input_measurement VALUES (?, ?, ?, ?)",
+                                       ((reference_key, observation.measurement_id, input_reference, identifier)
+                                        for input_reference, identifier in inputs))
             counts = {table: connection.execute(f"SELECT count(*) FROM {table}").fetchone()[0]
                       for table in ("sample", "assay", "mapping", "target", "measurement",
-                                    "assay_pair", "target_relation")}
+                                    "assay_pair", "target_relation", "reference_mapping")}
             connection.execute("INSERT INTO metadata VALUES ('counts', ?)", (_json(counts),))
             connection.commit()
             connection.close()
@@ -267,6 +304,54 @@ class QuantitativeDataset:
     def mappings(self, *, reference=None):
         return self._records("mapping", TargetMapping, reference=reference)
 
+    def reference_mappings(self, *, source_reference=None, target=None):
+        """Iterate original source-to-reference mappings, including unresolved IDs."""
+        scope = self._scope(source_reference)
+        if target is not None:
+            if not isinstance(target, ScopedTarget):
+                raise TypeError("target must be a ScopedTarget")
+            self._scope(target.reference)
+        if self._storage_version < 3:
+            return
+        query = "SELECT m.record FROM reference_mapping m"
+        conditions, params = [], []
+        if target is not None:
+            query += (" JOIN reference_assignment a ON m.reference_key=a.source_reference_key "
+                      "AND m.id=a.mapping_id")
+            conditions.extend(("a.target_reference_key=?", "a.target_key=?"))
+            params.extend((target.reference.key, _target_key(target)))
+        if scope is not None:
+            conditions.append("m.reference_key=?")
+            params.append(scope)
+        if conditions:
+            query += " WHERE " + " AND ".join(conditions)
+        query += " ORDER BY m.reference_key, m.id"
+        with closing(self._connect()) as connection:
+            for row in connection.execute(query, params):
+                yield ReferenceMapping.from_dict(json.loads(row[0]))
+
+    def mapped_measurements(self, target, *, mapping_statuses=("unique",),
+                            require_sequence=False, **observation_filters):
+        """Return original values plus mappings, never relabeled measurements.
+
+        Ambiguous or unresolved candidate membership requires explicit status
+        opt-in. Gene RNA cannot map to ORF abundance. require_sequence excludes
+        identifier-only mappings; ORF assignments always require their own
+        sequence/frame/position validation when the mapping is constructed.
+        """
+        allowed = set(mapping_statuses)
+        if not allowed or allowed - {"unique", "ambiguous", "unresolved"}:
+            raise ValueError("Choose explicit usable mapping statuses")
+        if type(require_sequence) is not bool:
+            raise TypeError("require_sequence must be a boolean")
+        if set(observation_filters) - {"sample_id", "assay_id", "quantity", "status"}:
+            raise ValueError("Mapped observation filters cannot override source/target reference identity")
+        for mapping in self.reference_mappings(target=target):
+            if mapping.status not in allowed or (require_sequence and not mapping.sequence_validated):
+                continue
+            for observation in self.measurements(target=mapping.source, **observation_filters):
+                yield MappedMeasurement(observation, mapping, target)
+
     def pairings(self, *, reference=None):
         """Iterate source-declared pairings; legacy releases have none."""
         if self._storage_version == 1:
@@ -275,7 +360,7 @@ class QuantitativeDataset:
         return self._records("assay_pair", AssayPair, reference=reference)
 
     def relations(self, *, reference=None):
-        """Iterate source-backed within-reference target relationships."""
+        """Iterate same-species relationships, optionally scoped by their left reference."""
         if self._storage_version == 1:
             self._scope(reference)
             return iter(())
@@ -307,14 +392,16 @@ class QuantitativeDataset:
             if pair_ids is not None and pair.pair_id not in pair_ids:
                 continue
             by_target = []
-            for assay in (pair.left_assay_id, pair.right_assay_id):
+            for assay_reference, assay in ((pair.reference, pair.left_assay_id),
+                                           (pair.right_reference, pair.right_assay_id)):
                 values = {}
                 if assay is not None:
-                    for value in self.measurements(reference=pair.reference, assay_id=assay):
+                    for value in self.measurements(reference=assay_reference, assay_id=assay):
                         values.setdefault(value.target.key, []).append(value)
                 by_target.append(values)
             for relation in relations:
-                if relation.left.reference.key == pair.reference.key:
+                if (relation.left.reference.key == pair.reference.key
+                        and relation.right.reference.key == pair.right_reference.key):
                     yield MatchedMeasurements(pair, relation,
                                               tuple(by_target[0].get(relation.left.key, ())),
                                               tuple(by_target[1].get(relation.right.key, ())))
@@ -353,6 +440,9 @@ class QuantitativeDataset:
             for row in connection.execute(query, [value for _, value in selected]):
                 values = json.loads(row[0])
                 values["target"] = ScopedTarget(self._references[row[1]], row[2], row[4], row[3])
+                if "input_references" in values:
+                    values["input_references"] = tuple(MeasurementReference.from_dict(value)
+                                                       for value in values["input_references"])
                 yield Measurement(**values)
 
     @property

{orfdata-0.4.0 → orfdata-0.5.0}/orfdata/dataset_cli.py

@@ -36,6 +36,12 @@ def run(argv=None, prog=None):
             subparser.add_argument("--target")
             subparser.add_argument("--kind")
             subparser.add_argument("--namespace")
+            subparser.add_argument("--mapped", action="store_true",
+                                   help="Return source observations with explicit reference mappings")
+            subparser.add_argument("--mapping-status", action="append",
+                                   help="Mapping status to include; default unique")
+            subparser.add_argument("--require-sequence", action="store_true",
+                                   help="Require recorded exact sequence validation for mapped queries")
         if command == "pairs":
             subparser.add_argument("--basis", action="append", required=True,
                                    help="Allowed pairing basis; repeat for multiple bases")
@@ -84,10 +90,19 @@ def run(argv=None, prog=None):
                     parser.error("Target queries require --target, --kind, --namespace and --reference")
                 references = {r.key: r for r in dataset.manifest.references}
                 target = ScopedTarget(references[args.reference], args.kind, args.target, args.namespace)
-            for measurement in dataset.measurements(
-                target=target, reference=args.reference, sample_id=args.sample,
-                assay_id=args.assay_id, quantity=args.quantity, status=args.status,
-            ):
+            filters = dict(sample_id=args.sample, assay_id=args.assay_id,
+                           quantity=args.quantity, status=args.status)
+            if args.mapped:
+                if target is None:
+                    parser.error("--mapped requires an exact target and reference")
+                measurements = dataset.mapped_measurements(
+                    target, mapping_statuses=args.mapping_status or ["unique"],
+                    require_sequence=args.require_sequence, **filters)
+            else:
+                if args.mapping_status or args.require_sequence:
+                    parser.error("Mapping filters require --mapped")
+                measurements = dataset.measurements(target=target, reference=args.reference, **filters)
+            for measurement in measurements:
                 sys.stdout.write(json.dumps(measurement.to_dict()) + "\n")
             return
         json.dump(result, sys.stdout, indent=2)

{orfdata-0.4.0 → orfdata-0.5.0}/orfdata/measurements.py

@@ -91,6 +91,9 @@ class _Record:
                 row["reference"] = ReferenceContext(**row["reference"])
                 targets.append(ScopedTarget(**row))
             values["targets"] = tuple(targets)
+        if "input_references" in values:
+            values["input_references"] = tuple(MeasurementReference.from_dict(row)
+                                               for row in values["input_references"])
         return cls(**values)
 
 
@@ -246,6 +249,33 @@ class Assay(_Record):
         _metadata(self, "normalization", "calibration", "attributes", "provenance")
 
 
+@dataclass(frozen=True)
+class MeasurementReference(_Record):
+    """An input observation identified across dataset releases and references."""
+
+    dataset_id: str
+    dataset_version: str
+    reference: ReferenceContext
+    measurement_id: str
+
+    def __post_init__(self):
+        for name in ("dataset_id", "dataset_version", "measurement_id"):
+            _identifier(getattr(self, name), name)
+        if not isinstance(self.reference, ReferenceContext):
+            raise TypeError("Measurement reference requires a ReferenceContext")
+
+    @property
+    def key(self):
+        return self.dataset_id, self.dataset_version, self.reference.key, self.measurement_id
+
+    @classmethod
+    def from_measurement(cls, measurement):
+        if not isinstance(measurement, Measurement):
+            raise TypeError("Expected a Measurement")
+        return cls(measurement.dataset_id, measurement.dataset_version,
+                   measurement.target.reference, measurement.measurement_id)
+
+
 @dataclass(frozen=True)
 class Measurement(_Record):
     """A native value with units, missingness, target resolution and provenance.
@@ -275,6 +305,7 @@ class Measurement(_Record):
     input_measurements: tuple = ()
     derivation: Mapping = field(default_factory=dict)
     provenance: Mapping = field(default_factory=dict)
+    input_references: tuple = ()
 
     def __post_init__(self):
         for name in ("measurement_id", "sample_id", "assay_id", "unit", "dataset_id", "dataset_version"):
@@ -314,11 +345,26 @@ class Measurement(_Record):
         if len(set(inputs)) != len(inputs) or self.measurement_id in inputs:
             raise ValueError("Derived input IDs must be unique and exclude the output")
         object.__setattr__(self, "input_measurements", inputs)
+        references = tuple(self.input_references)
+        if inputs and references:
+            raise ValueError("Choose legacy same-context input IDs or explicit input references, not both")
+        if any(not isinstance(value, MeasurementReference) for value in references):
+            raise TypeError("input_references must contain MeasurementReferences")
+        own_key = (self.dataset_id, self.dataset_version, self.target.reference.key, self.measurement_id)
+        keys = {value.key for value in references}
+        if len(keys) != len(references) or own_key in keys:
+            raise ValueError("Input references must be unique and exclude the output")
+        for value in references:
+            if (value.dataset_id, value.dataset_version) != (self.dataset_id, self.dataset_version):
+                raise ValueError("Derived inputs must belong to the same dataset release")
+            if value.reference.taxon_id != self.target.reference.taxon_id:
+                raise ValueError("Derived abundance inputs cannot cross species")
+        object.__setattr__(self, "input_references", references)
         if self.origin in ("derived", "predicted") and not self.derivation:
             raise ValueError("Derived/predicted values require a named method and assumptions")
         if self.origin in ("derived", "predicted") and isinstance(self.derivation, Mapping):
             _identifier(self.derivation.get("method"), "derivation method")
-        if self.origin == "observed" and (inputs or self.derivation):
+        if self.origin == "observed" and (inputs or references or self.derivation):
             raise ValueError("An observed value cannot contain a derivation")
         _metadata(self, "derivation", "provenance")