orfdata 0.2.0__tar.gz → 0.3.0__tar.gz

{orfdata-0.2.0/orfdata.egg-info → orfdata-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: orfdata
-Version: 0.2.0
+Version: 0.3.0
 Summary: Reference open reading frames and experimental translation evidence on annotated transcripts
 Author: OpenVax
 License-Expression: Apache-2.0
@@ -11,6 +11,10 @@ Provides-Extra: pyensembl
 Requires-Dist: pyensembl>=2.27.2; extra == "pyensembl"
 Provides-Extra: build-data
 Requires-Dist: openpyxl>=3.1; extra == "build-data"
+Provides-Extra: data
+Requires-Dist: datacache<2,>=1.25.1; extra == "data"
+Provides-Extra: columnar
+Requires-Dist: pyarrow>=14; extra == "columnar"
 Provides-Extra: dev
 Requires-Dist: pytest; extra == "dev"
 Requires-Dist: ruff; extra == "dev"
@@ -18,6 +22,7 @@ Requires-Dist: build; extra == "dev"
 Requires-Dist: twine; extra == "dev"
 Requires-Dist: openpyxl>=3.1; extra == "dev"
 Requires-Dist: pyensembl>=2.27.2; extra == "dev"
+Requires-Dist: datacache<2,>=1.25.1; extra == "dev"
 Dynamic: license-file
 
 # orfdata
@@ -95,5 +100,7 @@ bind real human, mouse, dog and yeast reference slices; those fixtures are not
 new experimental ORF catalogues. [Species and reference identity](docs/species.md).
 
 Install with `pip install orfdata[pyensembl]`. See the
+[measurement API](docs/measurements.md) for native units, explicit samples and
+assays, and DataCache-managed downloads, and the
 [quantitative, multispecies data roadmap](https://github.com/openvax/orfdata/issues/3)
-for planned RNA abundance, ribosome occupancy and protein measurements.
+for upcoming RNA abundance, ribosome occupancy and protein data releases.

{orfdata-0.2.0 → orfdata-0.3.0}/README.md

@@ -73,5 +73,7 @@ bind real human, mouse, dog and yeast reference slices; those fixtures are not
 new experimental ORF catalogues. [Species and reference identity](docs/species.md).
 
 Install with `pip install orfdata[pyensembl]`. See the
+[measurement API](docs/measurements.md) for native units, explicit samples and
+assays, and DataCache-managed downloads, and the
 [quantitative, multispecies data roadmap](https://github.com/openvax/orfdata/issues/3)
-for planned RNA abundance, ribosome occupancy and protein measurements.
+for upcoming RNA abundance, ribosome occupancy and protein data releases.

orfdata-0.3.0/docs/measurements.md

@@ -0,0 +1,134 @@
+# RNA, ribosome and protein measurements
+
+ORFdata adds evidence to PyEnsembl's reference transcripts and proteins. Its
+quantitative layer stores measurements of genes, transcripts, reference ORFs,
+proteins, protein groups and peptides. A measurement stays at the resolution
+the experiment supports. Parent-gene RNA abundance is useful context for an ORF;
+it is not a separate measurement of each ORF in that gene.
+
+Install `orfdata[data]` to manage datasets. The initial 0.3 registry is empty;
+the [data roadmap](https://github.com/openvax/orfdata/issues/3) tracks source
+imports. The bundled human ORF evidence catalogue remains available offline.
+Its pooled PSM counts describe spectra, not sample-level protein abundance.
+
+## Discover, download and query
+
+```python
+from orfdata import DatasetRegistry, ScopedTarget
+
+registry = DatasetRegistry()  # Metadata only; no download.
+releases = registry.list(species="dog", assay="digest_ms")
+```
+
+For a registered release, use `registry.info(name, version)` for its immutable
+manifest, `registry.download(name, version)` to fetch it, and
+`registry.open(name, version)` to read it offline. `inspect` reports missing,
+available or invalid files without downloading or repairing them.
+`download_metadata` returns the artifact's access terms and file hashes/sizes.
+An invalid cache
+requires an explicit `download(..., force=True)`; old reader paths survive.
+Python download options such as timeout, retries and progress go to DataCache.
+
+```python
+# dataset = registry.open(name, version)
+# transcript is a PyEnsembl Transcript in the dataset's exact reference context.
+target = ScopedTarget.from_pyensembl(
+    transcript, kind="transcript", namespace="ensembl:110"
+)
+for measurement in dataset.measurements(target=target):
+    print(measurement.value, measurement.unit, measurement.status)
+```
+
+Use `ScopedTarget.from_orf(bound_reference_orf, namespace=source_release)` for a
+validated reference ORF. Namespaces and identifiers must match the source's
+mapping table. The default PyEnsembl target uses its unversioned `id`; request
+`versioned=True` when the source uses versioned identifiers. There is no implicit
+mapping across annotation releases, gene isoforms, ORFs or species. A variant or
+fusion product does not inherit a native reference protein measurement.
+
+The CLI uses the same operations:
+
+```sh
+orfdata datasets list --species yeast
+orfdata datasets info DATASET --version RELEASE
+orfdata datasets download DATASET --version RELEASE
+orfdata datasets inspect DATASET --version RELEASE
+orfdata datasets query DATASET --version RELEASE --quantity protein_abundance
+```
+
+Queries stream JSON lines. `--reference ref:SHA256` selects an exact context;
+target queries also require `--target`, `--kind` and `--namespace`. Unqualified
+sample/assay ID queries across multiple references are rejected. Existing
+catalogue commands such as `orfdata --gene TPST1` retain their behavior.
+
+## What each value means
+
+`Measurement` retains the dataset/version, reference-scoped target, sample and
+assay IDs, value, native unit, linear/log scale and log base, uncertainty and its
+definition, detection limit, status, origin and provenance. Measured zero,
+below detection, not measured, not mapped and missing are distinct. Censored or
+missing values use `None`; native sentinels can remain in provenance.
+
+RNA counts/TPM/FPKM, ribosome occupancy, source-defined translation efficiency,
+LFQ/iBAQ/TMT values, peptide intensity, PSM counts, copies per cell and turnover
+remain distinct. No conversion is automatic. Derived/predicted values name a
+method and retain source/input provenance. An estimate from calibration remains
+an estimate; an intensity/TPM ratio does not acquire molecule-ratio units.
+
+`Sample` records biological units, pools and source labels. `Assay` separately
+records technical replicates, runs, fractions, TMT plex/channel/bridge,
+normalization and calibration. A sample's strain/breed belongs in its attributes;
+it need not be the strain used to assemble its reference genome. `TargetMapping`
+retains all candidate assignments or an unmapped reason. A protein-group
+representative does not become a uniquely measured protein. Explicit sample
+pairing and orthology comparisons follow in
+[issue #6](https://github.com/openvax/orfdata/issues/6).
+
+## Importing a source
+
+Define a `DatasetManifest` with exact reference contexts, raw source URLs,
+SHA-256 hashes and byte sizes, citations, file-specific access/redistribution
+terms, searched/measurable coverage, processing software/version/parameters and
+row accounting. Unavailable metadata stay unknown. Article licensing does not
+substitute for the data files' terms.
+
+An importer calls `QuantitativeDataset.create(path, manifest, samples=...,
+assays=..., measurements=..., mappings=...)`. These inputs may be generators;
+measurements are written incrementally. Every observation names a source asset
+or earlier input measurements. Derived inputs precede outputs in the same
+reference context. Duplicate IDs, broken foreign keys, wrong dataset releases
+and cross-reference links fail the import. Failed builds do not replace data.
+
+Use `build_dataset(manifest, builder, ...)` for a managed local import.
+`builder(source_paths, output_path)` reads the verified inputs and creates that
+SQLite dataset. DataCache publishes it atomically with input/output checksums
+and a processing receipt. `source_paths={asset_id: local_file}` supports manually
+acquired sources. Importing locally does not grant redistribution rights.
+
+A `DatasetRegistry(definitions, cache_root=...)` registers releases as:
+
+```python
+definitions = {
+    name: {
+        "default_version": version,
+        "versions": {version: {
+            "manifest": manifest.to_dict(),
+            "distribution": {"access_terms": terms, "redistributable": False},
+            "assets": {"dataset.sqlite": {
+                "url": url, "sha256": sha256, "size": size, "decompress": False,
+            }},
+        }},
+    }
+}
+```
+
+CLI users can supply this JSON with `orfdata datasets --registry FILE ...`.
+`ORFDATA_CACHE`, then `OPENVAX_DATA_CACHE`, then the platform's OpenVax cache
+select the root. ORFdata owns its `orfdata/` child. Release cache identities
+include manifest/reference and content hashes. SQLite partitions every table
+by reference key, with composite foreign keys; human, mouse, dog, yeast and
+arbitrary other taxa use the same schema. The core API needs neither pandas nor
+PyArrow. Optional `orfdata[columnar]` enables `dataset.export_parquet(path)`;
+query columns accompany full measurement JSON, preserving exact values and
+provenance even when a floating-point analysis column cannot represent an
+integer exactly.

{orfdata-0.2.0 → orfdata-0.3.0}/orfdata/__init__.py

@@ -12,7 +12,10 @@ from .catalog import (
     load_catalog,
     load_uorf_catalog,
 )
+from .dataset import QuantitativeDataset
+from .datasets import DatasetRegistry, build_dataset
 from .initiation_context import KOZAK_PREFERENCES, InitiationContext, annotate_initiation_context
+from .measurements import Assay, DatasetManifest, Measurement, Sample, SourceAsset, TargetMapping
 from .model import ORFModel, ReferenceORF
 from .reference import ReferenceContext, ScopedTarget
 from .version import __version__
@@ -28,6 +31,15 @@ __all__ = [
     "ReferenceORF",
     "ReferenceContext",
     "ScopedTarget",
+    "QuantitativeDataset",
+    "DatasetRegistry",
+    "build_dataset",
+    "DatasetManifest",
+    "SourceAsset",
+    "Sample",
+    "Assay",
+    "Measurement",
+    "TargetMapping",
     "load_catalog",
     "UORFCatalog",
     "UORFRecord",

{orfdata-0.2.0 → orfdata-0.3.0}/orfdata/cli.py

@@ -11,7 +11,14 @@ from .catalog import EVIDENCE_METHODS, UORFCatalog, load_catalog
 
 def run(argv=None, prog=None, *, catalog_loader=load_catalog):
     """Export reference records as JSON, TSV or FASTA."""
-    parser = argparse.ArgumentParser(description=__doc__, prog=prog)
+    argv = list(sys.argv[1:] if argv is None else argv)
+    if argv and argv[0] == "datasets":
+        from .dataset_cli import run as dataset_run
+
+        return dataset_run(argv[1:], prog=(prog or "orfdata") + " datasets")
+    parser = argparse.ArgumentParser(
+        description=__doc__, prog=prog,
+        epilog="For quantitative data management, use 'orfdata datasets --help'.")
     parser.add_argument("--catalog", help="Use a compatible local JSON/JSON.gz catalogue")
     parser.add_argument("--gene", help="Exact gene symbol or Ensembl gene ID")
     parser.add_argument("--transcript", help="Exact source transcript ID (unversioned)")

orfdata-0.3.0/orfdata/data/datasets.json

@@ -0,0 +1 @@
+{}

orfdata-0.3.0/orfdata/dataset.py

@@ -0,0 +1,281 @@
+"""Streaming, reference-partitioned quantitative datasets stored as SQLite."""
+
+import hashlib
+import json
+import os
+import sqlite3
+from contextlib import closing
+from pathlib import Path
+
+from .measurements import (
+    MEASUREMENT_STATUSES,
+    QUANTITIES,
+    Assay,
+    DatasetManifest,
+    Measurement,
+    Sample,
+    TargetMapping,
+)
+from .reference import ReferenceContext, ScopedTarget
+
+_SCHEMA = """
+CREATE TABLE metadata (name TEXT PRIMARY KEY, value TEXT NOT NULL);
+CREATE TABLE reference (key TEXT PRIMARY KEY, record TEXT NOT NULL);
+CREATE TABLE target (
+    reference_key TEXT NOT NULL REFERENCES reference(key),
+    key TEXT NOT NULL, kind TEXT NOT NULL, namespace TEXT NOT NULL,
+    identifier TEXT NOT NULL, PRIMARY KEY(reference_key, key));
+CREATE TABLE sample (
+    reference_key TEXT NOT NULL REFERENCES reference(key),
+    id TEXT NOT NULL, record TEXT NOT NULL, PRIMARY KEY(reference_key, id));
+CREATE TABLE assay (
+    reference_key TEXT NOT NULL, id TEXT NOT NULL, sample_id TEXT NOT NULL,
+    record TEXT NOT NULL, PRIMARY KEY(reference_key, id),
+    UNIQUE(reference_key, id, sample_id),
+    FOREIGN KEY(reference_key, sample_id) REFERENCES sample(reference_key, id));
+CREATE TABLE mapping (
+    reference_key TEXT NOT NULL REFERENCES reference(key),
+    id TEXT NOT NULL, record TEXT NOT NULL, PRIMARY KEY(reference_key, id));
+CREATE TABLE measurement (
+    reference_key TEXT NOT NULL, id TEXT NOT NULL, target_key TEXT NOT NULL,
+    sample_id TEXT NOT NULL, assay_id TEXT NOT NULL, quantity TEXT NOT NULL,
+    status TEXT NOT NULL, record TEXT NOT NULL, PRIMARY KEY(reference_key, id),
+    FOREIGN KEY(reference_key, target_key) REFERENCES target(reference_key, key),
+    FOREIGN KEY(reference_key, assay_id, sample_id)
+        REFERENCES assay(reference_key, id, sample_id));
+CREATE TABLE input_measurement (
+    reference_key TEXT NOT NULL, output_id TEXT NOT NULL, input_id TEXT NOT NULL,
+    PRIMARY KEY(reference_key, output_id, input_id),
+    FOREIGN KEY(reference_key, output_id) REFERENCES measurement(reference_key, id),
+    FOREIGN KEY(reference_key, input_id) REFERENCES measurement(reference_key, id));
+CREATE INDEX target_lookup ON target(reference_key, namespace, kind, identifier);
+CREATE INDEX observation_lookup ON measurement(reference_key, target_key, sample_id, assay_id);
+CREATE INDEX assay_lookup ON measurement(reference_key, assay_id, quantity, status);
+"""
+
+
+def _json(value):
+    return json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False)
+
+
+def _target_key(target):
+    return hashlib.sha256(_json(target.key).encode()).hexdigest()
+
+
+class QuantitativeDataset:
+    """A read-only release. Constructing or querying it never downloads data.
+
+    Local IDs are unique only within one dataset release AND reference context.
+    Iterators preserve that context even when IDs collide between species.
+    Registry-managed files are checked against pinned hashes before opening.
+    """
+
+    def __init__(self, path):
+        self.path = Path(path).resolve()
+        with closing(self._connect()) as connection:
+            if connection.execute("PRAGMA user_version").fetchone()[0] != 1:
+                raise ValueError("Unsupported quantitative database schema")
+            row = connection.execute("SELECT value FROM metadata WHERE name='manifest'").fetchone()
+            if row is None:
+                raise ValueError("Missing quantitative manifest")
+            self.manifest = DatasetManifest.from_dict(json.loads(row[0]))
+        self._references = {r.key: r for r in self.manifest.references}
+
+    def _connect(self):
+        # mode=ro prevents SQLite's default creation of an empty missing database.
+        connection = sqlite3.connect(self.path.as_uri() + "?mode=ro", uri=True)
+        connection.execute("PRAGMA query_only=ON")
+        return connection
+
+    @classmethod
+    def create(cls, path, manifest, *, samples, assays, measurements, mappings=()):
+        """Write a new dataset from iterators, validating scope and foreign keys.
+
+        Never overwrite an existing file. An interrupted/invalid build removes
+        only the new file it created. For atomic managed publication, call this
+        inside DataCache's materialize builder. Derived observations must follow
+        their input observations, preventing cyclic or dangling provenance.
+        """
+        if not isinstance(manifest, DatasetManifest):
+            raise TypeError("manifest must be a DatasetManifest")
+        path = Path(path)
+        descriptor = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
+        os.close(descriptor)
+        connection = None
+        try:
+            connection = sqlite3.connect(path)
+            connection.execute("PRAGMA foreign_keys=ON")
+            connection.executescript(_SCHEMA)
+            connection.execute("PRAGMA user_version=1")
+            connection.execute("INSERT INTO metadata VALUES ('manifest', ?)",
+                               (_json(manifest.to_dict()),))
+            for reference in manifest.references:
+                connection.execute("INSERT INTO reference VALUES (?, ?)",
+                                   (reference.key, _json(reference.to_dict())))
+            for sample in samples:
+                if not isinstance(sample, Sample):
+                    raise TypeError("samples must contain Samples")
+                connection.execute("INSERT INTO sample VALUES (?, ?, ?)",
+                                   (sample.reference.key, sample.sample_id, _json(sample.to_dict())))
+            for assay in assays:
+                if not isinstance(assay, Assay):
+                    raise TypeError("assays must contain Assays")
+                if assay.kind not in manifest.assay_kinds:
+                    raise ValueError("Assay kind absent from manifest coverage")
+                connection.execute("INSERT INTO assay VALUES (?, ?, ?, ?)",
+                                   (assay.reference.key, assay.assay_id, assay.sample_id,
+                                    _json(assay.to_dict())))
+            for mapping in mappings:
+                if not isinstance(mapping, TargetMapping):
+                    raise TypeError("mappings must contain TargetMappings")
+                connection.execute("INSERT INTO mapping VALUES (?, ?, ?)",
+                                   (mapping.reference.key, mapping.mapping_id,
+                                    _json(mapping.to_dict())))
+            source_ids = {s.asset_id for s in manifest.sources}
+            for observation in measurements:
+                if not isinstance(observation, Measurement):
+                    raise TypeError("measurements must contain Measurements")
+                if (observation.dataset_id, observation.dataset_version) != (manifest.dataset_id, manifest.version):
+                    raise ValueError("Measurement belongs to a different dataset release")
+                source = observation.provenance.get("source_asset")
+                if source is not None and source not in source_ids:
+                    raise ValueError("Measurement provenance names an unknown source asset")
+                if source is None and not observation.input_measurements:
+                    raise ValueError("Measurement requires a source_asset or input measurements")
+                reference_key = observation.target.reference.key
+                for identifier in observation.input_measurements:
+                    if connection.execute(
+                        "SELECT 1 FROM measurement WHERE reference_key=? AND id=?",
+                        (reference_key, identifier),
+                    ).fetchone() is None:
+                        raise ValueError("Derived inputs must precede outputs in the same reference context")
+                target = observation.target
+                target_key = _target_key(target)
+                connection.execute("INSERT OR IGNORE INTO target VALUES (?, ?, ?, ?, ?)",
+                                   (reference_key, target_key, target.kind,
+                                    target.namespace, target.identifier))
+                record = observation.to_dict()
+                record.pop("target")
+                connection.execute("INSERT INTO measurement VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
+                                   (reference_key, observation.measurement_id, target_key,
+                                    observation.sample_id, observation.assay_id,
+                                    observation.quantity, observation.status, _json(record)))
+                connection.executemany("INSERT INTO input_measurement VALUES (?, ?, ?)",
+                                       ((reference_key, observation.measurement_id, identifier)
+                                        for identifier in observation.input_measurements))
+            counts = {table: connection.execute(f"SELECT count(*) FROM {table}").fetchone()[0]
+                      for table in ("sample", "assay", "mapping", "target", "measurement")}
+            connection.execute("INSERT INTO metadata VALUES ('counts', ?)", (_json(counts),))
+            connection.commit()
+            connection.close()
+            connection = None
+            return cls(path)
+        except BaseException:
+            if connection is not None:
+                connection.close()
+            path.unlink()
+            raise
+
+    def _scope(self, reference):
+        if reference is None:
+            return None
+        key = reference.key if isinstance(reference, ReferenceContext) else reference
+        if key not in self._references:
+            raise ValueError("Reference context is not present in this dataset")
+        return key
+
+    def _records(self, table, cls, *, reference=None):
+        scope = self._scope(reference)
+        query = f"SELECT record FROM {table}"
+        params = ()
+        if scope is not None:
+            query += " WHERE reference_key=?"
+            params = (scope,)
+        query += " ORDER BY reference_key, id"
+        with closing(self._connect()) as connection:
+            for row in connection.execute(query, params):
+                yield cls.from_dict(json.loads(row[0]))
+
+    def samples(self, *, reference=None):
+        return self._records("sample", Sample, reference=reference)
+
+    def assays(self, *, reference=None):
+        return self._records("assay", Assay, reference=reference)
+
+    def mappings(self, *, reference=None):
+        return self._records("mapping", TargetMapping, reference=reference)
+
+    def measurements(self, *, target=None, reference=None, sample_id=None,
+                     assay_id=None, quantity=None, status=None):
+        """Iterate native observations; a target match is exact, never orthology.
+
+        Unqualified IDs across multiple references are rejected. To browse all
+        species together, omit ID filters; each result still carries its scope.
+        """
+        if quantity is not None and quantity not in QUANTITIES:
+            raise ValueError("Unknown measurement quantity")
+        if status is not None and status not in MEASUREMENT_STATUSES:
+            raise ValueError("Unknown measurement status")
+        scope = self._scope(reference)
+        if target is not None:
+            if not isinstance(target, ScopedTarget):
+                raise TypeError("target must be a ScopedTarget")
+            target_scope = self._scope(target.reference)
+            if scope is not None and scope != target_scope:
+                raise ValueError("Target and requested reference disagree")
+            scope = target_scope
+        if scope is None and len(self._references) > 1 and (sample_id or assay_id):
+            raise ValueError("An ID query across multiple references requires an explicit reference")
+        filters = dict(reference_key=scope, target_key=_target_key(target) if target else None,
+                       sample_id=sample_id, assay_id=assay_id, quantity=quantity, status=status)
+        selected = [(key, value) for key, value in filters.items() if value is not None]
+        query = ("SELECT m.record, m.reference_key, t.kind, t.namespace, t.identifier "
+                 "FROM measurement m JOIN target t ON m.reference_key=t.reference_key "
+                 "AND m.target_key=t.key")
+        if selected:
+            query += " WHERE " + " AND ".join(f"m.{key}=?" for key, _ in selected)
+        query += " ORDER BY m.reference_key, m.id"
+        with closing(self._connect()) as connection:
+            for row in connection.execute(query, [value for _, value in selected]):
+                values = json.loads(row[0])
+                values["target"] = ScopedTarget(self._references[row[1]], row[2], row[4], row[3])
+                yield Measurement(**values)
+
+    @property
+    def counts(self):
+        with closing(self._connect()) as connection:
+            result = json.loads(connection.execute(
+                "SELECT value FROM metadata WHERE name='counts'").fetchone()[0])
+        return result
+
+    def export_parquet(self, path, **query):
+        """Export query results in bounded batches; requires orfdata[columnar]."""
+        try:
+            import pyarrow as pa
+            import pyarrow.parquet as pq
+        except ImportError as exc:
+            raise ImportError("Install orfdata[columnar] to export Parquet") from exc
+        schema = pa.schema([(name, pa.string()) for name in
+                            ("dataset_id", "version", "reference_key", "measurement_id",
+                             "sample_id", "assay_id", "target_kind", "target_namespace",
+                             "target_identifier", "quantity", "unit", "status", "origin",
+                             "scale", "measurement_json")] + [("value_float64", pa.float64())])
+        with pq.ParquetWriter(path, schema) as writer:
+            batch = []
+            for value in self.measurements(**query):
+                batch.append(dict(dataset_id=self.manifest.dataset_id, version=self.manifest.version,
+                                  reference_key=value.target.reference.key,
+                                  measurement_id=value.measurement_id,
+                                  sample_id=value.sample_id, assay_id=value.assay_id,
+                                  target_kind=value.target.kind,
+                                  target_namespace=value.target.namespace,
+                                  target_identifier=value.target.identifier, quantity=value.quantity,
+                                  unit=value.unit, status=value.status, origin=value.origin,
+                                  scale=value.scale,
+                                  value_float64=float(value.value) if value.value is not None else None,
+                                  measurement_json=_json(value.to_dict())))
+                if len(batch) == 4096:
+                    writer.write_table(pa.Table.from_pylist(batch, schema=schema))
+                    batch = []
+            if batch:
+                writer.write_table(pa.Table.from_pylist(batch, schema=schema))

orfdata-0.3.0/orfdata/dataset_cli.py

@@ -0,0 +1,80 @@
+"""Discover, download, inspect and query quantitative datasets."""
+
+import argparse
+import json
+import sqlite3
+import sys
+
+from .datasets import DatasetRegistry
+from .reference import ScopedTarget
+
+
+def run(argv=None, prog=None):
+    parser = argparse.ArgumentParser(description=__doc__, prog=prog)
+    parser.add_argument("--registry", help="Local registry JSON (otherwise use packaged releases)")
+    parser.add_argument("--cache-root", help="Override the OpenVax data cache root")
+    commands = parser.add_subparsers(dest="command", required=True)
+    listing = commands.add_parser("list", help="List published releases and assay coverage")
+    listing.add_argument("--species")
+    listing.add_argument("--taxon-id", type=int)
+    listing.add_argument("--assay")
+    for command in ("info", "download", "inspect", "query"):
+        subparser = commands.add_parser(command)
+        subparser.add_argument("name")
+        subparser.add_argument("--version")
+        if command == "download":
+            subparser.add_argument("--force", action="store_true")
+            subparser.add_argument("--timeout", type=float)
+            subparser.add_argument("--quiet", action="store_true")
+            subparser.add_argument("--source-path", help="Use a local dataset.sqlite with the registered hash")
+        if command == "query":
+            subparser.add_argument("--reference", help="Exact ref:SHA256 context key")
+            subparser.add_argument("--sample")
+            subparser.add_argument("--assay-id")
+            subparser.add_argument("--quantity")
+            subparser.add_argument("--status")
+            subparser.add_argument("--target")
+            subparser.add_argument("--kind")
+            subparser.add_argument("--namespace")
+    args = parser.parse_args(argv)
+    try:
+        definitions = None
+        if args.registry:
+            with open(args.registry) as handle:
+                definitions = json.load(handle)
+        registry = DatasetRegistry(definitions, cache_root=args.cache_root)
+        if args.command == "list":
+            result = [m.to_dict() for m in registry.list(
+                species=args.species, taxon_id=args.taxon_id, assay=args.assay)]
+        elif args.command == "info":
+            result = registry.info(args.name, args.version).to_dict()
+            result["download"] = registry.download_metadata(args.name, args.version)
+        elif args.command == "inspect":
+            inspected = registry.inspect(args.name, args.version)
+            result = dict(status=inspected.status, path=str(inspected.path),
+                          error=str(inspected.error) if inspected.error else None)
+        elif args.command == "download":
+            options = dict(show_progress=not args.quiet)
+            if args.timeout is not None:
+                options["timeout"] = args.timeout
+            sources = {"dataset.sqlite": args.source_path} if args.source_path else None
+            result = registry.download(args.name, args.version, force=args.force,
+                                       source_paths=sources, **options)
+        else:
+            dataset = registry.open(args.name, args.version)
+            target = None
+            if any((args.target, args.kind, args.namespace)):
+                if not all((args.target, args.kind, args.namespace, args.reference)):
+                    parser.error("Target queries require --target, --kind, --namespace and --reference")
+                references = {r.key: r for r in dataset.manifest.references}
+                target = ScopedTarget(references[args.reference], args.kind, args.target, args.namespace)
+            for measurement in dataset.measurements(
+                target=target, reference=args.reference, sample_id=args.sample,
+                assay_id=args.assay_id, quantity=args.quantity, status=args.status,
+            ):
+                sys.stdout.write(json.dumps(measurement.to_dict()) + "\n")
+            return
+        json.dump(result, sys.stdout, indent=2)
+        sys.stdout.write("\n")
+    except (ValueError, TypeError, KeyError, OSError, ImportError, sqlite3.Error) as exc:
+        parser.error(str(exc))