altar-chrombpnet 0.1.0__tar.gz

altar_chrombpnet-0.1.0/.gitignore

@@ -0,0 +1,32 @@
+*.pyc
+*~
+**/__pycache__/*
+*.swp
+.vscode/
+.idea/
+.DS_Store
+.env*
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+.coverage
+htmlcov/
+coverage.xml
+pytest.xml
+.hypothesis/
+.python-version
+.venv/
+.venv-*/
+*.egg-info/
+build/
+dist/
+site/
+*.sqlite3
+*.log
+
+# Generated variant indexes and datasets. The small canonical gene table is tracked.
+altar/altar/variants/data/ccres.dnatree
+altar/altar/variants/data/region_annotations.parquet
+altar/altar/variants/data/variants.pkl.gz
+altar/altar/variants/data/raw/
+

altar_chrombpnet-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Riya Sinha
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

altar_chrombpnet-0.1.0/PKG-INFO

@@ -0,0 +1,62 @@
+Metadata-Version: 2.5
+Name: altar-chrombpnet
+Version: 0.1.0
+Summary: ChromBPNet model binding for large-scale Altar variant scoring
+Project-URL: Documentation, https://kundajelab.github.io/altar/
+Project-URL: Issues, https://github.com/kundajelab/altar/issues
+Project-URL: Repository, https://github.com/kundajelab/altar
+Author: Riya Sinha
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: altar<0.2,>=0.1
+Provides-Extra: test
+Requires-Dist: pytest-asyncio>=0.24; extra == 'test'
+Requires-Dist: pytest>=8; extra == 'test'
+Description-Content-Type: text/markdown
+
+# Altar ChromBPNet binding
+
+`altar-chrombpnet` is the lightweight control-plane binding between Altar and the independently locked
+ChromBPNet runtime under `runtimes/chrombpnet`. Installing it registers `CHROMBPNET` through Altar's standard
+model-plugin entry point; the base `altar` distribution does not contain ChromBPNet-specific science.
+
+```bash
+pip install altar-chrombpnet
+```
+
+```python
+from altar.models import get_model_plugin
+from altar_chrombpnet import ChromBPNetConfiguration
+
+plugin = get_model_plugin("CHROMBPNET")
+```
+
+The binding owns ChromBPNet configuration, score schemas, prioritization, preprocessing, scoring, and
+interpretation plans. It depends only on Altar's public façades and Pydantic. TensorFlow, model code, weights,
+reference genomes, peaks, and provider SDKs remain outside the binding.
+
+The binding also owns ChromBPNet's logical paths (`altar_chrombpnet.layouts`): the derived per-fold peak
+distributions and interval index, the staged peaks and motif set, and the interpretation run tree. Altar core
+provides only the model-neutral `ModelArtifactLayout` and `ScoringLayout`; every model-owned ChromBPNet file is
+composed from `model_resource_file`, so preparation and scoring name the same files under one layout. Staged
+file names come from the declared `weights_format` and `peaks_compression`, never from a resource URI.
+
+Preprocessing can emit a manifest that records the ordered weights, peaks, genome, strand policy, producer
+image, and output digests. That provenance is additive: the normal scoring plan consumes the conventional
+peak-distribution and safe interval-index paths directly and does not require a manifest, an Altar-owned
+directory, or an adoption step. Standalone runtime callers may supply both `--preprocessing-manifest` and
+`--preprocessing-identity` to opt into strict verification. Historical scores and externally managed
+artifacts are not rewritten or recomputed.
+
+Existing callers migrating from Altar core must install this package and replace imports such as
+`from altar.models import ChromBPNetPlugin` with `from altar_chrombpnet import ChromBPNetPlugin`. Registry-based
+callers need only install the package; the `CHROMBPNET` key is unchanged.

altar_chrombpnet-0.1.0/README.md

@@ -0,0 +1,37 @@
+# Altar ChromBPNet binding
+
+`altar-chrombpnet` is the lightweight control-plane binding between Altar and the independently locked
+ChromBPNet runtime under `runtimes/chrombpnet`. Installing it registers `CHROMBPNET` through Altar's standard
+model-plugin entry point; the base `altar` distribution does not contain ChromBPNet-specific science.
+
+```bash
+pip install altar-chrombpnet
+```
+
+```python
+from altar.models import get_model_plugin
+from altar_chrombpnet import ChromBPNetConfiguration
+
+plugin = get_model_plugin("CHROMBPNET")
+```
+
+The binding owns ChromBPNet configuration, score schemas, prioritization, preprocessing, scoring, and
+interpretation plans. It depends only on Altar's public façades and Pydantic. TensorFlow, model code, weights,
+reference genomes, peaks, and provider SDKs remain outside the binding.
+
+The binding also owns ChromBPNet's logical paths (`altar_chrombpnet.layouts`): the derived per-fold peak
+distributions and interval index, the staged peaks and motif set, and the interpretation run tree. Altar core
+provides only the model-neutral `ModelArtifactLayout` and `ScoringLayout`; every model-owned ChromBPNet file is
+composed from `model_resource_file`, so preparation and scoring name the same files under one layout. Staged
+file names come from the declared `weights_format` and `peaks_compression`, never from a resource URI.
+
+Preprocessing can emit a manifest that records the ordered weights, peaks, genome, strand policy, producer
+image, and output digests. That provenance is additive: the normal scoring plan consumes the conventional
+peak-distribution and safe interval-index paths directly and does not require a manifest, an Altar-owned
+directory, or an adoption step. Standalone runtime callers may supply both `--preprocessing-manifest` and
+`--preprocessing-identity` to opt into strict verification. Historical scores and externally managed
+artifacts are not rewritten or recomputed.
+
+Existing callers migrating from Altar core must install this package and replace imports such as
+`from altar.models import ChromBPNetPlugin` with `from altar_chrombpnet import ChromBPNetPlugin`. Registry-based
+callers need only install the package; the `CHROMBPNET` key is unchanged.

altar_chrombpnet-0.1.0/altar_chrombpnet/__init__.py

@@ -0,0 +1,65 @@
+# Copyright 2026 Riya Sinha
+"""ChromBPNet model binding for Altar."""
+
+from altar_chrombpnet.layouts import (
+    MOTIF_SET_FILENAME,
+    PEAK_INTERVAL_INDEX_FILENAME,
+    PREPROCESSING_MANIFEST_FILENAME,
+    ChromBPNetInterpretationLayout,
+    PeaksCompression,
+    WeightsFormat,
+    fold_weights_file,
+    model_dir,
+    motif_set_file,
+    peak_distribution_file,
+    peak_interval_index_file,
+    peaks_file,
+    preprocessing_manifest_file,
+)
+from altar_chrombpnet.plugin import (
+    CHROMBPNET_DEFAULT_IMAGE,
+    FINEMO_IMAGE,
+    KIND_INTERPRETATION_AVERAGE,
+    KIND_INTERPRETATION_FOLDS,
+    KIND_INTERPRETATION_MOTIF,
+    KIND_INTERPRETATION_PLOT,
+    KIND_MODEL_PREPROCESSING,
+    KIND_SCORING_FOLDS,
+    KIND_SCORING_SUMMARIZATION,
+    NUM_FOLDS,
+    ChromBPNetConfiguration,
+    ChromBPNetPlugin,
+    ChromBPNetRunInputs,
+    ShardKind,
+)
+
+
+__all__ = [
+    "CHROMBPNET_DEFAULT_IMAGE",
+    "FINEMO_IMAGE",
+    "KIND_INTERPRETATION_AVERAGE",
+    "KIND_INTERPRETATION_FOLDS",
+    "KIND_INTERPRETATION_MOTIF",
+    "KIND_INTERPRETATION_PLOT",
+    "KIND_MODEL_PREPROCESSING",
+    "KIND_SCORING_FOLDS",
+    "KIND_SCORING_SUMMARIZATION",
+    "MOTIF_SET_FILENAME",
+    "NUM_FOLDS",
+    "PEAK_INTERVAL_INDEX_FILENAME",
+    "PREPROCESSING_MANIFEST_FILENAME",
+    "ChromBPNetConfiguration",
+    "ChromBPNetInterpretationLayout",
+    "ChromBPNetPlugin",
+    "ChromBPNetRunInputs",
+    "PeaksCompression",
+    "ShardKind",
+    "WeightsFormat",
+    "fold_weights_file",
+    "model_dir",
+    "motif_set_file",
+    "peak_distribution_file",
+    "peak_interval_index_file",
+    "peaks_file",
+    "preprocessing_manifest_file",
+]

altar_chrombpnet-0.1.0/altar_chrombpnet/layouts.py

@@ -0,0 +1,108 @@
+# Copyright 2026 Riya Sinha
+"""ChromBPNet-owned logical paths for preparation, scoring, and interpretation.
+
+Altar core supplies only model-neutral paths (`ModelArtifactLayout` and `ScoringLayout`): the staged genome,
+model-owned resource files, variant inputs, and result tables. The files below are ChromBPNet science: the
+per-fold peak-activity distributions and peak interval index that preparation derives, the matched peak set,
+the TF-MoDISco motif set used for hit calling, and the interpretation run tree. Each model-owned path is built
+through the caller's `ModelArtifactLayout.model_resource_file`, so a scoring plan names exactly the files that
+the preparation plan wrote under the same layout.
+
+Staged file names come from declared configuration (`weights_format`, `peaks_compression`), never from a
+resource URI: a content-addressed URI need not carry a file extension.
+"""
+
+from __future__ import annotations
+import posixpath
+from dataclasses import dataclass
+from typing import TYPE_CHECKING, Literal
+
+from altar.models import ModelArtifactLayout
+
+
+if TYPE_CHECKING:
+    from collections.abc import Mapping
+
+
+WeightsFormat = Literal["h5"]
+"""Serialized fold-weight formats the ChromBPNet runtime loads. Keras loads a single-file model by suffix."""
+
+PeaksCompression = Literal["gzip", "none"]
+"""Compression of the narrowPeak-schema peak set. The runtime infers decompression from the staged suffix."""
+
+PEAK_INTERVAL_INDEX_FILENAME = "peaks.intervals.jsonl"
+MOTIF_SET_FILENAME = "motifs.h5"
+PREPROCESSING_MANIFEST_FILENAME = "preprocessing.manifest.json"
+
+_WEIGHTS_EXTENSIONS: Mapping[str, str] = {"h5": ".h5"}
+_PEAKS_FILENAMES: Mapping[str, str] = {"gzip": "peaks.bed.gz", "none": "peaks.bed"}
+
+
+def fold_weights_file(layout: ModelArtifactLayout, model_id: str, fold: int, weights_format: WeightsFormat) -> str:
+    """Return one fold's staged weights, `models/{model_id}/fold_{fold}_model.h5` for `weights_format="h5"`."""
+    return layout.fold_model_file(model_id, fold, _WEIGHTS_EXTENSIONS[weights_format])
+
+
+def peaks_file(layout: ModelArtifactLayout, model_id: str, compression: PeaksCompression) -> str:
+    """Return the staged raw peak set, `models/{model_id}/peaks.bed.gz` or `.../peaks.bed`."""
+    return layout.model_resource_file(model_id, _PEAKS_FILENAMES[compression])
+
+
+def peak_distribution_file(layout: ModelArtifactLayout, model_id: str, fold: int) -> str:
+    """Return one fold's derived peak-activity distribution, `models/{model_id}/fold_{fold}_peak_distribution.npy`."""
+    return layout.model_resource_file(model_id, f"fold_{fold}_peak_distribution.npy")
+
+
+def peak_interval_index_file(layout: ModelArtifactLayout, model_id: str) -> str:
+    """Return the safe serialized peak interval index used to rebuild the in-memory peak tree."""
+    return layout.model_resource_file(model_id, PEAK_INTERVAL_INDEX_FILENAME)
+
+
+def preprocessing_manifest_file(layout: ModelArtifactLayout, model_id: str) -> str:
+    """Return the optional preparation provenance manifest written beside the derived artifacts."""
+    return layout.model_resource_file(model_id, PREPROCESSING_MANIFEST_FILENAME)
+
+
+def motif_set_file(layout: ModelArtifactLayout, model_id: str) -> str:
+    """Return the staged TF-MoDISco motif set, `models/{model_id}/motifs.h5`.
+
+    The motif set is a model-owned configured resource, so it lives beside that model's weights rather than at
+    a shared storage-root path that two models could overwrite.
+    """
+    return layout.model_resource_file(model_id, MOTIF_SET_FILENAME)
+
+
+def model_dir(layout: ModelArtifactLayout, model_id: str) -> str:
+    """Return the directory that holds model-owned files: preparation's `-o` output directory."""
+    return posixpath.dirname(peak_interval_index_file(layout, model_id))
+
+
+@dataclass(frozen=True)
+class ChromBPNetInterpretationLayout(ModelArtifactLayout):
+    """Paths for a ChromBPNet interpretation run, one method per file or directory the stage commands name.
+
+    Inherits the shared genome and model-file paths from `ModelArtifactLayout`. Several outputs are directories
+    rather than single files: a fold's contribution scores, the averaged run directory, and the per-allele hit
+    directories. The model runtime and finemo write a tree of files into each, so those stages do not declare
+    file `Transfer`s the way scoring does.
+    """
+
+    def run_dir(self, run_id: str, model_id: str) -> str:
+        return posixpath.join("interpretations", run_id, model_id)
+
+    def model_variants_file(self, run_id: str, model_id: str) -> str:
+        return posixpath.join(self.run_dir(run_id, model_id), "variants.tsv")
+
+    def fold_results_dir(self, run_id: str, model_id: str, fold: int) -> str:
+        return posixpath.join(self.run_dir(run_id, model_id), f"fold_{fold}")
+
+    def variants_with_plots_file(self, run_id: str, model_id: str) -> str:
+        return posixpath.join(self.run_dir(run_id, model_id), "variants_w_interpret_plots.tsv")
+
+    def hitcaller_region_file(self, run_id: str, model_id: str, allele: str) -> str:
+        """Return the per-allele finemo input the average step writes, `…/{allele}_finemo_input.npz`."""
+        return posixpath.join(self.run_dir(run_id, model_id), f"{allele}_finemo_input.npz")
+
+    def hits_dir(self, run_id: str, model_id: str, allele: str) -> str:
+        """Return the per-allele finemo call-hits output directory, `…/ref_hits` or `…/alt_hits`."""
+        return posixpath.join(self.run_dir(run_id, model_id), f"{allele}_hits")

altar_chrombpnet-0.1.0/altar_chrombpnet/local_scoring.py

@@ -0,0 +1,194 @@
+# Copyright 2026 Riya Sinha
+"""Helpers for running a ChromBPNet scoring plan on your own machine.
+
+`examples/scoring/run.py` and the local run tests both use these, so the logic for turning a plan into
+container submissions — build the plan, submit each shard, fan in the folds, collect the output — lives in
+one place.
+
+The helpers need no cloud infrastructure. They combine the reference `ChromBPNetPlugin`, any
+`ExecutionBackend`, and the reference reconcile loop. A caller supplies a backend and a `prep` function that
+maps each planned spec to the spec actually submitted: the identity function for a real run, or `stubify`
+for a run with no container image and no GPU.
+"""
+
+from __future__ import annotations
+import asyncio
+import dataclasses
+import os
+from collections.abc import Callable
+from pathlib import Path
+
+from altar.execution import (
+    ContainerTaskSpec,
+    ExecutionBackend,
+    GroupSpec,
+    InMemoryTaskLedger,
+    PathResolver,
+    ReconcileResult,
+    TaskHandle,
+    all_folds,
+    group_by,
+    run_reconcile_loop,
+    single,
+)
+from altar.models import (
+    ContainerScoringPlan,
+    ResourceReference,
+    ScoringLayout,
+    ScoringRequest,
+)
+from altar.testing import make_stub_sources, source_transfers, stubify
+
+from altar_chrombpnet.plugin import (
+    KIND_SCORING_FOLDS,
+    KIND_SCORING_SUMMARIZATION,
+    ChromBPNetConfiguration,
+    ChromBPNetPlugin,
+    ChromBPNetRunInputs,
+)
+
+
+# Real runs must provide an immutable published digest.
+CHROMBPNET_RUNTIME_IMAGE_ENV = "ALTAR_CHROMBPNET_IMAGE"
+_TEST_CHROMBPNET_RUNTIME_IMAGE = "test/chrombpnet@sha256:" + "a" * 64
+
+# Prep maps a planned spec to the spec to submit: identity for a real run, `stubify` for a stub run.
+Prep = Callable[[ContainerTaskSpec], ContainerTaskSpec]
+
+__all__ = [
+    "CHROMBPNET_RUNTIME_IMAGE_ENV",
+    "Prep",
+    "build_scoring_plan",
+    "drive_scoring_run",
+    "make_stub_sources",
+    "remap_chrombpnet_runtime_image",
+    "resolve_chrombpnet_runtime_image",
+    "stubify",
+]
+
+
+def resolve_chrombpnet_runtime_image() -> str:
+    """Return the configured ChromBPNet runtime image."""
+    image = os.environ.get(CHROMBPNET_RUNTIME_IMAGE_ENV)
+    if image is None:
+        msg = f"set {CHROMBPNET_RUNTIME_IMAGE_ENV} to an immutable ChromBPNet image digest"
+        raise ValueError(msg)
+    return image
+
+
+def remap_chrombpnet_runtime_image(spec: ContainerTaskSpec, image: str | None) -> ContainerTaskSpec:
+    """Replace the plan's ChromBPNet image while preserving other stage images."""
+    if image and spec.kind in {KIND_SCORING_FOLDS, KIND_SCORING_SUMMARIZATION}:
+        return dataclasses.replace(spec, image=image)
+    return spec
+
+
+def build_scoring_plan(  # noqa: PLR0913
+    resolver: PathResolver,
+    *,
+    job_id: str = "example-job",
+    model_id: str = "example-model",
+    genome: str = "hg38",
+    accession_pattern: str = "chrombpnet.fold_{fold}.h5",
+    image: str = _TEST_CHROMBPNET_RUNTIME_IMAGE,
+) -> ContainerScoringPlan:
+    """Build a five-fold ChromBPNet plan plus its summarize shard against `resolver`'s mount."""
+    plugin = ChromBPNetPlugin()
+    configuration = ChromBPNetConfiguration(
+        weights=tuple(
+            ResourceReference(
+                uri=accession_pattern.format(fold=fold),
+                digest=f"sha256:{fold + 1:064x}",
+            )
+            for fold in range(5)
+        ),
+        peaks=ResourceReference(uri="peaks.bed.gz", digest=f"sha256:{15:064x}"),
+        image=image,
+    )
+    return plugin.build_scoring_plan(
+        ScoringRequest(
+            job_id=job_id,
+            model_id=model_id,
+            configuration=configuration,
+            run_inputs=ChromBPNetRunInputs(
+                genome_build=genome,
+                genome=ResourceReference(
+                    uri=ScoringLayout().genome_fasta(genome),
+                    digest=f"sha256:{'f' * 64}",
+                ),
+            ),
+            output=plugin.default_output,
+            resolver=resolver,
+            layout=ScoringLayout(),
+        )
+    )
+
+
+async def drive_scoring_run(  # noqa: PLR0913
+    backend: ExecutionBackend,
+    plan: ContainerScoringPlan,
+    *,
+    prep: Prep,
+    source_dir: str,
+    results_dir: str,
+    interval_s: float = 1.0,
+    max_ticks: int = 180,
+    max_in_flight: int | None = None,
+) -> tuple[str, list[ReconcileResult]]:
+    """Run `plan` to completion on `backend` and return `(final_result_host_path, tick_results)`.
+
+    It stages and submits each shard, runs the label-based fan-in that waits for all five folds before
+    submitting summarize, and collects the summarize output. `examples/scoring/run.py` runs this same drive
+    with progress printing added.
+
+    `prep` maps each planned spec to the spec actually submitted: the identity function for a real run, or
+    `stubify` for a stub run.
+
+    `max_in_flight` caps how many fold shards run at once. Without a cap, every shard is submitted as soon as
+    it is staged. On a host with a single GPU that oversubscribes the card: each ChromBPNet fold container
+    claims most of the GPU, so five running at once fail with `CUDA_ERROR_OUT_OF_MEMORY`. Deciding how much
+    compute to use is the caller's job, not the core library's, so this knob lives on the driver. Pass `1` on a
+    one-GPU host; leave it `None` where the backend autoscales, such as Modal or a Kubernetes cluster.
+    """
+    num_folds = ChromBPNetPlugin().num_folds
+    ledger = InMemoryTaskLedger()
+    summarize = plan.summarize
+    if summarize is None:
+        msg = "scoring plan has no summarize shard to fan into"
+        raise ValueError(msg)
+
+    in_flight: list[TaskHandle] = []
+
+    async def _await_slot() -> None:
+        """Wait until fewer than `max_in_flight` submitted folds are still running."""
+        if not max_in_flight:
+            return
+        while len(in_flight) >= max_in_flight:
+            statuses = await backend.poll(in_flight)
+            done = {s.handle.external_id for s in statuses if s.state.is_terminal}
+            if done:
+                in_flight[:] = [h for h in in_flight if h.external_id not in done]
+            else:
+                await asyncio.sleep(interval_s)
+
+    for shard in plan.shards:
+        await _await_slot()
+        await backend.stage_inputs(source_transfers(shard.inputs, source_dir))
+        handle = await backend.submit(prep(shard))
+        await ledger.record(handle, kind=shard.kind)
+        in_flight.append(handle)
+
+    async def on_folds_ready(_group_key: str, _handles: list[TaskHandle]) -> None:
+        handle = await backend.submit(prep(summarize))
+        await ledger.record(handle, kind=summarize.kind)
+
+    async def on_summarize_ready(_group_key: str, _handles: list[TaskHandle]) -> None:
+        await backend.collect_outputs(source_transfers(summarize.outputs, results_dir))
+
+    specs = [
+        GroupSpec(KIND_SCORING_FOLDS, group_by("job_id", "model_id"), all_folds(num_folds), on_folds_ready),
+        GroupSpec(KIND_SCORING_SUMMARIZATION, group_by("job_id", "model_id"), single(), on_summarize_ready),
+    ]
+    results = await run_reconcile_loop(backend, ledger, specs, interval_s=interval_s, max_ticks=max_ticks)
+    final = Path(results_dir) / summarize.outputs[0].logical_path
+    return str(final), results