zipoop 0.1.0__tar.gz → 0.2.1__tar.gz

zipoop-0.2.1/PKG-INFO

@@ -0,0 +1,68 @@
+Metadata-Version: 2.4
+Name: zipoop
+Version: 0.2.1
+Summary: Decoder and reader for Zipoop teacher shards (per-window neural compression of ENCODE signal tracks), v3 and v4 archives
+Author: zatumout
+License-Expression: MIT
+Project-URL: Dataset, https://huggingface.co/datasets/zatumout/zipoop_32k1885_010_v4
+Project-URL: DatasetV3, https://huggingface.co/datasets/zatumout/zipoop_32k1885_010
+Keywords: genomics,ENCODE,neural compression,distillation
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.26
+Requires-Dist: torch>=2.2
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == "test"
+Dynamic: license-file
+
+# zipoop
+
+Decoder and reader for the Zipoop teacher shards: 89,438 windows of 32 kb, 1,885 ENCODE tracks (GRCh38), stored as one `<chrom>.npz` per chromosome plus `MANIFEST.tsv`.
+
+Two equivalent releases are published:
+
+- [zatumout/zipoop_32k1885_010_v4](https://huggingface.co/datasets/zatumout/zipoop_32k1885_010_v4) — `pooper-dna-v4`, 89.3 GiB, **requires `zipoop>=0.2`**;
+- [zatumout/zipoop_32k1885_010](https://huggingface.co/datasets/zatumout/zipoop_32k1885_010) — the older `pooper-dna-v3` encoding, 128.4 GiB, read by 0.1.x as well.
+
+Both decode to the same signals; v4 only changes how the arrays are stored (byte-shuffled members, LZMA, and a float16 sparse FFT with one float32 scale per track).
+
+## Data layout
+
+`<root>/<chrom>.npz` (25 files) plus `MANIFEST.tsv` (chrom, windows, members, bytes, sha256). Each file is a zip with members `<window>/<key>.npy` (161 arrays per window in v3, 164 in v4), windows in genome order. Window names are `<chrom>_<start>_<end>`.
+
+## Usage
+
+```python
+from zipoop import Dataset
+with Dataset("/path/to/weights_npz") as ds:
+    w = ds.load_window(ds.windows("chr21")[0])   # TeacherWindow: key, dna (L,), signal (L, T), valid (L, T)
+```
+
+```
+zipoop verify /path/to/weights_npz      # size + sha256 against MANIFEST.tsv
+zipoop show   /path/to/weights_npz chr21_10027008_10059776
+```
+
+Lower level: `Dataset.read(window)` gives the parsed `Shard`, `Decoder(shard).decode(start, end, tracks)` decodes any slice or track subset. Evaluation helpers (`window_pearson`, `summarize`, `calibrate`, `restore_real`) are in `zipoop.evaluation`.
+
+`zipoop.evaluation.subwindow_metrics(pred, target, valid, subwindow=32768)` is the evaluator frozen in the D5 pipeline: it splits the length axis into contiguous sub-windows (4 x 32,768 bp inside a 131,072 bp segment) and returns `pearson` / `proj` / `nmse` / `total_err` / `eligible` arrays of shape (sub-windows, tracks), with the same clamps and the same eligibility rule (at least 2 observed positions and a non-constant target). The aggregation used by the frozen pipeline is the mean Pearson over eligible pairs and the median `proj` / `nmse`.
+
+Rules baked into `load_window` (see `zipoop/teacher.py`): RNA-seq tracks are 0 and valid where unrecorded; other tracks stay missing there. Opening a chromosome file parses its whole zip directory (chr1 about 3.6 s and 0.9 GB), so keep the `Dataset` open; forked DataLoader workers reopen their own handles.
+
+## Supported formats
+
+| archive | `format` | `hybrid_format` | storage |
+|---|---|---|---|
+| v3 | `pooper-dna-v3` | `fft-plus-dna-residual-v1` | deflate, `fft_coefficients` complex64 |
+| v4 | `pooper-dna-v4` | `fft-plus-dna-residual-v2` | LZMA, byte-shuffled floats, `fft_packed` float16 + `fft_scale` float32 |
+
+`read_shard` accepts both and rejects anything else with a clear error. The v4 decode path is verified against v3: parameters, masks and calibrations are bit-identical, the FFT payload differs only by float16 rounding (max relative error about 4.7e-4), and decoded windows reach a per-track Pearson of at least 0.9999997 against the v3 decode.
+
+Tests: `ZIPOOP_DATA=<weights_npz> pytest tests` (the pooper-equivalence test also needs the original shards and the pooper package). The v4 storage encoding has dataset-free unit tests in `tests/test_v4_format.py`.
+
+## License and data terms
+
+The code is MIT-licensed (see `LICENSE`; keep the copyright notice "zatumout"). The shard dataset is released separately under CC BY 4.0 (credit: zatumout), commercial use and model distillation allowed. The shards and `zipoop/data/encode-panel_1885.json` are derived from public ENCODE tracks: when you use them, also cite and acknowledge the ENCODE Consortium as described at https://www.encodeproject.org/help/citing-encode.

zipoop-0.2.1/README.md

@@ -0,0 +1,48 @@
+# zipoop
+
+Decoder and reader for the Zipoop teacher shards: 89,438 windows of 32 kb, 1,885 ENCODE tracks (GRCh38), stored as one `<chrom>.npz` per chromosome plus `MANIFEST.tsv`.
+
+Two equivalent releases are published:
+
+- [zatumout/zipoop_32k1885_010_v4](https://huggingface.co/datasets/zatumout/zipoop_32k1885_010_v4) — `pooper-dna-v4`, 89.3 GiB, **requires `zipoop>=0.2`**;
+- [zatumout/zipoop_32k1885_010](https://huggingface.co/datasets/zatumout/zipoop_32k1885_010) — the older `pooper-dna-v3` encoding, 128.4 GiB, read by 0.1.x as well.
+
+Both decode to the same signals; v4 only changes how the arrays are stored (byte-shuffled members, LZMA, and a float16 sparse FFT with one float32 scale per track).
+
+## Data layout
+
+`<root>/<chrom>.npz` (25 files) plus `MANIFEST.tsv` (chrom, windows, members, bytes, sha256). Each file is a zip with members `<window>/<key>.npy` (161 arrays per window in v3, 164 in v4), windows in genome order. Window names are `<chrom>_<start>_<end>`.
+
+## Usage
+
+```python
+from zipoop import Dataset
+with Dataset("/path/to/weights_npz") as ds:
+    w = ds.load_window(ds.windows("chr21")[0])   # TeacherWindow: key, dna (L,), signal (L, T), valid (L, T)
+```
+
+```
+zipoop verify /path/to/weights_npz      # size + sha256 against MANIFEST.tsv
+zipoop show   /path/to/weights_npz chr21_10027008_10059776
+```
+
+Lower level: `Dataset.read(window)` gives the parsed `Shard`, `Decoder(shard).decode(start, end, tracks)` decodes any slice or track subset. Evaluation helpers (`window_pearson`, `summarize`, `calibrate`, `restore_real`) are in `zipoop.evaluation`.
+
+`zipoop.evaluation.subwindow_metrics(pred, target, valid, subwindow=32768)` is the evaluator frozen in the D5 pipeline: it splits the length axis into contiguous sub-windows (4 x 32,768 bp inside a 131,072 bp segment) and returns `pearson` / `proj` / `nmse` / `total_err` / `eligible` arrays of shape (sub-windows, tracks), with the same clamps and the same eligibility rule (at least 2 observed positions and a non-constant target). The aggregation used by the frozen pipeline is the mean Pearson over eligible pairs and the median `proj` / `nmse`.
+
+Rules baked into `load_window` (see `zipoop/teacher.py`): RNA-seq tracks are 0 and valid where unrecorded; other tracks stay missing there. Opening a chromosome file parses its whole zip directory (chr1 about 3.6 s and 0.9 GB), so keep the `Dataset` open; forked DataLoader workers reopen their own handles.
+
+## Supported formats
+
+| archive | `format` | `hybrid_format` | storage |
+|---|---|---|---|
+| v3 | `pooper-dna-v3` | `fft-plus-dna-residual-v1` | deflate, `fft_coefficients` complex64 |
+| v4 | `pooper-dna-v4` | `fft-plus-dna-residual-v2` | LZMA, byte-shuffled floats, `fft_packed` float16 + `fft_scale` float32 |
+
+`read_shard` accepts both and rejects anything else with a clear error. The v4 decode path is verified against v3: parameters, masks and calibrations are bit-identical, the FFT payload differs only by float16 rounding (max relative error about 4.7e-4), and decoded windows reach a per-track Pearson of at least 0.9999997 against the v3 decode.
+
+Tests: `ZIPOOP_DATA=<weights_npz> pytest tests` (the pooper-equivalence test also needs the original shards and the pooper package). The v4 storage encoding has dataset-free unit tests in `tests/test_v4_format.py`.
+
+## License and data terms
+
+The code is MIT-licensed (see `LICENSE`; keep the copyright notice "zatumout"). The shard dataset is released separately under CC BY 4.0 (credit: zatumout), commercial use and model distillation allowed. The shards and `zipoop/data/encode-panel_1885.json` are derived from public ENCODE tracks: when you use them, also cite and acknowledge the ENCODE Consortium as described at https://www.encodeproject.org/help/citing-encode.

{zipoop-0.1.0 → zipoop-0.2.1}/pyproject.toml

@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "zipoop"
-version = "0.1.0"
-description = "Decoder and reader for Zipoop teacher shards (per-window neural compression of ENCODE signal tracks)"
+version = "0.2.1"
+description = "Decoder and reader for Zipoop teacher shards (per-window neural compression of ENCODE signal tracks), v3 and v4 archives"
 readme = "README.md"
 requires-python = ">=3.11"
 license = "MIT"
@@ -19,7 +19,8 @@ dependencies = ["numpy>=1.26", "torch>=2.2"]
 test = ["pytest>=8"]
 
 [project.urls]
-Dataset = "https://huggingface.co/datasets/zatumout/zipoop-teacher-shards"
+Dataset = "https://huggingface.co/datasets/zatumout/zipoop_32k1885_010_v4"
+DatasetV3 = "https://huggingface.co/datasets/zatumout/zipoop_32k1885_010"
 
 [project.scripts]
 zipoop = "zipoop.cli:main"

zipoop-0.2.1/tests/test_eval_subwindow.py

@@ -0,0 +1,67 @@
+"""Parity tests for zipoop.evaluation.subwindow_metrics against the frozen D5 evaluator."""
+import numpy as np
+import torch
+
+from zipoop.evaluation import subwindow_metrics
+
+
+def frozen_subwindow_metrics(p, y, m, N, W):
+    """Verbatim copy of the evaluator frozen in zipoop_32k1885_010_128k2rd_D5_frozen_20261017/d5_genome.py."""
+    out = {k: [] for k in ("pearson", "proj", "nmse", "total_err", "eligible")}
+    for k in range(N):
+        s = slice(k * W, (k + 1) * W)
+        mf = m[s].float(); n = mf.sum(0).clamp_min(1)
+        yc = (y[s] - (y[s] * mf).sum(0) / n) * mf; pc = (p[s] - (p[s] * mf).sum(0) / n) * mf
+        vy = yc.square().sum(0); yy = (y[s].square() * mf).sum(0)
+        out["pearson"].append((pc * yc).sum(0) / (pc.square().sum(0) * vy).clamp_min(1e-24).sqrt())
+        out["proj"].append((p[s] * y[s] * mf).sum(0) / yy.clamp_min(1e-30))
+        out["nmse"].append(((p[s] - y[s]).square() * mf).sum(0) / yy.clamp_min(1e-30))
+        out["total_err"].append(((p[s] * mf).sum(0) - (y[s] * mf).sum(0)).abs() / (y[s] * mf).sum(0).abs().clamp_min(1e-30))
+        out["eligible"].append((n >= 2) & (vy > 1e-12))
+    return {k: torch.stack(v).cpu().numpy() for k, v in out.items()}
+
+
+def test_matches_frozen_evaluator():
+    torch.manual_seed(0)
+    L, T, W, N = 4 * 256, 7, 256, 4
+    target = torch.randn(L, T)
+    pred = target * 0.9 + torch.randn(L, T) * 0.3
+    valid = torch.rand(L, T) > 0.2
+    valid[:W] = False                     # first sub-window has no observations
+    target[:, 3] = 1.0                    # a constant track must be ineligible
+
+    want = frozen_subwindow_metrics(pred, target, valid, N, W)
+    got = subwindow_metrics(pred, target, valid, subwindow=W)
+
+    assert set(want) == set(got)
+    for k in want:
+        assert got[k].shape == (N, T), (k, got[k].shape)
+        assert np.array_equal(got[k], want[k]) if got[k].dtype == bool else np.allclose(got[k], want[k], rtol=0, atol=0), k
+    assert not got["eligible"][0].any()
+    assert not got["eligible"][:, 3].any()
+
+
+def test_identity_prediction():
+    torch.manual_seed(1)
+    L, T, W = 2 * 128, 5, 128
+    target = torch.randn(L, T)
+    valid = torch.ones(L, T, dtype=torch.bool)
+    m = subwindow_metrics(target.clone(), target, valid, subwindow=W)
+    assert np.allclose(m["pearson"], 1.0)
+    assert np.allclose(m["proj"], 1.0)
+    assert np.allclose(m["nmse"], 0.0)
+    assert np.allclose(m["total_err"], 0.0)
+    assert m["eligible"].all()
+
+
+def test_accepts_numpy_and_validates_shape():
+    t = np.random.RandomState(0).randn(2 * 64, 3).astype(np.float32)
+    v = np.ones_like(t, dtype=bool)
+    m = subwindow_metrics(t, t, v, subwindow=64)
+    assert m["pearson"].shape == (2, 3)
+    try:
+        subwindow_metrics(t, t[:100], v, subwindow=64)
+    except ValueError:
+        pass
+    else:
+        raise AssertionError("shape mismatch must raise")

zipoop-0.2.1/tests/test_v4_format.py

@@ -0,0 +1,60 @@
+"""Unit tests for the v4 storage encoding (no dataset required)."""
+import numpy as np
+
+from zipoop.shard import HYBRID_FORMAT_V1, HYBRID_FORMAT_V2, _Reader
+
+
+def shuffle(a):
+    """Mirror of the writer: float/complex arrays -> (itemsize, n) uint8 columns."""
+    v = np.asarray(a)
+    return np.ascontiguousarray(v).view(np.uint8).reshape(-1, v.dtype.itemsize).T.copy()
+
+
+def spec_of(**arrays):
+    return {k: {"dtype": np.asarray(v).dtype.str, "shape": list(np.asarray(v).shape)} for k, v in arrays.items()}
+
+
+def test_byte_shuffle_roundtrip():
+    rng = np.random.default_rng(0)
+    a = rng.standard_normal((7, 5)).astype(np.float32)
+    view = {"gain": shuffle(a)}
+    out = _Reader(view, spec_of(gain=a), HYBRID_FORMAT_V1)["gain"]
+    assert out.dtype == np.float32
+    assert np.array_equal(out, a)
+
+
+def test_unshuffled_members_pass_through():
+    a = np.arange(12, dtype=np.uint16)
+    assert np.array_equal(_Reader({"fft_indices": a}, {}, HYBRID_FORMAT_V1)["fft_indices"], a)
+
+
+def test_fft_v2_dequantization():
+    rng = np.random.default_rng(1)
+    tracks, bins = 4, 6
+    coefficients = (rng.standard_normal((tracks, bins)) + 1j * rng.standard_normal((tracks, bins))).astype(np.complex64)
+    pairs = np.stack((coefficients.real, coefficients.imag), axis=-1)
+    scale = np.abs(pairs).max(axis=(1, 2), keepdims=True).astype(np.float32)
+    packed = (pairs / scale).astype(np.float16)
+    scale = scale.reshape(tracks, 1, 1)
+    indices = np.sort(rng.choice(64, size=bins, replace=False)).astype(np.uint16)[None, :].repeat(tracks, 0)
+
+    view = {"fft_indices": indices, "fft_packed": shuffle(packed), "fft_scale": shuffle(scale)}
+    reader = _Reader(view, spec_of(fft_packed=packed, fft_scale=scale), HYBRID_FORMAT_V2)
+    got_i, got_c = reader.fft()
+    assert np.array_equal(got_i, indices)
+    assert got_c.dtype == np.complex64
+    assert got_c.shape == (tracks, bins)
+    reference = packed.astype(np.float32) * scale           # the same float32 reference the writer used
+    reference = reference[..., 0] + 1j * reference[..., 1]
+    assert np.allclose(got_c, reference, rtol=0, atol=1e-6)
+    # and it stays within float16 rounding of the original complex64 payload
+    assert np.max(np.abs(got_c - coefficients)) < 0.02 * np.max(np.abs(coefficients))
+
+
+def test_v3_reader_uses_complex_payload():
+    tracks, bins = 2, 3
+    coefficients = (np.ones((tracks, bins)) * (1 + 2j)).astype(np.complex64)
+    indices = np.zeros((tracks, bins), dtype=np.uint16)
+    got_i, got_c = _Reader({"fft_indices": indices, "fft_coefficients": coefficients}, {}, HYBRID_FORMAT_V1).fft()
+    assert np.array_equal(got_i, indices)
+    assert np.array_equal(got_c, coefficients)

{zipoop-0.1.0 → zipoop-0.2.1}/zipoop/__init__.py

@@ -1,11 +1,11 @@
 """Zipoop: decoder and reader for the per-window teacher shards (merged per-chromosome npz)."""
 from .dataset import Dataset, verify
 from .decoder import Decoder
-from .evaluation import calibrate, restore_real, summarize, window_pearson
+from .evaluation import calibrate, restore_real, subwindow_metrics, summarize, window_pearson
 from .panel import panel_tracks, zero_tracks
 from .shard import Shard, read_shard
 from .teacher import TeacherWindow, load_window
 
-__version__ = "0.1.0"
+__version__ = "0.2.1"
 __all__ = ["Dataset", "verify", "Decoder", "Shard", "read_shard", "TeacherWindow", "load_window",
-           "panel_tracks", "zero_tracks", "calibrate", "restore_real", "summarize", "window_pearson"]
+           "panel_tracks", "zero_tracks", "calibrate", "restore_real", "subwindow_metrics", "summarize", "window_pearson"]

{zipoop-0.1.0 → zipoop-0.2.1}/zipoop/evaluation.py

@@ -61,6 +61,59 @@ def window_pearson(pred, target, valid, window=WINDOW, chunk=256):
     return torch.cat(cols, 1)
 
 
+def subwindow_metrics(pred, target, valid, subwindow=WINDOW):
+    """Per (sub-window, track) metrics, the convention of the frozen D5 evaluator.
+
+    ``pred`` and ``target`` are (L, T) signals, ``valid`` the (L, T) observed-position mask; the length
+    axis is split into ``L // subwindow`` contiguous blocks (the D5 pipeline uses 4 x 32,768 bp inside a
+    131,072 bp segment). Returns a dict of numpy arrays with shape (sub-windows, tracks):
+
+    ``pearson``    masked 1 bp Pearson
+    ``proj``       projection ratio  sum(pred * target) / sum(target ** 2)
+    ``nmse``       sum((pred - target) ** 2) / sum(target ** 2)
+    ``total_err``  |sum(pred) - sum(target)| / |sum(target)|
+    ``eligible``   at least 2 observed positions and a non-constant target inside that sub-window
+
+    The percentages and clamps (1e-24, 1e-30) are the frozen evaluator's, unchanged.
+    """
+    if torch.is_tensor(pred):
+        pred = pred.detach()
+    else:
+        pred = torch.as_tensor(np.asarray(pred))
+    if torch.is_tensor(target):
+        target = target.detach()
+    else:
+        target = torch.as_tensor(np.asarray(target))
+    if torch.is_tensor(valid):
+        valid = valid.detach()
+    else:
+        valid = torch.as_tensor(np.asarray(valid))
+    if pred.shape != target.shape or pred.shape != valid.shape:
+        raise ValueError(f"shape mismatch: pred {tuple(pred.shape)}, target {tuple(target.shape)}, valid {tuple(valid.shape)}")
+    length, tracks = target.shape
+    if not 0 < subwindow <= length or length % subwindow:
+        raise ValueError(f"length {length} is not a multiple of subwindow {subwindow}")
+
+    pred = pred.float()
+    target = target.float()
+    out = {k: [] for k in ("pearson", "proj", "nmse", "total_err", "eligible")}
+    for k in range(length // subwindow):
+        s = slice(k * subwindow, (k + 1) * subwindow)
+        mf = valid[s].float()
+        n = mf.sum(0).clamp_min(1)
+        yc = (target[s] - (target[s] * mf).sum(0) / n) * mf
+        pc = (pred[s] - (pred[s] * mf).sum(0) / n) * mf
+        vy = yc.square().sum(0)
+        yy = (target[s].square() * mf).sum(0)
+        out["pearson"].append((pc * yc).sum(0) / (pc.square().sum(0) * vy).clamp_min(1e-24).sqrt())
+        out["proj"].append((pred[s] * target[s] * mf).sum(0) / yy.clamp_min(1e-30))
+        out["nmse"].append(((pred[s] - target[s]).square() * mf).sum(0) / yy.clamp_min(1e-30))
+        out["total_err"].append(((pred[s] * mf).sum(0) - (target[s] * mf).sum(0)).abs()
+                                / (target[s] * mf).sum(0).abs().clamp_min(1e-30))
+        out["eligible"].append((n >= 2) & (vy > 1e-12))
+    return {k: torch.stack(v).cpu().numpy() for k, v in out.items()}
+
+
 def summarize(p, tracks=None):
     """Mean/median over evaluable pairs, plus per-modality and per-window means."""
     tracks = panel_tracks() if tracks is None else tracks

zipoop-0.2.1/zipoop/shard.py

@@ -0,0 +1,158 @@
+"""Parse and validate one teacher shard (a ``pooper-dna-v3`` or ``pooper-dna-v4`` hybrid archive) into a typed record."""
+from __future__ import annotations
+
+import json
+import os
+from contextlib import contextmanager
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+from .masks import MASK_FORMAT, decode_mask
+from .network import ModelConfig
+
+SHARD_FORMAT_V3 = "pooper-dna-v3"
+SHARD_FORMAT_V4 = "pooper-dna-v4"
+SHARD_FORMATS = (SHARD_FORMAT_V3, SHARD_FORMAT_V4)
+HYBRID_FORMAT_V1 = "fft-plus-dna-residual-v1"
+HYBRID_FORMAT_V2 = "fft-plus-dna-residual-v2"
+HYBRID_FORMATS = (HYBRID_FORMAT_V1, HYBRID_FORMAT_V2)
+STORAGE_ENCODING_V4 = "byte-shuffle-lzma-v1"
+
+
+class ShardView:
+    """Read-only mapping over one window inside a merged per-chromosome npz (members "<window>/<key>")."""
+
+    def __init__(self, npz, key: str, members):
+        self._npz, self.key, self._members = npz, key, frozenset(members)
+
+    def __contains__(self, name):
+        return name in self._members
+
+    def __getitem__(self, name):
+        if name not in self._members:
+            raise KeyError(name)
+        return self._npz[f"{self.key}/{name}"]
+
+
+@dataclass
+class Shard:
+    key: str
+    shape: tuple[int, int]          # (length, tracks)
+    metadata: dict
+    config: ModelConfig
+    dna: np.ndarray                 # (L,) uint8
+    valid: np.ndarray               # (L, T) bool
+    parameters: dict[str, np.ndarray]
+    mean: np.ndarray                # per-track target scaling and calibration, (T,)
+    power: np.ndarray
+    gain: np.ndarray
+    offset: np.ndarray
+    fft_indices: np.ndarray         # (T, K) retained rfft bins
+    fft_coefficients: np.ndarray    # (T, K) complex64
+    final_gain: np.ndarray
+    final_offset: np.ndarray
+
+
+@contextmanager
+def _opened(source):
+    if isinstance(source, (str, os.PathLike)):
+        with np.load(source, allow_pickle=False) as z:
+            yield z, Path(source).stem
+    else:
+        yield source, source.key
+
+
+class _Reader:
+    """Member access that transparently undoes the v4 byte-shuffle and FFT quantization."""
+
+    def __init__(self, view, spec, hybrid_format):
+        self._view = view
+        self._spec = spec
+        self._hybrid = hybrid_format
+
+    def __contains__(self, name):
+        return name in self._view
+
+    def raw(self, name):
+        return self._view[name]
+
+    def __getitem__(self, name):
+        a = self._view[name]
+        meta = self._spec.get(name)
+        if meta is not None:
+            # v4 stores float/complex arrays as (itemsize, n) uint8 columns.
+            packed = np.ascontiguousarray(np.asarray(a))
+            if packed.dtype != np.uint8 or packed.ndim != 2:
+                raise ValueError(f"Invalid byte-shuffled member {name!r}")
+            b = np.ascontiguousarray(packed.T)
+            a = b.view(np.dtype(meta["dtype"])).reshape(tuple(meta["shape"]))
+        return a
+
+    def fft(self):
+        """(indices, complex64 coefficients) for either hybrid format."""
+        indices = np.asarray(self["fft_indices"])
+        if self._hybrid == HYBRID_FORMAT_V2:
+            packed = np.asarray(self["fft_packed"], dtype=np.float32)
+            scale = np.asarray(self["fft_scale"], dtype=np.float32)
+            if packed.ndim != 3 or packed.shape[-1] != 2 or packed.shape[:2] != indices.shape:
+                raise ValueError("Invalid packed FFT payload")
+            coefficients = (packed[..., 0] + 1j * packed[..., 1]) * scale.reshape(-1, 1)
+            return indices, np.ascontiguousarray(coefficients, dtype=np.complex64)
+        return indices, np.asarray(self["fft_coefficients"], dtype=np.complex64)
+
+
+def read_shard(source) -> Shard:
+    """``source``: a shard path, or a ShardView from :class:`zipoop.Dataset`."""
+    with _opened(source) as (z, key):
+        if "format" not in z or str(z["format"]) not in SHARD_FORMATS:
+            raise ValueError(f"Unsupported shard format: {z['format'] if 'format' in z else None}")
+        if "hybrid_format" not in z or str(z["hybrid_format"]) not in HYBRID_FORMATS:
+            raise ValueError("Unsupported hybrid archive")
+        if "valid_encoding" not in z or str(z["valid_encoding"]) != MASK_FORMAT:
+            raise ValueError("Unsupported mask encoding")
+        spec = {}
+        if "byte_shuffle" in z:
+            spec = json.loads(z["byte_shuffle"].tobytes())
+            if not isinstance(spec, dict):
+                raise ValueError("Invalid byte-shuffle index")
+            if "storage_encoding" in z and str(z["storage_encoding"]) != STORAGE_ENCODING_V4:
+                raise ValueError(f"Unsupported storage encoding: {str(z['storage_encoding'])}")
+        view = _Reader(z, spec, str(z["hybrid_format"]))
+        shape = tuple(int(n) for n in z["shape"])
+        if len(shape) != 2:
+            raise ValueError("Invalid shape")
+        length, tracks = shape
+        dna = view["dna"].copy()
+        if dna.shape != (length,) or np.any(dna > 4):
+            raise ValueError("Invalid DNA data in archive")
+        names = json.loads(view["parameter_names"].tobytes())
+        if len(names) != len(set(names)):
+            raise ValueError("Duplicate parameter names")
+        fft_indices, fft_coefficients = view.fft()
+        shard = Shard(
+            key=key, shape=shape, metadata=json.loads(view["metadata"].tobytes()),
+            config=ModelConfig.from_json(json.loads(view["model_config"].tobytes())),
+            dna=dna, valid=decode_mask(view["valid_v2"].tobytes(), shape),
+            parameters={name: view[f"parameter_{i}"].copy() for i, name in enumerate(names)},
+            fft_indices=fft_indices.copy(), fft_coefficients=fft_coefficients.copy(),
+            **{name: view[name].copy() for name in ("mean", "power", "gain", "offset",
+                                                    "final_gain", "final_offset")})
+    _validate(shard)
+    return shard
+
+
+def _validate(s: Shard) -> None:
+    length, tracks = s.shape
+    if any(a.shape != (tracks,) for a in (s.mean, s.power, s.gain, s.offset)):
+        raise ValueError("Invalid track scales")
+    i, c = s.fft_indices, s.fft_coefficients
+    if (i.ndim != 2 or i.shape[0] != tracks or c.shape != i.shape or not np.issubdtype(i.dtype, np.integer)
+            or (i < 0).any() or (i > length // 2).any() or not np.isfinite(c).all()):
+        raise ValueError("Invalid FFT payload")
+    for a in (s.final_gain, s.final_offset):
+        if a.shape != (tracks,) or not np.isfinite(a).all():
+            raise ValueError("Invalid final-signal calibration")
+    if (s.final_gain < 0).any():
+        raise ValueError("Invalid final-signal calibration")

zipoop-0.2.1/zipoop.egg-info/PKG-INFO

@@ -0,0 +1,68 @@
+Metadata-Version: 2.4
+Name: zipoop
+Version: 0.2.1
+Summary: Decoder and reader for Zipoop teacher shards (per-window neural compression of ENCODE signal tracks), v3 and v4 archives
+Author: zatumout
+License-Expression: MIT
+Project-URL: Dataset, https://huggingface.co/datasets/zatumout/zipoop_32k1885_010_v4
+Project-URL: DatasetV3, https://huggingface.co/datasets/zatumout/zipoop_32k1885_010
+Keywords: genomics,ENCODE,neural compression,distillation
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.26
+Requires-Dist: torch>=2.2
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == "test"
+Dynamic: license-file
+
+# zipoop
+
+Decoder and reader for the Zipoop teacher shards: 89,438 windows of 32 kb, 1,885 ENCODE tracks (GRCh38), stored as one `<chrom>.npz` per chromosome plus `MANIFEST.tsv`.
+
+Two equivalent releases are published:
+
+- [zatumout/zipoop_32k1885_010_v4](https://huggingface.co/datasets/zatumout/zipoop_32k1885_010_v4) — `pooper-dna-v4`, 89.3 GiB, **requires `zipoop>=0.2`**;
+- [zatumout/zipoop_32k1885_010](https://huggingface.co/datasets/zatumout/zipoop_32k1885_010) — the older `pooper-dna-v3` encoding, 128.4 GiB, read by 0.1.x as well.
+
+Both decode to the same signals; v4 only changes how the arrays are stored (byte-shuffled members, LZMA, and a float16 sparse FFT with one float32 scale per track).
+
+## Data layout
+
+`<root>/<chrom>.npz` (25 files) plus `MANIFEST.tsv` (chrom, windows, members, bytes, sha256). Each file is a zip with members `<window>/<key>.npy` (161 arrays per window in v3, 164 in v4), windows in genome order. Window names are `<chrom>_<start>_<end>`.
+
+## Usage
+
+```python
+from zipoop import Dataset
+with Dataset("/path/to/weights_npz") as ds:
+    w = ds.load_window(ds.windows("chr21")[0])   # TeacherWindow: key, dna (L,), signal (L, T), valid (L, T)
+```
+
+```
+zipoop verify /path/to/weights_npz      # size + sha256 against MANIFEST.tsv
+zipoop show   /path/to/weights_npz chr21_10027008_10059776
+```
+
+Lower level: `Dataset.read(window)` gives the parsed `Shard`, `Decoder(shard).decode(start, end, tracks)` decodes any slice or track subset. Evaluation helpers (`window_pearson`, `summarize`, `calibrate`, `restore_real`) are in `zipoop.evaluation`.
+
+`zipoop.evaluation.subwindow_metrics(pred, target, valid, subwindow=32768)` is the evaluator frozen in the D5 pipeline: it splits the length axis into contiguous sub-windows (4 x 32,768 bp inside a 131,072 bp segment) and returns `pearson` / `proj` / `nmse` / `total_err` / `eligible` arrays of shape (sub-windows, tracks), with the same clamps and the same eligibility rule (at least 2 observed positions and a non-constant target). The aggregation used by the frozen pipeline is the mean Pearson over eligible pairs and the median `proj` / `nmse`.
+
+Rules baked into `load_window` (see `zipoop/teacher.py`): RNA-seq tracks are 0 and valid where unrecorded; other tracks stay missing there. Opening a chromosome file parses its whole zip directory (chr1 about 3.6 s and 0.9 GB), so keep the `Dataset` open; forked DataLoader workers reopen their own handles.
+
+## Supported formats
+
+| archive | `format` | `hybrid_format` | storage |
+|---|---|---|---|
+| v3 | `pooper-dna-v3` | `fft-plus-dna-residual-v1` | deflate, `fft_coefficients` complex64 |
+| v4 | `pooper-dna-v4` | `fft-plus-dna-residual-v2` | LZMA, byte-shuffled floats, `fft_packed` float16 + `fft_scale` float32 |
+
+`read_shard` accepts both and rejects anything else with a clear error. The v4 decode path is verified against v3: parameters, masks and calibrations are bit-identical, the FFT payload differs only by float16 rounding (max relative error about 4.7e-4), and decoded windows reach a per-track Pearson of at least 0.9999997 against the v3 decode.
+
+Tests: `ZIPOOP_DATA=<weights_npz> pytest tests` (the pooper-equivalence test also needs the original shards and the pooper package). The v4 storage encoding has dataset-free unit tests in `tests/test_v4_format.py`.
+
+## License and data terms
+
+The code is MIT-licensed (see `LICENSE`; keep the copyright notice "zatumout"). The shard dataset is released separately under CC BY 4.0 (credit: zatumout), commercial use and model distillation allowed. The shards and `zipoop/data/encode-panel_1885.json` are derived from public ENCODE tracks: when you use them, also cite and acknowledge the ENCODE Consortium as described at https://www.encodeproject.org/help/citing-encode.

{zipoop-0.1.0 → zipoop-0.2.1}/zipoop.egg-info/SOURCES.txt

@@ -1,6 +1,8 @@
 LICENSE
 README.md
 pyproject.toml
+tests/test_eval_subwindow.py
+tests/test_v4_format.py
 tests/test_zipoop.py
 zipoop/__init__.py
 zipoop/cli.py

zipoop-0.1.0/PKG-INFO

@@ -1,51 +0,0 @@
-Metadata-Version: 2.4
-Name: zipoop
-Version: 0.1.0
-Summary: Decoder and reader for Zipoop teacher shards (per-window neural compression of ENCODE signal tracks)
-Author: zatumout
-License-Expression: MIT
-Project-URL: Dataset, https://huggingface.co/datasets/zatumout/zipoop-teacher-shards
-Keywords: genomics,ENCODE,neural compression,distillation
-Classifier: Programming Language :: Python :: 3
-Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
-Requires-Python: >=3.11
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: numpy>=1.26
-Requires-Dist: torch>=2.2
-Provides-Extra: test
-Requires-Dist: pytest>=8; extra == "test"
-Dynamic: license-file
-
-# zipoop
-
-Decoder and reader for the Zipoop teacher shards (dataset: https://huggingface.co/datasets/zatumout/zipoop-teacher-shards): 89,438 windows of 32 kb, 1,885 ENCODE tracks, stored as one `<chrom>.npz` per chromosome (`weights_npz`, with `MANIFEST.tsv`).
-
-## Data layout
-
-`<root>/<chrom>.npz` (25 files) plus `MANIFEST.tsv` (chrom, windows, members, bytes, sha256). Each file is a deflate zip with members `<window>/<key>.npy` (161 arrays per window), windows in genome order. Window names are `<chrom>_<start>_<end>`.
-
-## Usage
-
-```python
-from zipoop import Dataset
-with Dataset("/path/to/weights_npz") as ds:
-    w = ds.load_window(ds.windows("chr21")[0])   # TeacherWindow: key, dna (L,), signal (L, T), valid (L, T)
-```
-
-```
-zipoop verify /path/to/weights_npz      # size + sha256 against MANIFEST.tsv
-zipoop show   /path/to/weights_npz chr21_10027008_10059776
-```
-
-Lower level: `Dataset.read(window)` gives the parsed `Shard`, `Decoder(shard).decode(start, end, tracks)` decodes any slice or track subset. Evaluation helpers (`window_pearson`, `summarize`, `calibrate`, `restore_real`) are in `zipoop.evaluation`.
-
-Rules baked into `load_window` (see `zipoop/teacher.py`): RNA-seq tracks are 0 and valid where unrecorded; other tracks stay missing there. Opening a chromosome file parses its whole zip directory (chr1 about 3.6 s and 0.9 GB), so keep the `Dataset` open; forked DataLoader workers reopen their own handles.
-
-Only the shard variant in this dataset is supported (hybrid `pooper-dna-v3`, max-pool, sinusoidal positions, mask `track-rle-lzma-v2`); other variants raise a clear error.
-
-Tests: `ZIPOOP_DATA=<weights_npz> pytest tests` (the pooper-equivalence test also needs the original shards and the pooper package).
-
-## License and data terms
-
-The code is MIT-licensed (see `LICENSE`; keep the copyright notice "zatumout"). The shard dataset is released separately under CC BY 4.0 (credit: zatumout), commercial use and model distillation allowed. The shards and `zipoop/data/encode-panel_1885.json` are derived from public ENCODE tracks: when you use them, also cite and acknowledge the ENCODE Consortium as described at https://www.encodeproject.org/help/citing-encode.

zipoop-0.1.0/README.md

@@ -1,32 +0,0 @@
-# zipoop
-
-Decoder and reader for the Zipoop teacher shards (dataset: https://huggingface.co/datasets/zatumout/zipoop-teacher-shards): 89,438 windows of 32 kb, 1,885 ENCODE tracks, stored as one `<chrom>.npz` per chromosome (`weights_npz`, with `MANIFEST.tsv`).
-
-## Data layout
-
-`<root>/<chrom>.npz` (25 files) plus `MANIFEST.tsv` (chrom, windows, members, bytes, sha256). Each file is a deflate zip with members `<window>/<key>.npy` (161 arrays per window), windows in genome order. Window names are `<chrom>_<start>_<end>`.
-
-## Usage
-
-```python
-from zipoop import Dataset
-with Dataset("/path/to/weights_npz") as ds:
-    w = ds.load_window(ds.windows("chr21")[0])   # TeacherWindow: key, dna (L,), signal (L, T), valid (L, T)
-```
-
-```
-zipoop verify /path/to/weights_npz      # size + sha256 against MANIFEST.tsv
-zipoop show   /path/to/weights_npz chr21_10027008_10059776
-```
-
-Lower level: `Dataset.read(window)` gives the parsed `Shard`, `Decoder(shard).decode(start, end, tracks)` decodes any slice or track subset. Evaluation helpers (`window_pearson`, `summarize`, `calibrate`, `restore_real`) are in `zipoop.evaluation`.
-
-Rules baked into `load_window` (see `zipoop/teacher.py`): RNA-seq tracks are 0 and valid where unrecorded; other tracks stay missing there. Opening a chromosome file parses its whole zip directory (chr1 about 3.6 s and 0.9 GB), so keep the `Dataset` open; forked DataLoader workers reopen their own handles.
-
-Only the shard variant in this dataset is supported (hybrid `pooper-dna-v3`, max-pool, sinusoidal positions, mask `track-rle-lzma-v2`); other variants raise a clear error.
-
-Tests: `ZIPOOP_DATA=<weights_npz> pytest tests` (the pooper-equivalence test also needs the original shards and the pooper package).
-
-## License and data terms
-
-The code is MIT-licensed (see `LICENSE`; keep the copyright notice "zatumout"). The shard dataset is released separately under CC BY 4.0 (credit: zatumout), commercial use and model distillation allowed. The shards and `zipoop/data/encode-panel_1885.json` are derived from public ENCODE tracks: when you use them, also cite and acknowledge the ENCODE Consortium as described at https://www.encodeproject.org/help/citing-encode.

zipoop-0.1.0/zipoop/shard.py

@@ -1,105 +0,0 @@
-"""Parse and validate one teacher shard (a ``pooper-dna-v3`` hybrid archive) into a typed record."""
-from __future__ import annotations
-
-import json
-import os
-from contextlib import contextmanager
-from dataclasses import dataclass
-from pathlib import Path
-
-import numpy as np
-
-from .masks import MASK_FORMAT, decode_mask
-from .network import ModelConfig
-
-SHARD_FORMAT = "pooper-dna-v3"
-HYBRID_FORMAT = "fft-plus-dna-residual-v1"
-
-
-class ShardView:
-    """Read-only mapping over one window inside a merged per-chromosome npz (members "<window>/<key>")."""
-
-    def __init__(self, npz, key: str, members):
-        self._npz, self.key, self._members = npz, key, frozenset(members)
-
-    def __contains__(self, name):
-        return name in self._members
-
-    def __getitem__(self, name):
-        if name not in self._members:
-            raise KeyError(name)
-        return self._npz[f"{self.key}/{name}"]
-
-
-@dataclass
-class Shard:
-    key: str
-    shape: tuple[int, int]          # (length, tracks)
-    metadata: dict
-    config: ModelConfig
-    dna: np.ndarray                 # (L,) uint8
-    valid: np.ndarray               # (L, T) bool
-    parameters: dict[str, np.ndarray]
-    mean: np.ndarray                # per-track target scaling and calibration, (T,)
-    power: np.ndarray
-    gain: np.ndarray
-    offset: np.ndarray
-    fft_indices: np.ndarray         # (T, K) retained rfft bins
-    fft_coefficients: np.ndarray    # (T, K) complex64
-    final_gain: np.ndarray
-    final_offset: np.ndarray
-
-
-@contextmanager
-def _opened(source):
-    if isinstance(source, (str, os.PathLike)):
-        with np.load(source, allow_pickle=False) as z:
-            yield z, Path(source).stem
-    else:
-        yield source, source.key
-
-
-def read_shard(source) -> Shard:
-    """``source``: a shard path, or a ShardView from :class:`zipoop.Dataset`."""
-    with _opened(source) as (z, key):
-        if "format" not in z or str(z["format"]) != SHARD_FORMAT:
-            raise ValueError(f"Unsupported shard format: {z['format'] if 'format' in z else None}")
-        if "hybrid_format" not in z or str(z["hybrid_format"]) != HYBRID_FORMAT:
-            raise ValueError("Unsupported hybrid archive")
-        if "valid_encoding" not in z or str(z["valid_encoding"]) != MASK_FORMAT:
-            raise ValueError("Unsupported mask encoding")
-        shape = tuple(int(n) for n in z["shape"])
-        if len(shape) != 2:
-            raise ValueError("Invalid shape")
-        length, tracks = shape
-        dna = z["dna"].copy()
-        if dna.shape != (length,) or np.any(dna > 4):
-            raise ValueError("Invalid DNA data in archive")
-        names = json.loads(z["parameter_names"].tobytes())
-        if len(names) != len(set(names)):
-            raise ValueError("Duplicate parameter names")
-        shard = Shard(
-            key=key, shape=shape, metadata=json.loads(z["metadata"].tobytes()),
-            config=ModelConfig.from_json(json.loads(z["model_config"].tobytes())),
-            dna=dna, valid=decode_mask(z["valid_v2"].tobytes(), shape),
-            parameters={name: z[f"parameter_{i}"].copy() for i, name in enumerate(names)},
-            **{name: z[name].copy() for name in ("mean", "power", "gain", "offset")},
-            fft_indices=z["fft_indices"].copy(), fft_coefficients=z["fft_coefficients"].copy(),
-            final_gain=z["final_gain"].copy(), final_offset=z["final_offset"].copy())
-    _validate(shard)
-    return shard
-
-
-def _validate(s: Shard) -> None:
-    length, tracks = s.shape
-    if any(a.shape != (tracks,) for a in (s.mean, s.power, s.gain, s.offset)):
-        raise ValueError("Invalid track scales")
-    i, c = s.fft_indices, s.fft_coefficients
-    if (i.ndim != 2 or i.shape[0] != tracks or c.shape != i.shape or not np.issubdtype(i.dtype, np.integer)
-            or (i < 0).any() or (i > length // 2).any() or not np.isfinite(c).all()):
-        raise ValueError("Invalid FFT payload")
-    for a in (s.final_gain, s.final_offset):
-        if a.shape != (tracks,) or not np.isfinite(a).all():
-            raise ValueError("Invalid final-signal calibration")
-    if (s.final_gain < 0).any():
-        raise ValueError("Invalid final-signal calibration")

zipoop-0.1.0/zipoop.egg-info/PKG-INFO

@@ -1,51 +0,0 @@
-Metadata-Version: 2.4
-Name: zipoop
-Version: 0.1.0
-Summary: Decoder and reader for Zipoop teacher shards (per-window neural compression of ENCODE signal tracks)
-Author: zatumout
-License-Expression: MIT
-Project-URL: Dataset, https://huggingface.co/datasets/zatumout/zipoop-teacher-shards
-Keywords: genomics,ENCODE,neural compression,distillation
-Classifier: Programming Language :: Python :: 3
-Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
-Requires-Python: >=3.11
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: numpy>=1.26
-Requires-Dist: torch>=2.2
-Provides-Extra: test
-Requires-Dist: pytest>=8; extra == "test"
-Dynamic: license-file
-
-# zipoop
-
-Decoder and reader for the Zipoop teacher shards (dataset: https://huggingface.co/datasets/zatumout/zipoop-teacher-shards): 89,438 windows of 32 kb, 1,885 ENCODE tracks, stored as one `<chrom>.npz` per chromosome (`weights_npz`, with `MANIFEST.tsv`).
-
-## Data layout
-
-`<root>/<chrom>.npz` (25 files) plus `MANIFEST.tsv` (chrom, windows, members, bytes, sha256). Each file is a deflate zip with members `<window>/<key>.npy` (161 arrays per window), windows in genome order. Window names are `<chrom>_<start>_<end>`.
-
-## Usage
-
-```python
-from zipoop import Dataset
-with Dataset("/path/to/weights_npz") as ds:
-    w = ds.load_window(ds.windows("chr21")[0])   # TeacherWindow: key, dna (L,), signal (L, T), valid (L, T)
-```
-
-```
-zipoop verify /path/to/weights_npz      # size + sha256 against MANIFEST.tsv
-zipoop show   /path/to/weights_npz chr21_10027008_10059776
-```
-
-Lower level: `Dataset.read(window)` gives the parsed `Shard`, `Decoder(shard).decode(start, end, tracks)` decodes any slice or track subset. Evaluation helpers (`window_pearson`, `summarize`, `calibrate`, `restore_real`) are in `zipoop.evaluation`.
-
-Rules baked into `load_window` (see `zipoop/teacher.py`): RNA-seq tracks are 0 and valid where unrecorded; other tracks stay missing there. Opening a chromosome file parses its whole zip directory (chr1 about 3.6 s and 0.9 GB), so keep the `Dataset` open; forked DataLoader workers reopen their own handles.
-
-Only the shard variant in this dataset is supported (hybrid `pooper-dna-v3`, max-pool, sinusoidal positions, mask `track-rle-lzma-v2`); other variants raise a clear error.
-
-Tests: `ZIPOOP_DATA=<weights_npz> pytest tests` (the pooper-equivalence test also needs the original shards and the pooper package).
-
-## License and data terms
-
-The code is MIT-licensed (see `LICENSE`; keep the copyright notice "zatumout"). The shard dataset is released separately under CC BY 4.0 (credit: zatumout), commercial use and model distillation allowed. The shards and `zipoop/data/encode-panel_1885.json` are derived from public ENCODE tracks: when you use them, also cite and acknowledge the ENCODE Consortium as described at https://www.encodeproject.org/help/citing-encode.