zipoop 0.1.0__tar.gz → 0.2.0__tar.gz

{zipoop-0.1.0 → zipoop-0.2.0}/PKG-INFO

@@ -1,10 +1,11 @@
 Metadata-Version: 2.4
 Name: zipoop
-Version: 0.1.0
-Summary: Decoder and reader for Zipoop teacher shards (per-window neural compression of ENCODE signal tracks)
+Version: 0.2.0
+Summary: Decoder and reader for Zipoop teacher shards (per-window neural compression of ENCODE signal tracks), v3 and v4 archives
 Author: zatumout
 License-Expression: MIT
-Project-URL: Dataset, https://huggingface.co/datasets/zatumout/zipoop-teacher-shards
+Project-URL: Dataset, https://huggingface.co/datasets/zatumout/zipoop_32k1885_010_v4
+Project-URL: DatasetV3, https://huggingface.co/datasets/zatumout/zipoop_32k1885_010
 Keywords: genomics,ENCODE,neural compression,distillation
 Classifier: Programming Language :: Python :: 3
 Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
@@ -19,11 +20,18 @@ Dynamic: license-file
 
 # zipoop
 
-Decoder and reader for the Zipoop teacher shards (dataset: https://huggingface.co/datasets/zatumout/zipoop-teacher-shards): 89,438 windows of 32 kb, 1,885 ENCODE tracks, stored as one `<chrom>.npz` per chromosome (`weights_npz`, with `MANIFEST.tsv`).
+Decoder and reader for the Zipoop teacher shards: 89,438 windows of 32 kb, 1,885 ENCODE tracks (GRCh38), stored as one `<chrom>.npz` per chromosome plus `MANIFEST.tsv`.
+
+Two equivalent releases are published:
+
+- [zatumout/zipoop_32k1885_010_v4](https://huggingface.co/datasets/zatumout/zipoop_32k1885_010_v4) — `pooper-dna-v4`, 89.3 GiB, **requires `zipoop>=0.2`**;
+- [zatumout/zipoop_32k1885_010](https://huggingface.co/datasets/zatumout/zipoop_32k1885_010) — the older `pooper-dna-v3` encoding, 128.4 GiB, read by 0.1.x as well.
+
+Both decode to the same signals; v4 only changes how the arrays are stored (byte-shuffled members, LZMA, and a float16 sparse FFT with one float32 scale per track).
 
 ## Data layout
 
-`<root>/<chrom>.npz` (25 files) plus `MANIFEST.tsv` (chrom, windows, members, bytes, sha256). Each file is a deflate zip with members `<window>/<key>.npy` (161 arrays per window), windows in genome order. Window names are `<chrom>_<start>_<end>`.
+`<root>/<chrom>.npz` (25 files) plus `MANIFEST.tsv` (chrom, windows, members, bytes, sha256). Each file is a zip with members `<window>/<key>.npy` (161 arrays per window in v3, 164 in v4), windows in genome order. Window names are `<chrom>_<start>_<end>`.
 
 ## Usage
 
@@ -42,9 +50,16 @@ Lower level: `Dataset.read(window)` gives the parsed `Shard`, `Decoder(shard).de
 
 Rules baked into `load_window` (see `zipoop/teacher.py`): RNA-seq tracks are 0 and valid where unrecorded; other tracks stay missing there. Opening a chromosome file parses its whole zip directory (chr1 about 3.6 s and 0.9 GB), so keep the `Dataset` open; forked DataLoader workers reopen their own handles.
 
-Only the shard variant in this dataset is supported (hybrid `pooper-dna-v3`, max-pool, sinusoidal positions, mask `track-rle-lzma-v2`); other variants raise a clear error.
+## Supported formats
+
+| archive | `format` | `hybrid_format` | storage |
+|---|---|---|---|
+| v3 | `pooper-dna-v3` | `fft-plus-dna-residual-v1` | deflate, `fft_coefficients` complex64 |
+| v4 | `pooper-dna-v4` | `fft-plus-dna-residual-v2` | LZMA, byte-shuffled floats, `fft_packed` float16 + `fft_scale` float32 |
+
+`read_shard` accepts both and rejects anything else with a clear error. The v4 decode path is verified against v3: parameters, masks and calibrations are bit-identical, the FFT payload differs only by float16 rounding (max relative error about 4.7e-4), and decoded windows reach a per-track Pearson of at least 0.9999997 against the v3 decode.
 
-Tests: `ZIPOOP_DATA=<weights_npz> pytest tests` (the pooper-equivalence test also needs the original shards and the pooper package).
+Tests: `ZIPOOP_DATA=<weights_npz> pytest tests` (the pooper-equivalence test also needs the original shards and the pooper package). The v4 storage encoding has dataset-free unit tests in `tests/test_v4_format.py`.
 
 ## License and data terms
 

zipoop-0.2.0/README.md

@@ -0,0 +1,46 @@
+# zipoop
+
+Decoder and reader for the Zipoop teacher shards: 89,438 windows of 32 kb, 1,885 ENCODE tracks (GRCh38), stored as one `<chrom>.npz` per chromosome plus `MANIFEST.tsv`.
+
+Two equivalent releases are published:
+
+- [zatumout/zipoop_32k1885_010_v4](https://huggingface.co/datasets/zatumout/zipoop_32k1885_010_v4) — `pooper-dna-v4`, 89.3 GiB, **requires `zipoop>=0.2`**;
+- [zatumout/zipoop_32k1885_010](https://huggingface.co/datasets/zatumout/zipoop_32k1885_010) — the older `pooper-dna-v3` encoding, 128.4 GiB, read by 0.1.x as well.
+
+Both decode to the same signals; v4 only changes how the arrays are stored (byte-shuffled members, LZMA, and a float16 sparse FFT with one float32 scale per track).
+
+## Data layout
+
+`<root>/<chrom>.npz` (25 files) plus `MANIFEST.tsv` (chrom, windows, members, bytes, sha256). Each file is a zip with members `<window>/<key>.npy` (161 arrays per window in v3, 164 in v4), windows in genome order. Window names are `<chrom>_<start>_<end>`.
+
+## Usage
+
+```python
+from zipoop import Dataset
+with Dataset("/path/to/weights_npz") as ds:
+    w = ds.load_window(ds.windows("chr21")[0])   # TeacherWindow: key, dna (L,), signal (L, T), valid (L, T)
+```
+
+```
+zipoop verify /path/to/weights_npz      # size + sha256 against MANIFEST.tsv
+zipoop show   /path/to/weights_npz chr21_10027008_10059776
+```
+
+Lower level: `Dataset.read(window)` gives the parsed `Shard`, `Decoder(shard).decode(start, end, tracks)` decodes any slice or track subset. Evaluation helpers (`window_pearson`, `summarize`, `calibrate`, `restore_real`) are in `zipoop.evaluation`.
+
+Rules baked into `load_window` (see `zipoop/teacher.py`): RNA-seq tracks are 0 and valid where unrecorded; other tracks stay missing there. Opening a chromosome file parses its whole zip directory (chr1 about 3.6 s and 0.9 GB), so keep the `Dataset` open; forked DataLoader workers reopen their own handles.
+
+## Supported formats
+
+| archive | `format` | `hybrid_format` | storage |
+|---|---|---|---|
+| v3 | `pooper-dna-v3` | `fft-plus-dna-residual-v1` | deflate, `fft_coefficients` complex64 |
+| v4 | `pooper-dna-v4` | `fft-plus-dna-residual-v2` | LZMA, byte-shuffled floats, `fft_packed` float16 + `fft_scale` float32 |
+
+`read_shard` accepts both and rejects anything else with a clear error. The v4 decode path is verified against v3: parameters, masks and calibrations are bit-identical, the FFT payload differs only by float16 rounding (max relative error about 4.7e-4), and decoded windows reach a per-track Pearson of at least 0.9999997 against the v3 decode.
+
+Tests: `ZIPOOP_DATA=<weights_npz> pytest tests` (the pooper-equivalence test also needs the original shards and the pooper package). The v4 storage encoding has dataset-free unit tests in `tests/test_v4_format.py`.
+
+## License and data terms
+
+The code is MIT-licensed (see `LICENSE`; keep the copyright notice "zatumout"). The shard dataset is released separately under CC BY 4.0 (credit: zatumout), commercial use and model distillation allowed. The shards and `zipoop/data/encode-panel_1885.json` are derived from public ENCODE tracks: when you use them, also cite and acknowledge the ENCODE Consortium as described at https://www.encodeproject.org/help/citing-encode.

{zipoop-0.1.0 → zipoop-0.2.0}/pyproject.toml

@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "zipoop"
-version = "0.1.0"
-description = "Decoder and reader for Zipoop teacher shards (per-window neural compression of ENCODE signal tracks)"
+version = "0.2.0"
+description = "Decoder and reader for Zipoop teacher shards (per-window neural compression of ENCODE signal tracks), v3 and v4 archives"
 readme = "README.md"
 requires-python = ">=3.11"
 license = "MIT"
@@ -19,7 +19,8 @@ dependencies = ["numpy>=1.26", "torch>=2.2"]
 test = ["pytest>=8"]
 
 [project.urls]
-Dataset = "https://huggingface.co/datasets/zatumout/zipoop-teacher-shards"
+Dataset = "https://huggingface.co/datasets/zatumout/zipoop_32k1885_010_v4"
+DatasetV3 = "https://huggingface.co/datasets/zatumout/zipoop_32k1885_010"
 
 [project.scripts]
 zipoop = "zipoop.cli:main"

zipoop-0.2.0/tests/test_v4_format.py

@@ -0,0 +1,60 @@
+"""Unit tests for the v4 storage encoding (no dataset required)."""
+import numpy as np
+
+from zipoop.shard import HYBRID_FORMAT_V1, HYBRID_FORMAT_V2, _Reader
+
+
+def shuffle(a):
+    """Mirror of the writer: float/complex arrays -> (itemsize, n) uint8 columns."""
+    v = np.asarray(a)
+    return np.ascontiguousarray(v).view(np.uint8).reshape(-1, v.dtype.itemsize).T.copy()
+
+
+def spec_of(**arrays):
+    return {k: {"dtype": np.asarray(v).dtype.str, "shape": list(np.asarray(v).shape)} for k, v in arrays.items()}
+
+
+def test_byte_shuffle_roundtrip():
+    rng = np.random.default_rng(0)
+    a = rng.standard_normal((7, 5)).astype(np.float32)
+    view = {"gain": shuffle(a)}
+    out = _Reader(view, spec_of(gain=a), HYBRID_FORMAT_V1)["gain"]
+    assert out.dtype == np.float32
+    assert np.array_equal(out, a)
+
+
+def test_unshuffled_members_pass_through():
+    a = np.arange(12, dtype=np.uint16)
+    assert np.array_equal(_Reader({"fft_indices": a}, {}, HYBRID_FORMAT_V1)["fft_indices"], a)
+
+
+def test_fft_v2_dequantization():
+    rng = np.random.default_rng(1)
+    tracks, bins = 4, 6
+    coefficients = (rng.standard_normal((tracks, bins)) + 1j * rng.standard_normal((tracks, bins))).astype(np.complex64)
+    pairs = np.stack((coefficients.real, coefficients.imag), axis=-1)
+    scale = np.abs(pairs).max(axis=(1, 2), keepdims=True).astype(np.float32)
+    packed = (pairs / scale).astype(np.float16)
+    scale = scale.reshape(tracks, 1, 1)
+    indices = np.sort(rng.choice(64, size=bins, replace=False)).astype(np.uint16)[None, :].repeat(tracks, 0)
+
+    view = {"fft_indices": indices, "fft_packed": shuffle(packed), "fft_scale": shuffle(scale)}
+    reader = _Reader(view, spec_of(fft_packed=packed, fft_scale=scale), HYBRID_FORMAT_V2)
+    got_i, got_c = reader.fft()
+    assert np.array_equal(got_i, indices)
+    assert got_c.dtype == np.complex64
+    assert got_c.shape == (tracks, bins)
+    reference = packed.astype(np.float32) * scale           # the same float32 reference the writer used
+    reference = reference[..., 0] + 1j * reference[..., 1]
+    assert np.allclose(got_c, reference, rtol=0, atol=1e-6)
+    # and it stays within float16 rounding of the original complex64 payload
+    assert np.max(np.abs(got_c - coefficients)) < 0.02 * np.max(np.abs(coefficients))
+
+
+def test_v3_reader_uses_complex_payload():
+    tracks, bins = 2, 3
+    coefficients = (np.ones((tracks, bins)) * (1 + 2j)).astype(np.complex64)
+    indices = np.zeros((tracks, bins), dtype=np.uint16)
+    got_i, got_c = _Reader({"fft_indices": indices, "fft_coefficients": coefficients}, {}, HYBRID_FORMAT_V1).fft()
+    assert np.array_equal(got_i, indices)
+    assert np.array_equal(got_c, coefficients)

{zipoop-0.1.0 → zipoop-0.2.0}/zipoop/__init__.py

@@ -6,6 +6,6 @@ from .panel import panel_tracks, zero_tracks
 from .shard import Shard, read_shard
 from .teacher import TeacherWindow, load_window
 
-__version__ = "0.1.0"
+__version__ = "0.2.0"
 __all__ = ["Dataset", "verify", "Decoder", "Shard", "read_shard", "TeacherWindow", "load_window",
            "panel_tracks", "zero_tracks", "calibrate", "restore_real", "summarize", "window_pearson"]

zipoop-0.2.0/zipoop/shard.py

@@ -0,0 +1,158 @@
+"""Parse and validate one teacher shard (a ``pooper-dna-v3`` or ``pooper-dna-v4`` hybrid archive) into a typed record."""
+from __future__ import annotations
+
+import json
+import os
+from contextlib import contextmanager
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+from .masks import MASK_FORMAT, decode_mask
+from .network import ModelConfig
+
+SHARD_FORMAT_V3 = "pooper-dna-v3"
+SHARD_FORMAT_V4 = "pooper-dna-v4"
+SHARD_FORMATS = (SHARD_FORMAT_V3, SHARD_FORMAT_V4)
+HYBRID_FORMAT_V1 = "fft-plus-dna-residual-v1"
+HYBRID_FORMAT_V2 = "fft-plus-dna-residual-v2"
+HYBRID_FORMATS = (HYBRID_FORMAT_V1, HYBRID_FORMAT_V2)
+STORAGE_ENCODING_V4 = "byte-shuffle-lzma-v1"
+
+
+class ShardView:
+    """Read-only mapping over one window inside a merged per-chromosome npz (members "<window>/<key>")."""
+
+    def __init__(self, npz, key: str, members):
+        self._npz, self.key, self._members = npz, key, frozenset(members)
+
+    def __contains__(self, name):
+        return name in self._members
+
+    def __getitem__(self, name):
+        if name not in self._members:
+            raise KeyError(name)
+        return self._npz[f"{self.key}/{name}"]
+
+
+@dataclass
+class Shard:
+    key: str
+    shape: tuple[int, int]          # (length, tracks)
+    metadata: dict
+    config: ModelConfig
+    dna: np.ndarray                 # (L,) uint8
+    valid: np.ndarray               # (L, T) bool
+    parameters: dict[str, np.ndarray]
+    mean: np.ndarray                # per-track target scaling and calibration, (T,)
+    power: np.ndarray
+    gain: np.ndarray
+    offset: np.ndarray
+    fft_indices: np.ndarray         # (T, K) retained rfft bins
+    fft_coefficients: np.ndarray    # (T, K) complex64
+    final_gain: np.ndarray
+    final_offset: np.ndarray
+
+
+@contextmanager
+def _opened(source):
+    if isinstance(source, (str, os.PathLike)):
+        with np.load(source, allow_pickle=False) as z:
+            yield z, Path(source).stem
+    else:
+        yield source, source.key
+
+
+class _Reader:
+    """Member access that transparently undoes the v4 byte-shuffle and FFT quantization."""
+
+    def __init__(self, view, spec, hybrid_format):
+        self._view = view
+        self._spec = spec
+        self._hybrid = hybrid_format
+
+    def __contains__(self, name):
+        return name in self._view
+
+    def raw(self, name):
+        return self._view[name]
+
+    def __getitem__(self, name):
+        a = self._view[name]
+        meta = self._spec.get(name)
+        if meta is not None:
+            # v4 stores float/complex arrays as (itemsize, n) uint8 columns.
+            packed = np.ascontiguousarray(np.asarray(a))
+            if packed.dtype != np.uint8 or packed.ndim != 2:
+                raise ValueError(f"Invalid byte-shuffled member {name!r}")
+            b = np.ascontiguousarray(packed.T)
+            a = b.view(np.dtype(meta["dtype"])).reshape(tuple(meta["shape"]))
+        return a
+
+    def fft(self):
+        """(indices, complex64 coefficients) for either hybrid format."""
+        indices = np.asarray(self["fft_indices"])
+        if self._hybrid == HYBRID_FORMAT_V2:
+            packed = np.asarray(self["fft_packed"], dtype=np.float32)
+            scale = np.asarray(self["fft_scale"], dtype=np.float32)
+            if packed.ndim != 3 or packed.shape[-1] != 2 or packed.shape[:2] != indices.shape:
+                raise ValueError("Invalid packed FFT payload")
+            coefficients = (packed[..., 0] + 1j * packed[..., 1]) * scale.reshape(-1, 1)
+            return indices, np.ascontiguousarray(coefficients, dtype=np.complex64)
+        return indices, np.asarray(self["fft_coefficients"], dtype=np.complex64)
+
+
+def read_shard(source) -> Shard:
+    """``source``: a shard path, or a ShardView from :class:`zipoop.Dataset`."""
+    with _opened(source) as (z, key):
+        if "format" not in z or str(z["format"]) not in SHARD_FORMATS:
+            raise ValueError(f"Unsupported shard format: {z['format'] if 'format' in z else None}")
+        if "hybrid_format" not in z or str(z["hybrid_format"]) not in HYBRID_FORMATS:
+            raise ValueError("Unsupported hybrid archive")
+        if "valid_encoding" not in z or str(z["valid_encoding"]) != MASK_FORMAT:
+            raise ValueError("Unsupported mask encoding")
+        spec = {}
+        if "byte_shuffle" in z:
+            spec = json.loads(z["byte_shuffle"].tobytes())
+            if not isinstance(spec, dict):
+                raise ValueError("Invalid byte-shuffle index")
+            if "storage_encoding" in z and str(z["storage_encoding"]) != STORAGE_ENCODING_V4:
+                raise ValueError(f"Unsupported storage encoding: {str(z['storage_encoding'])}")
+        view = _Reader(z, spec, str(z["hybrid_format"]))
+        shape = tuple(int(n) for n in z["shape"])
+        if len(shape) != 2:
+            raise ValueError("Invalid shape")
+        length, tracks = shape
+        dna = view["dna"].copy()
+        if dna.shape != (length,) or np.any(dna > 4):
+            raise ValueError("Invalid DNA data in archive")
+        names = json.loads(view["parameter_names"].tobytes())
+        if len(names) != len(set(names)):
+            raise ValueError("Duplicate parameter names")
+        fft_indices, fft_coefficients = view.fft()
+        shard = Shard(
+            key=key, shape=shape, metadata=json.loads(view["metadata"].tobytes()),
+            config=ModelConfig.from_json(json.loads(view["model_config"].tobytes())),
+            dna=dna, valid=decode_mask(view["valid_v2"].tobytes(), shape),
+            parameters={name: view[f"parameter_{i}"].copy() for i, name in enumerate(names)},
+            fft_indices=fft_indices.copy(), fft_coefficients=fft_coefficients.copy(),
+            **{name: view[name].copy() for name in ("mean", "power", "gain", "offset",
+                                                    "final_gain", "final_offset")})
+    _validate(shard)
+    return shard
+
+
+def _validate(s: Shard) -> None:
+    length, tracks = s.shape
+    if any(a.shape != (tracks,) for a in (s.mean, s.power, s.gain, s.offset)):
+        raise ValueError("Invalid track scales")
+    i, c = s.fft_indices, s.fft_coefficients
+    if (i.ndim != 2 or i.shape[0] != tracks or c.shape != i.shape or not np.issubdtype(i.dtype, np.integer)
+            or (i < 0).any() or (i > length // 2).any() or not np.isfinite(c).all()):
+        raise ValueError("Invalid FFT payload")
+    for a in (s.final_gain, s.final_offset):
+        if a.shape != (tracks,) or not np.isfinite(a).all():
+            raise ValueError("Invalid final-signal calibration")
+    if (s.final_gain < 0).any():
+        raise ValueError("Invalid final-signal calibration")

{zipoop-0.1.0 → zipoop-0.2.0}/zipoop.egg-info/PKG-INFO

@@ -1,10 +1,11 @@
 Metadata-Version: 2.4
 Name: zipoop
-Version: 0.1.0
-Summary: Decoder and reader for Zipoop teacher shards (per-window neural compression of ENCODE signal tracks)
+Version: 0.2.0
+Summary: Decoder and reader for Zipoop teacher shards (per-window neural compression of ENCODE signal tracks), v3 and v4 archives
 Author: zatumout
 License-Expression: MIT
-Project-URL: Dataset, https://huggingface.co/datasets/zatumout/zipoop-teacher-shards
+Project-URL: Dataset, https://huggingface.co/datasets/zatumout/zipoop_32k1885_010_v4
+Project-URL: DatasetV3, https://huggingface.co/datasets/zatumout/zipoop_32k1885_010
 Keywords: genomics,ENCODE,neural compression,distillation
 Classifier: Programming Language :: Python :: 3
 Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
@@ -19,11 +20,18 @@ Dynamic: license-file
 
 # zipoop
 
-Decoder and reader for the Zipoop teacher shards (dataset: https://huggingface.co/datasets/zatumout/zipoop-teacher-shards): 89,438 windows of 32 kb, 1,885 ENCODE tracks, stored as one `<chrom>.npz` per chromosome (`weights_npz`, with `MANIFEST.tsv`).
+Decoder and reader for the Zipoop teacher shards: 89,438 windows of 32 kb, 1,885 ENCODE tracks (GRCh38), stored as one `<chrom>.npz` per chromosome plus `MANIFEST.tsv`.
+
+Two equivalent releases are published:
+
+- [zatumout/zipoop_32k1885_010_v4](https://huggingface.co/datasets/zatumout/zipoop_32k1885_010_v4) — `pooper-dna-v4`, 89.3 GiB, **requires `zipoop>=0.2`**;
+- [zatumout/zipoop_32k1885_010](https://huggingface.co/datasets/zatumout/zipoop_32k1885_010) — the older `pooper-dna-v3` encoding, 128.4 GiB, read by 0.1.x as well.
+
+Both decode to the same signals; v4 only changes how the arrays are stored (byte-shuffled members, LZMA, and a float16 sparse FFT with one float32 scale per track).
 
 ## Data layout
 
-`<root>/<chrom>.npz` (25 files) plus `MANIFEST.tsv` (chrom, windows, members, bytes, sha256). Each file is a deflate zip with members `<window>/<key>.npy` (161 arrays per window), windows in genome order. Window names are `<chrom>_<start>_<end>`.
+`<root>/<chrom>.npz` (25 files) plus `MANIFEST.tsv` (chrom, windows, members, bytes, sha256). Each file is a zip with members `<window>/<key>.npy` (161 arrays per window in v3, 164 in v4), windows in genome order. Window names are `<chrom>_<start>_<end>`.
 
 ## Usage
 
@@ -42,9 +50,16 @@ Lower level: `Dataset.read(window)` gives the parsed `Shard`, `Decoder(shard).de
 
 Rules baked into `load_window` (see `zipoop/teacher.py`): RNA-seq tracks are 0 and valid where unrecorded; other tracks stay missing there. Opening a chromosome file parses its whole zip directory (chr1 about 3.6 s and 0.9 GB), so keep the `Dataset` open; forked DataLoader workers reopen their own handles.
 
-Only the shard variant in this dataset is supported (hybrid `pooper-dna-v3`, max-pool, sinusoidal positions, mask `track-rle-lzma-v2`); other variants raise a clear error.
+## Supported formats
+
+| archive | `format` | `hybrid_format` | storage |
+|---|---|---|---|
+| v3 | `pooper-dna-v3` | `fft-plus-dna-residual-v1` | deflate, `fft_coefficients` complex64 |
+| v4 | `pooper-dna-v4` | `fft-plus-dna-residual-v2` | LZMA, byte-shuffled floats, `fft_packed` float16 + `fft_scale` float32 |
+
+`read_shard` accepts both and rejects anything else with a clear error. The v4 decode path is verified against v3: parameters, masks and calibrations are bit-identical, the FFT payload differs only by float16 rounding (max relative error about 4.7e-4), and decoded windows reach a per-track Pearson of at least 0.9999997 against the v3 decode.
 
-Tests: `ZIPOOP_DATA=<weights_npz> pytest tests` (the pooper-equivalence test also needs the original shards and the pooper package).
+Tests: `ZIPOOP_DATA=<weights_npz> pytest tests` (the pooper-equivalence test also needs the original shards and the pooper package). The v4 storage encoding has dataset-free unit tests in `tests/test_v4_format.py`.
 
 ## License and data terms
 

{zipoop-0.1.0 → zipoop-0.2.0}/zipoop.egg-info/SOURCES.txt

@@ -1,6 +1,7 @@
 LICENSE
 README.md
 pyproject.toml
+tests/test_v4_format.py
 tests/test_zipoop.py
 zipoop/__init__.py
 zipoop/cli.py

zipoop-0.1.0/README.md

@@ -1,32 +0,0 @@
-# zipoop
-
-Decoder and reader for the Zipoop teacher shards (dataset: https://huggingface.co/datasets/zatumout/zipoop-teacher-shards): 89,438 windows of 32 kb, 1,885 ENCODE tracks, stored as one `<chrom>.npz` per chromosome (`weights_npz`, with `MANIFEST.tsv`).
-
-## Data layout
-
-`<root>/<chrom>.npz` (25 files) plus `MANIFEST.tsv` (chrom, windows, members, bytes, sha256). Each file is a deflate zip with members `<window>/<key>.npy` (161 arrays per window), windows in genome order. Window names are `<chrom>_<start>_<end>`.
-
-## Usage
-
-```python
-from zipoop import Dataset
-with Dataset("/path/to/weights_npz") as ds:
-    w = ds.load_window(ds.windows("chr21")[0])   # TeacherWindow: key, dna (L,), signal (L, T), valid (L, T)
-```
-
-```
-zipoop verify /path/to/weights_npz      # size + sha256 against MANIFEST.tsv
-zipoop show   /path/to/weights_npz chr21_10027008_10059776
-```
-
-Lower level: `Dataset.read(window)` gives the parsed `Shard`, `Decoder(shard).decode(start, end, tracks)` decodes any slice or track subset. Evaluation helpers (`window_pearson`, `summarize`, `calibrate`, `restore_real`) are in `zipoop.evaluation`.
-
-Rules baked into `load_window` (see `zipoop/teacher.py`): RNA-seq tracks are 0 and valid where unrecorded; other tracks stay missing there. Opening a chromosome file parses its whole zip directory (chr1 about 3.6 s and 0.9 GB), so keep the `Dataset` open; forked DataLoader workers reopen their own handles.
-
-Only the shard variant in this dataset is supported (hybrid `pooper-dna-v3`, max-pool, sinusoidal positions, mask `track-rle-lzma-v2`); other variants raise a clear error.
-
-Tests: `ZIPOOP_DATA=<weights_npz> pytest tests` (the pooper-equivalence test also needs the original shards and the pooper package).
-
-## License and data terms
-
-The code is MIT-licensed (see `LICENSE`; keep the copyright notice "zatumout"). The shard dataset is released separately under CC BY 4.0 (credit: zatumout), commercial use and model distillation allowed. The shards and `zipoop/data/encode-panel_1885.json` are derived from public ENCODE tracks: when you use them, also cite and acknowledge the ENCODE Consortium as described at https://www.encodeproject.org/help/citing-encode.

zipoop-0.1.0/zipoop/shard.py

@@ -1,105 +0,0 @@
-"""Parse and validate one teacher shard (a ``pooper-dna-v3`` hybrid archive) into a typed record."""
-from __future__ import annotations
-
-import json
-import os
-from contextlib import contextmanager
-from dataclasses import dataclass
-from pathlib import Path
-
-import numpy as np
-
-from .masks import MASK_FORMAT, decode_mask
-from .network import ModelConfig
-
-SHARD_FORMAT = "pooper-dna-v3"
-HYBRID_FORMAT = "fft-plus-dna-residual-v1"
-
-
-class ShardView:
-    """Read-only mapping over one window inside a merged per-chromosome npz (members "<window>/<key>")."""
-
-    def __init__(self, npz, key: str, members):
-        self._npz, self.key, self._members = npz, key, frozenset(members)
-
-    def __contains__(self, name):
-        return name in self._members
-
-    def __getitem__(self, name):
-        if name not in self._members:
-            raise KeyError(name)
-        return self._npz[f"{self.key}/{name}"]
-
-
-@dataclass
-class Shard:
-    key: str
-    shape: tuple[int, int]          # (length, tracks)
-    metadata: dict
-    config: ModelConfig
-    dna: np.ndarray                 # (L,) uint8
-    valid: np.ndarray               # (L, T) bool
-    parameters: dict[str, np.ndarray]
-    mean: np.ndarray                # per-track target scaling and calibration, (T,)
-    power: np.ndarray
-    gain: np.ndarray
-    offset: np.ndarray
-    fft_indices: np.ndarray         # (T, K) retained rfft bins
-    fft_coefficients: np.ndarray    # (T, K) complex64
-    final_gain: np.ndarray
-    final_offset: np.ndarray
-
-
-@contextmanager
-def _opened(source):
-    if isinstance(source, (str, os.PathLike)):
-        with np.load(source, allow_pickle=False) as z:
-            yield z, Path(source).stem
-    else:
-        yield source, source.key
-
-
-def read_shard(source) -> Shard:
-    """``source``: a shard path, or a ShardView from :class:`zipoop.Dataset`."""
-    with _opened(source) as (z, key):
-        if "format" not in z or str(z["format"]) != SHARD_FORMAT:
-            raise ValueError(f"Unsupported shard format: {z['format'] if 'format' in z else None}")
-        if "hybrid_format" not in z or str(z["hybrid_format"]) != HYBRID_FORMAT:
-            raise ValueError("Unsupported hybrid archive")
-        if "valid_encoding" not in z or str(z["valid_encoding"]) != MASK_FORMAT:
-            raise ValueError("Unsupported mask encoding")
-        shape = tuple(int(n) for n in z["shape"])
-        if len(shape) != 2:
-            raise ValueError("Invalid shape")
-        length, tracks = shape
-        dna = z["dna"].copy()
-        if dna.shape != (length,) or np.any(dna > 4):
-            raise ValueError("Invalid DNA data in archive")
-        names = json.loads(z["parameter_names"].tobytes())
-        if len(names) != len(set(names)):
-            raise ValueError("Duplicate parameter names")
-        shard = Shard(
-            key=key, shape=shape, metadata=json.loads(z["metadata"].tobytes()),
-            config=ModelConfig.from_json(json.loads(z["model_config"].tobytes())),
-            dna=dna, valid=decode_mask(z["valid_v2"].tobytes(), shape),
-            parameters={name: z[f"parameter_{i}"].copy() for i, name in enumerate(names)},
-            **{name: z[name].copy() for name in ("mean", "power", "gain", "offset")},
-            fft_indices=z["fft_indices"].copy(), fft_coefficients=z["fft_coefficients"].copy(),
-            final_gain=z["final_gain"].copy(), final_offset=z["final_offset"].copy())
-    _validate(shard)
-    return shard
-
-
-def _validate(s: Shard) -> None:
-    length, tracks = s.shape
-    if any(a.shape != (tracks,) for a in (s.mean, s.power, s.gain, s.offset)):
-        raise ValueError("Invalid track scales")
-    i, c = s.fft_indices, s.fft_coefficients
-    if (i.ndim != 2 or i.shape[0] != tracks or c.shape != i.shape or not np.issubdtype(i.dtype, np.integer)
-            or (i < 0).any() or (i > length // 2).any() or not np.isfinite(c).all()):
-        raise ValueError("Invalid FFT payload")
-    for a in (s.final_gain, s.final_offset):
-        if a.shape != (tracks,) or not np.isfinite(a).all():
-            raise ValueError("Invalid final-signal calibration")
-    if (s.final_gain < 0).any():
-        raise ValueError("Invalid final-signal calibration")