scigantic-empiar 0.1.0__tar.gz

scigantic_empiar-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Scigantic
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

scigantic_empiar-0.1.0/PKG-INFO

@@ -0,0 +1,85 @@
+Metadata-Version: 2.4
+Name: scigantic-empiar
+Version: 0.1.0
+Summary: Explore the EMPIAR cryo-EM archive from Python — stream any of ~3,000 datasets (8.9 PiB) over parallel HTTP range reads, nothing downloaded.
+Author: Scigantic
+License: MIT
+Project-URL: Homepage, https://github.com/scigantic/scigantic-empiar
+Project-URL: EMPIAR, https://www.ebi.ac.uk/empiar/
+Keywords: cryo-em,cryo-et,empiar,mrc,structural-biology,microscopy
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: requests
+Provides-Extra: viz
+Requires-Dist: matplotlib; extra == "viz"
+Requires-Dist: pandas; extra == "viz"
+Requires-Dist: pillow; extra == "viz"
+Requires-Dist: ipython; extra == "viz"
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: matplotlib; extra == "dev"
+Requires-Dist: pandas; extra == "dev"
+Dynamic: license-file
+
+# scigantic_empiar
+
+Explore [EMPIAR](https://www.ebi.ac.uk/empiar/) — EMBL-EBI's public archive of **raw cryo-EM / cryo-ET image data** (~3,000 datasets, ~8.9 PiB) — from Python, **without downloading anything**.
+
+EMPIAR is served over EBI's public HTTPS at ~1.5 MB/s per connection. `scigantic_empiar` parallelises HTTP **range** reads (8-way ≈ 5–10 MB/s) so you can pull a single frame from a many-GB entry in seconds, decode the MRC, and render the micrograph + its power spectrum — nothing is copied to disk.
+
+```python
+import scigantic_empiar as se
+
+se.preview(10406)                      # render the micrograph below, in seconds
+se.EmpiarClient().summary(10406)       # title, pixel size, method, DOI, EMDB/PDB cross-refs
+se.EmpiarCatalog().search("ribosome")  # search the whole archive by metadata (instant)
+```
+
+![A motion-corrected 70S-ribosome micrograph (EMPIAR-10406) and its power spectrum, rendered by se.preview(10406)](https://raw.githubusercontent.com/scigantic/scigantic-empiar/main/docs/preview_10406.png)
+
+*One frame of EMPIAR-10406 (a 70S-ribosome dataset) pulled straight from EBI over parallel range reads — the carbon-foil edge, ice, and particles are visible at left; the FFT is at right. Nothing was downloaded to disk.*
+
+## Install
+
+```bash
+pip install "scigantic-empiar[viz] @ git+https://github.com/scigantic/scigantic-empiar"
+```
+
+Core (`numpy`, `requests`) is enough for the readers; `[viz]` adds `matplotlib` / `pandas` / `pillow` for `preview()` and the catalog gallery.
+
+## What it does
+
+| | |
+|---|---|
+| `preview(id)` | micrograph / tomogram-slice + power spectrum, rendered from a lazy parallel-range read |
+| `read_mrc_frame(id)` / `read_mrc_average(id)` | one frame / a mean of frames as a NumPy array + header |
+| `thumbnail(id)` | small preview array (a few-MB central-strip read) — used to build catalogs |
+| `find_mrc(id)` | resolve an entry's first MRC, recursing the (often nested) `data/` layout |
+| `pread(url, off, len)` | the 8-way parallel HTTP range reader under it all |
+| `EmpiarClient` | per-entry metadata from EMPIAR's REST API (cached) |
+| `EmpiarCatalog` | search + a visual thumbnail gallery across all entries (from a prebuilt index) |
+| `add_to_fast_workspace(id)` | mirror an entry to S3 for full-speed reprocessing (RELION/EMAN2) |
+
+## Why parallel range reads
+
+EBI throttles per connection (~1.5 MB/s) and past ~8 concurrent connections. `pread` splits a read into ~8 concurrent range requests, which aggregates to ~5–10 MB/s — enough to *look* at any entry interactively. For heavy reprocessing of a whole multi-hundred-GB dataset, mirror it to fast storage first (`add_to_fast_workspace`); streaming a full entry at 1.5 MB/s isn't practical.
+
+## Existing work
+
+The job splits in two: parse MRC, and read bytes from a remote file. Both have existing libraries; neither covers the specific case here.
+
+- [`mrcfile`](https://github.com/ccpem/mrcfile) (CCP-EM) is the standard MRC reader. Its lazy mode is a numpy `memmap`, which needs a local filesystem path — it does not issue HTTP range requests. `scigantic_empiar` parses the 1024-byte header directly (`parse_mrc_header`) to seek to one frame of a remote file without a local copy.
+- [`fsspec`](https://filesystem-spec.readthedocs.io/) `HTTPFileSystem` turns byte reads into HTTP range requests and can fetch many ranges concurrently ([`cat_ranges`](https://filesystem-spec.readthedocs.io/en/latest/async.html)). `pread` is a small equivalent, kept dependency-free and tuned to EBI's ~8-connection throttle; moving the transport onto `fsspec` is a reasonable later change.
+- [`copick`](https://github.com/copick/copick) (CZI, [Protein Science 2026](https://onlinelibrary.wiley.com/doi/10.1002/pro.70578)) is the closest cryo-EM analog: an fsspec-backed, server-less dataset API with lazy reads. It assumes data stored as OME-Zarr (chunked, multiscale). EMPIAR entries are raw MRC/TIFF, so copick needs a per-entry zarr conversion first — the conversion that MRC's flat layout lets `scigantic_empiar` skip.
+
+## Notes
+
+- MRC/MRCS (movies, micrographs, tomograms, particle stacks) and some TIFF. Files often nest a couple subdir levels down; `find_mrc` handles that.
+- Entry ids are opaque numbers — discover datasets by **metadata** (`EmpiarCatalog.search`, or the EMPIAR website), not by listing the tree.
+- Inside a [Scigantic](https://scigantic.com) cryo-EM notebook this is preinstalled and the archive is also FUSE-mounted at `$SCIGANTIC_MOUNT_PATH`; standalone, it streams straight from EBI.
+
+## License
+
+MIT.

scigantic_empiar-0.1.0/README.md

@@ -0,0 +1,60 @@
+# scigantic_empiar
+
+Explore [EMPIAR](https://www.ebi.ac.uk/empiar/) — EMBL-EBI's public archive of **raw cryo-EM / cryo-ET image data** (~3,000 datasets, ~8.9 PiB) — from Python, **without downloading anything**.
+
+EMPIAR is served over EBI's public HTTPS at ~1.5 MB/s per connection. `scigantic_empiar` parallelises HTTP **range** reads (8-way ≈ 5–10 MB/s) so you can pull a single frame from a many-GB entry in seconds, decode the MRC, and render the micrograph + its power spectrum — nothing is copied to disk.
+
+```python
+import scigantic_empiar as se
+
+se.preview(10406)                      # render the micrograph below, in seconds
+se.EmpiarClient().summary(10406)       # title, pixel size, method, DOI, EMDB/PDB cross-refs
+se.EmpiarCatalog().search("ribosome")  # search the whole archive by metadata (instant)
+```
+
+![A motion-corrected 70S-ribosome micrograph (EMPIAR-10406) and its power spectrum, rendered by se.preview(10406)](https://raw.githubusercontent.com/scigantic/scigantic-empiar/main/docs/preview_10406.png)
+
+*One frame of EMPIAR-10406 (a 70S-ribosome dataset) pulled straight from EBI over parallel range reads — the carbon-foil edge, ice, and particles are visible at left; the FFT is at right. Nothing was downloaded to disk.*
+
+## Install
+
+```bash
+pip install "scigantic-empiar[viz] @ git+https://github.com/scigantic/scigantic-empiar"
+```
+
+Core (`numpy`, `requests`) is enough for the readers; `[viz]` adds `matplotlib` / `pandas` / `pillow` for `preview()` and the catalog gallery.
+
+## What it does
+
+| | |
+|---|---|
+| `preview(id)` | micrograph / tomogram-slice + power spectrum, rendered from a lazy parallel-range read |
+| `read_mrc_frame(id)` / `read_mrc_average(id)` | one frame / a mean of frames as a NumPy array + header |
+| `thumbnail(id)` | small preview array (a few-MB central-strip read) — used to build catalogs |
+| `find_mrc(id)` | resolve an entry's first MRC, recursing the (often nested) `data/` layout |
+| `pread(url, off, len)` | the 8-way parallel HTTP range reader under it all |
+| `EmpiarClient` | per-entry metadata from EMPIAR's REST API (cached) |
+| `EmpiarCatalog` | search + a visual thumbnail gallery across all entries (from a prebuilt index) |
+| `add_to_fast_workspace(id)` | mirror an entry to S3 for full-speed reprocessing (RELION/EMAN2) |
+
+## Why parallel range reads
+
+EBI throttles per connection (~1.5 MB/s) and past ~8 concurrent connections. `pread` splits a read into ~8 concurrent range requests, which aggregates to ~5–10 MB/s — enough to *look* at any entry interactively. For heavy reprocessing of a whole multi-hundred-GB dataset, mirror it to fast storage first (`add_to_fast_workspace`); streaming a full entry at 1.5 MB/s isn't practical.
+
+## Existing work
+
+The job splits in two: parse MRC, and read bytes from a remote file. Both have existing libraries; neither covers the specific case here.
+
+- [`mrcfile`](https://github.com/ccpem/mrcfile) (CCP-EM) is the standard MRC reader. Its lazy mode is a numpy `memmap`, which needs a local filesystem path — it does not issue HTTP range requests. `scigantic_empiar` parses the 1024-byte header directly (`parse_mrc_header`) to seek to one frame of a remote file without a local copy.
+- [`fsspec`](https://filesystem-spec.readthedocs.io/) `HTTPFileSystem` turns byte reads into HTTP range requests and can fetch many ranges concurrently ([`cat_ranges`](https://filesystem-spec.readthedocs.io/en/latest/async.html)). `pread` is a small equivalent, kept dependency-free and tuned to EBI's ~8-connection throttle; moving the transport onto `fsspec` is a reasonable later change.
+- [`copick`](https://github.com/copick/copick) (CZI, [Protein Science 2026](https://onlinelibrary.wiley.com/doi/10.1002/pro.70578)) is the closest cryo-EM analog: an fsspec-backed, server-less dataset API with lazy reads. It assumes data stored as OME-Zarr (chunked, multiscale). EMPIAR entries are raw MRC/TIFF, so copick needs a per-entry zarr conversion first — the conversion that MRC's flat layout lets `scigantic_empiar` skip.
+
+## Notes
+
+- MRC/MRCS (movies, micrographs, tomograms, particle stacks) and some TIFF. Files often nest a couple subdir levels down; `find_mrc` handles that.
+- Entry ids are opaque numbers — discover datasets by **metadata** (`EmpiarCatalog.search`, or the EMPIAR website), not by listing the tree.
+- Inside a [Scigantic](https://scigantic.com) cryo-EM notebook this is preinstalled and the archive is also FUSE-mounted at `$SCIGANTIC_MOUNT_PATH`; standalone, it streams straight from EBI.
+
+## License
+
+MIT.

scigantic_empiar-0.1.0/pyproject.toml

@@ -0,0 +1,32 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "scigantic-empiar"
+version = "0.1.0"
+description = "Explore the EMPIAR cryo-EM archive from Python — stream any of ~3,000 datasets (8.9 PiB) over parallel HTTP range reads, nothing downloaded."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+authors = [{ name = "Scigantic" }]
+keywords = ["cryo-em", "cryo-et", "empiar", "mrc", "structural-biology", "microscopy"]
+dependencies = [
+    "numpy",
+    "requests",
+]
+
+[project.optional-dependencies]
+# Needed for preview() (rendering) and EmpiarCatalog gallery/search.
+viz = ["matplotlib", "pandas", "pillow", "ipython"]
+dev = ["pytest", "matplotlib", "pandas"]
+
+[project.urls]
+Homepage = "https://github.com/scigantic/scigantic-empiar"
+"EMPIAR" = "https://www.ebi.ac.uk/empiar/"
+
+[tool.setuptools]
+packages = ["scigantic_empiar"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

scigantic_empiar-0.1.0/scigantic_empiar/__init__.py

@@ -0,0 +1,51 @@
+"""scigantic_empiar — explore EMPIAR cryo-EM data over HTTP, without downloading.
+
+    import scigantic_empiar as se
+    se.preview(10002)                      # micrograph + power spectrum, in seconds
+    se.EmpiarCatalog().search("ribosome")  # search the whole archive
+
+The package is split into focused modules; this file re-exports the public API so
+the flat ``se.<name>`` calls above keep working:
+
+    config     locations + shared HTTP session (env-overridable)
+    reader     parallel HTTP range reads (pread) + path/URL helpers
+    mrc        MRC/MRCS parsing, file discovery, NumPy readers
+    catalog    metadata client (EmpiarClient) + searchable catalog (EmpiarCatalog)
+    render     inline micrograph + power-spectrum rendering (needs matplotlib)
+    workspace  mirror an entry to fast storage for reprocessing
+"""
+from __future__ import annotations
+
+from .catalog import EmpiarCatalog, EmpiarClient
+from .config import API, CATALOG_URL, EBI, FAST_BUCKET, FAST_MNT, MOUNT
+from .mrc import (
+    find_mrc,
+    parse_mrc_header,
+    power_spectrum,
+    read_mrc,
+    read_mrc_average,
+    read_mrc_frame,
+    thumbnail,
+)
+from .reader import entry_url, fast_path, list_files, pread
+from .render import preview
+from .workspace import add_to_fast_workspace
+
+__version__ = "0.1.0"
+
+__all__ = [
+    # rendering
+    "preview",
+    # readers
+    "read_mrc", "read_mrc_frame", "read_mrc_average", "thumbnail",
+    "find_mrc", "parse_mrc_header", "power_spectrum",
+    # transport
+    "pread", "list_files", "entry_url", "fast_path",
+    # metadata / catalog
+    "EmpiarClient", "EmpiarCatalog",
+    # fast workspace
+    "add_to_fast_workspace",
+    # config constants
+    "MOUNT", "EBI", "API", "CATALOG_URL", "FAST_BUCKET", "FAST_MNT",
+    "__version__",
+]

scigantic_empiar-0.1.0/scigantic_empiar/catalog.py

@@ -0,0 +1,86 @@
+"""EMPIAR metadata client + a searchable, visual catalog across all entries."""
+from __future__ import annotations
+import functools
+import os
+
+from .config import API, CATALOG_URL, MOUNT, session
+
+
+class EmpiarClient:
+    """Per-entry metadata from EMPIAR's REST API (cached)."""
+
+    @functools.lru_cache(maxsize=4096)
+    def entry(self, entry_id):
+        eid = str(entry_id).replace("EMPIAR-", "")
+        r = session.get(f"{API}/{eid}/", timeout=30)
+        r.raise_for_status()
+        d = r.json()
+        e = d.get(f"EMPIAR-{eid}") or (list(d.values())[0] if d else {})
+        return e if isinstance(e, dict) else {}
+
+    def summary(self, entry_id):
+        e = self.entry(entry_id)
+        iss = e.get("imagesets") or [{}]
+        i0 = iss[0] if isinstance(iss[0], dict) else {}
+        return dict(
+            id=str(entry_id).replace("EMPIAR-", ""),
+            title=e.get("title", ""),
+            size=e.get("dataset_size", ""),
+            format=i0.get("data_format") or i0.get("header_format"),
+            category=i0.get("category"),
+            release_date=e.get("release_date"),
+            doi=e.get("entry_doi"),
+        )
+
+
+class EmpiarCatalog:
+    """Searchable, visual catalog across the whole archive.
+
+    Loads a prebuilt index (id, title, size, method, thumbnail per entry) so
+    search/filter over all ~3,000 entries is instant. Falls back to the mount
+    listing when no index is available.
+    """
+
+    def __init__(self, url=CATALOG_URL):
+        self.url = url
+        self._df = None
+
+    def load(self):
+        import pandas as pd
+        if self._df is not None:
+            return self._df
+        try:
+            self._df = pd.DataFrame(session.get(self.url, timeout=30).json())
+        except Exception:
+            ids = sorted(os.listdir(MOUNT)) if os.path.isdir(MOUNT) else []
+            self._df = pd.DataFrame({"id": ids})
+        return self._df
+
+    def search(self, query=None, method=None, max_gb=None, limit=50):
+        df = self.load().copy()
+        if query and "title" in df:
+            df = df[df["title"].str.contains(query, case=False, na=False)]
+        if method and "method" in df:
+            df = df[df["method"].str.contains(method, case=False, na=False)]
+        if max_gb and "size_gb" in df:
+            df = df[df["size_gb"].fillna(1e9) <= max_gb]
+        return df.head(limit)
+
+    def gallery(self, df=None, cols=4):
+        """Render a thumbnail gallery (HTML) for a set of entries."""
+        from IPython.display import HTML
+        df = self.load() if df is None else df
+        cells = []
+        for _, r in df.iterrows():
+            thumb = r.get("thumbnail_url") or ""
+            img = (
+                f'<img src="{thumb}" style="width:100%;border-radius:6px">' if thumb
+                else '<div style="height:120px;background:#eee;border-radius:6px"></div>'
+            )
+            cells.append(
+                f'<div style="width:{100 // cols - 2}%;display:inline-block;vertical-align:top;'
+                f'margin:1%;font:11px sans-serif">{img}'
+                f'<b>EMPIAR-{r.get("id", "")}</b><br>{str(r.get("title", ""))[:70]}'
+                f'<br><span style="color:#888">{r.get("size", "")}</span></div>'
+            )
+        return HTML("<div>" + "".join(cells) + "</div>")

scigantic_empiar-0.1.0/scigantic_empiar/config.py

@@ -0,0 +1,35 @@
+"""Locations + the shared HTTP session for scigantic_empiar.
+
+Everything is overridable via env vars so a Scigantic notebook (mounted at
+``$SCIGANTIC_MOUNT_PATH``, with its own catalog/fast buckets) and a standalone
+install behave sensibly without config.
+"""
+from __future__ import annotations
+import os
+import requests
+
+# The whole EMPIAR tree is FUSE-mounted here inside a Scigantic notebook. When
+# absent (standalone use), the readers stream straight from EBI over HTTPS.
+MOUNT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/http-archive/data")
+
+# EBI's public data endpoints.
+EBI = "https://ftp.ebi.ac.uk/empiar/world_availability"
+API = "https://www.ebi.ac.uk/empiar/api/entry"
+
+# Prebuilt per-entry metadata + thumbnail index (id, title, size, method,
+# thumbnail_url) that powers EmpiarCatalog search/gallery.
+CATALOG_URL = os.environ.get(
+    "SCIGANTIC_EMPIAR_CATALOG",
+    "https://scigantic-empiar-catalog.s3.amazonaws.com/catalog.json",
+)
+
+# S3 fast-workspace: full-speed mirrored copies of chosen entries.
+FAST_BUCKET = os.environ.get("SCIGANTIC_EMPIAR_FAST_BUCKET", "scigantic-empiar-fast")
+FAST_MNT = os.environ.get("SCIGANTIC_EMPIAR_FAST_MNT", "/mnt/empiar-fast")
+
+# Descriptive UA (contact) is the polite convention for automated EBI/NCBI
+# access and avoids some abuse filters.
+USER_AGENT = "scigantic-empiar/0.1 (+https://github.com/scigantic/scigantic-empiar; mailto:support@scigantic.com)"
+
+session = requests.Session()
+session.headers.update({"User-Agent": USER_AGENT})

scigantic_empiar-0.1.0/scigantic_empiar/mrc.py

@@ -0,0 +1,119 @@
+"""MRC/MRCS parsing, file discovery, and NumPy array readers (no plotting)."""
+from __future__ import annotations
+import os
+import struct
+
+import numpy as np
+
+from .reader import entry_url, fast_path, list_files, pread
+
+MRC_EXT = (".mrcs", ".mrc", ".st", ".ali", ".rec", ".mrc.bz2")
+# MRC mode -> numpy dtype (the common cryo-EM subset).
+MODE_DTYPE = {0: np.int8, 1: np.int16, 2: np.float32, 6: np.uint16, 12: np.float16}
+
+
+def parse_mrc_header(b: bytes) -> dict:
+    """Parse a 1024-byte MRC header. Returns dims, dtype, extended-header size,
+    pixel size (Angstrom), and the byte offset where image data begins."""
+    nx, ny, nz, mode = struct.unpack("<4i", b[:16])
+    nsymbt = struct.unpack("<i", b[92:96])[0]
+    mx, my, mz = struct.unpack("<3i", b[28:40])   # grid
+    xlen, ylen, zlen = struct.unpack("<3f", b[40:52])  # cell (A)
+    apix = (xlen / mx) if mx else 0.0
+    dt = np.dtype(MODE_DTYPE.get(mode, np.float32))
+    return dict(
+        nx=nx, ny=ny, nz=nz, mode=mode, nsymbt=nsymbt, dtype=dt,
+        apix=round(apix, 3), frame_bytes=nx * ny * dt.itemsize,
+        data0=1024 + nsymbt,
+    )
+
+
+def find_mrc(entry_id, subdir="data", depth=2):
+    """First MRC-like file under an entry — checks ``data/`` then up to two
+    subdir levels (tomo tilt-series / particle stacks / per-session dirs nest
+    their MRCs a couple levels down). Returns a subdir-relative path, or None."""
+    entries = list_files(entry_id, subdir)
+    hits = [f for f in entries if f.lower().endswith(MRC_EXT)]
+    if hits:
+        return f"{subdir}/{hits[0]}"
+    if depth > 0:
+        subs = [f.rstrip("/") for f in entries if "." not in f.rstrip("/")]  # dirs
+        for s in subs[:8]:
+            r = find_mrc(entry_id, f"{subdir}/{s}", depth - 1)
+            if r:
+                return r
+    return None
+
+
+def read_mrc(entry_id, filename=None, subdir="data"):
+    """Resolve an entry+file to ``(url, filename, header)``, reading only the
+    header. ``filename`` may be a path relative to the entry root; if omitted,
+    the first MRC found (recursing subdirs) is used."""
+    if filename is None:
+        rel = find_mrc(entry_id, subdir)
+        if not rel:
+            raise FileNotFoundError(f"no MRC files under EMPIAR-{entry_id}/{subdir}")
+        parts = rel.split("/")
+        subdir, filename = "/".join(parts[:-1]), parts[-1]
+    fp = fast_path(entry_id)
+    url = os.path.join(fp, subdir, filename) if fp else entry_url(entry_id, subdir, filename)
+    return url, filename, parse_mrc_header(pread(url, 0, 1024, 1))
+
+
+def read_mrc_frame(entry_id, filename=None, frame=0, nthreads=8):
+    """One 2D frame/slice as a float32 array (+ header)."""
+    url, fn, h = read_mrc(entry_id, filename)
+    off = h["data0"] + int(frame) * h["frame_bytes"]
+    buf = pread(url, off, h["frame_bytes"], nthreads)
+    arr = np.frombuffer(buf, dtype=h["dtype"]).astype(np.float32).reshape(h["ny"], h["nx"])
+    h["file"] = fn
+    return arr, h
+
+
+def read_mrc_average(entry_id, filename=None, n_frames=4, nthreads=8):
+    """Average the first ``n_frames`` — a poor-man's motion-corrected image
+    (much cleaner than one raw frame)."""
+    url, fn, h = read_mrc(entry_id, filename)
+    n = max(1, min(n_frames, h["nz"] or 1))
+    buf = pread(url, h["data0"], n * h["frame_bytes"], nthreads)
+    stack = np.frombuffer(buf, dtype=h["dtype"]).astype(np.float32).reshape(n, h["ny"], h["nx"])
+    h["file"] = fn
+    h["n_averaged"] = n
+    return stack.mean(0), h
+
+
+def thumbnail(entry_id, filename=None, size=320, nthreads=8):
+    """A small uint8 preview (central strip of the first MRC), reading only ~a
+    few MB — used to build catalogs. Returns (uint8 array, header)."""
+    url, fn, h = read_mrc(entry_id, filename)
+    rows = min(h["ny"], max(size, 256))
+    row0 = max(0, (h["ny"] - rows) // 2)
+    off = h["data0"] + row0 * h["nx"] * h["dtype"].itemsize
+    buf = pread(url, off, rows * h["nx"] * h["dtype"].itemsize, nthreads)
+    band = np.frombuffer(buf, dtype=h["dtype"]).astype(np.float32).reshape(rows, h["nx"])
+    f = max(1, max(band.shape) // size)
+    small = band[::f, ::f]
+    lo, hi = np.percentile(small, [2, 98])
+    small = np.clip((small - lo) / (hi - lo + 1e-9), 0, 1)
+    h["file"] = fn
+    return (small * 255).astype(np.uint8), h
+
+
+def _hann2d(shape):
+    return np.outer(np.hanning(shape[0]), np.hanning(shape[1]))
+
+
+def power_spectrum(arr, bin_to=1024):
+    """Windowed, log-scaled, [0,1]-normalised power spectrum (Thon rings)."""
+    f = max(1, min(arr.shape) // bin_to)
+    a = arr[::f, ::f].astype(np.float32)
+    a = (a - a.mean()) / (a.std() + 1e-6)
+    a = a * _hann2d(a.shape)
+    ps = np.log1p(np.abs(np.fft.fftshift(np.fft.fft2(a))))
+    lo, hi = np.percentile(ps, [1, 99.5])
+    return np.clip((ps - lo) / (hi - lo + 1e-9), 0, 1)
+
+
+def downsample(a, target=900):
+    f = max(1, min(a.shape) // target)
+    return a[::f, ::f]

scigantic_empiar-0.1.0/scigantic_empiar/reader.py

@@ -0,0 +1,74 @@
+"""The fast lane over EBI: parallel HTTP range reads, plus path/URL helpers.
+
+EBI serves ~1.5 MB/s per connection and throttles past ~8, so ``pread`` splits a
+read into up to 8 concurrent range requests, which aggregates to ~5-10 MB/s.
+"""
+from __future__ import annotations
+import math
+import os
+import re
+from concurrent.futures import ThreadPoolExecutor
+
+from .config import EBI, FAST_MNT, MOUNT, session
+
+
+def entry_url(entry_id, *parts) -> str:
+    """EBI HTTPS URL for an entry (optionally a file/dir under it)."""
+    eid = str(entry_id).replace("EMPIAR-", "").lstrip("0") or "0"
+    tail = "/".join(str(p).strip("/") for p in parts if p is not None)
+    return f"{EBI}/{eid}/" + (tail if tail else "")
+
+
+def fast_path(entry_id):
+    """Local path to a mirrored (fast) copy of an entry, or None if not mirrored."""
+    eid = str(entry_id).replace("EMPIAR-", "")
+    p = os.path.join(FAST_MNT, eid)
+    return p if os.path.isdir(p) else None
+
+
+def _get_range(url, start, end, retries=2) -> bytes:
+    last = None
+    for attempt in range(retries + 1):
+        try:
+            r = session.get(url, headers={"Range": f"bytes={start}-{end}"}, timeout=60)
+            r.raise_for_status()
+            return r.content
+        except Exception as exc:  # noqa: BLE001 - retry any transient error
+            last = exc
+    raise last  # type: ignore[misc]
+
+
+def pread(url, offset, length, nthreads=8) -> bytes:
+    """Read ``length`` bytes at ``offset`` from ``url`` via up to ``nthreads``
+    parallel range GETs. A local/``file://`` path is read directly (used when an
+    entry is mirrored to the fast workspace)."""
+    if url.startswith("/") or url.startswith("file:"):
+        with open(url.replace("file://", ""), "rb") as fh:
+            fh.seek(offset)
+            return fh.read(length)
+    if length <= 0:
+        return b""
+    n = max(1, min(nthreads, math.ceil(length / (1 << 20))))
+    step = math.ceil(length / n)
+    spans, o = [], offset
+    while o < offset + length:
+        end = min(o + step, offset + length) - 1
+        spans.append((o, end))
+        o = end + 1
+    if len(spans) == 1:
+        return _get_range(url, *spans[0])
+    with ThreadPoolExecutor(max_workers=len(spans)) as ex:
+        parts = list(ex.map(lambda s: _get_range(url, s[0], s[1]), spans))
+    return b"".join(parts)
+
+
+def list_files(entry_id, subdir="data"):
+    """Filenames under an entry's subdir — from the mount if present, else by
+    parsing EBI's autoindex HTML."""
+    eid = str(entry_id).replace("EMPIAR-", "")
+    local = os.path.join(MOUNT, eid, subdir)
+    if os.path.isdir(local):
+        return sorted(os.listdir(local))
+    html = session.get(entry_url(eid, subdir) + "/", timeout=30).text
+    out = [m for m in re.findall(r'href="([^"?/][^"]*)"', html) if not m.startswith("..")]
+    return sorted(set(out))

scigantic_empiar-0.1.0/scigantic_empiar/render.py

@@ -0,0 +1,58 @@
+"""Inline visualization — micrograph/slice + power spectrum (needs matplotlib).
+
+Named ``render`` (not ``preview``) to avoid colliding with the ``preview()``
+function it exports, which the package surfaces as ``scigantic_empiar.preview``.
+"""
+from __future__ import annotations
+
+import numpy as np
+
+from .catalog import EmpiarClient
+from .mrc import downsample, power_spectrum, read_mrc_average, read_mrc_frame
+
+
+def preview(entry_id, filename=None, average=False, n_frames=4, cmap="gray",
+            apix=None, nthreads=8, figsize=(11, 5.3)):
+    """Render a micrograph (or tomogram slice) + its power spectrum inline.
+
+    Reads only what it needs over parallel range requests — a single frame is a
+    few seconds even on a multi-hundred-GB entry, nothing downloaded. Never
+    raises: on any read error it prints the entry's metadata + how to drill in.
+    Pass ``average=True`` for a cleaner (heavier) mean-of-frames image.
+    """
+    import matplotlib.pyplot as plt
+
+    eid = str(entry_id).replace("EMPIAR-", "")
+    try:
+        if average:
+            img, h = read_mrc_average(entry_id, filename, n_frames, nthreads)
+            sub = f"mean of {h.get('n_averaged', 1)} frames"
+        else:
+            img, h = read_mrc_frame(entry_id, filename, 0, nthreads)
+            sub = "frame 0"
+    except Exception as e:  # noqa: BLE001 - never crash the caller/notebook
+        try:
+            s = EmpiarClient().summary(eid)
+            print(f"EMPIAR-{eid}: {s.get('title', '')}  ({s.get('size', '?')}, {s.get('format', '?')})")
+        except Exception:
+            pass
+        print(f"Couldn't auto-locate an MRC to preview ({e}).")
+        print(f"Explore the layout:  list_files({eid})  then  list_files({eid}, 'data/<subdir>')")
+        print(f"Preview a specific file:  preview({eid}, filename='<subdir>/<file>.mrc')")
+        return None
+
+    if not apix:
+        apix = h["apix"] if h["apix"] else None
+    px = f", {apix} Å/px" if apix else ""
+    disp = downsample(img, 900)
+    lo, hi = np.percentile(disp, [2, 98])
+
+    fig, ax = plt.subplots(1, 2, figsize=figsize)
+    ax[0].imshow(np.clip(disp, lo, hi), cmap=cmap)
+    ax[0].set_title(f"EMPIAR-{eid} · {h['file']}\n{h['nx']}×{h['ny']}{px} · {sub}", fontsize=9)
+    ax[0].axis("off")
+    ax[1].imshow(power_spectrum(img), cmap="magma")
+    ax[1].set_title("power spectrum (FFT) — Thon rings", fontsize=9)
+    ax[1].axis("off")
+    fig.tight_layout()
+    return fig

scigantic_empiar-0.1.0/scigantic_empiar/workspace.py

@@ -0,0 +1,39 @@
+"""Tier 2: mirror a whole entry to fast storage for heavy reprocessing.
+
+Streaming a full multi-hundred-GB entry from EBI at ~1.5 MB/s isn't practical,
+so for RELION/EMAN2-style work you copy it once (via ``rclone``, which does its
+own multi-connection transfer) to an S3 fast bucket, then read at local speed.
+"""
+from __future__ import annotations
+import shutil
+import subprocess
+
+from .config import EBI, FAST_BUCKET, FAST_MNT
+from .reader import entry_url, fast_path
+
+
+def add_to_fast_workspace(entry_id, bucket=FAST_BUCKET, dry_run=False):
+    """Mirror EMPIAR-<id> from EBI into the S3 fast bucket with ``rclone``.
+
+    Returns the ``s3://`` destination. If the entry is already visible under the
+    local fast mount, returns its path without recopying. ``dry_run=True`` returns
+    the command it would run (a list) instead of executing.
+    """
+    eid = str(entry_id).replace("EMPIAR-", "")
+    existing = fast_path(eid)
+    if existing:
+        return existing
+
+    src = entry_url(eid).replace("https://", ":http:")  # rclone :http: backend
+    dst = f":s3:{bucket}/{eid}"
+    cmd = ["rclone", "copy", "--http-url", EBI.rsplit("/", 1)[0],
+           "--transfers", "8", "--checkers", "8", src, dst]
+    if dry_run:
+        return cmd
+    if shutil.which("rclone") is None:
+        raise RuntimeError(
+            "rclone not found. Inside a Scigantic notebook this runs server-side; "
+            "standalone, install rclone and configure an :s3: remote."
+        )
+    subprocess.run(cmd, check=True)
+    return f"s3://{bucket}/{eid}  (mount at {FAST_MNT}/{eid} to read locally)"

scigantic_empiar-0.1.0/scigantic_empiar.egg-info/PKG-INFO

@@ -0,0 +1,85 @@
+Metadata-Version: 2.4
+Name: scigantic-empiar
+Version: 0.1.0
+Summary: Explore the EMPIAR cryo-EM archive from Python — stream any of ~3,000 datasets (8.9 PiB) over parallel HTTP range reads, nothing downloaded.
+Author: Scigantic
+License: MIT
+Project-URL: Homepage, https://github.com/scigantic/scigantic-empiar
+Project-URL: EMPIAR, https://www.ebi.ac.uk/empiar/
+Keywords: cryo-em,cryo-et,empiar,mrc,structural-biology,microscopy
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: requests
+Provides-Extra: viz
+Requires-Dist: matplotlib; extra == "viz"
+Requires-Dist: pandas; extra == "viz"
+Requires-Dist: pillow; extra == "viz"
+Requires-Dist: ipython; extra == "viz"
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: matplotlib; extra == "dev"
+Requires-Dist: pandas; extra == "dev"
+Dynamic: license-file
+
+# scigantic_empiar
+
+Explore [EMPIAR](https://www.ebi.ac.uk/empiar/) — EMBL-EBI's public archive of **raw cryo-EM / cryo-ET image data** (~3,000 datasets, ~8.9 PiB) — from Python, **without downloading anything**.
+
+EMPIAR is served over EBI's public HTTPS at ~1.5 MB/s per connection. `scigantic_empiar` parallelises HTTP **range** reads (8-way ≈ 5–10 MB/s) so you can pull a single frame from a many-GB entry in seconds, decode the MRC, and render the micrograph + its power spectrum — nothing is copied to disk.
+
+```python
+import scigantic_empiar as se
+
+se.preview(10406)                      # render the micrograph below, in seconds
+se.EmpiarClient().summary(10406)       # title, pixel size, method, DOI, EMDB/PDB cross-refs
+se.EmpiarCatalog().search("ribosome")  # search the whole archive by metadata (instant)
+```
+
+![A motion-corrected 70S-ribosome micrograph (EMPIAR-10406) and its power spectrum, rendered by se.preview(10406)](https://raw.githubusercontent.com/scigantic/scigantic-empiar/main/docs/preview_10406.png)
+
+*One frame of EMPIAR-10406 (a 70S-ribosome dataset) pulled straight from EBI over parallel range reads — the carbon-foil edge, ice, and particles are visible at left; the FFT is at right. Nothing was downloaded to disk.*
+
+## Install
+
+```bash
+pip install "scigantic-empiar[viz] @ git+https://github.com/scigantic/scigantic-empiar"
+```
+
+Core (`numpy`, `requests`) is enough for the readers; `[viz]` adds `matplotlib` / `pandas` / `pillow` for `preview()` and the catalog gallery.
+
+## What it does
+
+| | |
+|---|---|
+| `preview(id)` | micrograph / tomogram-slice + power spectrum, rendered from a lazy parallel-range read |
+| `read_mrc_frame(id)` / `read_mrc_average(id)` | one frame / a mean of frames as a NumPy array + header |
+| `thumbnail(id)` | small preview array (a few-MB central-strip read) — used to build catalogs |
+| `find_mrc(id)` | resolve an entry's first MRC, recursing the (often nested) `data/` layout |
+| `pread(url, off, len)` | the 8-way parallel HTTP range reader under it all |
+| `EmpiarClient` | per-entry metadata from EMPIAR's REST API (cached) |
+| `EmpiarCatalog` | search + a visual thumbnail gallery across all entries (from a prebuilt index) |
+| `add_to_fast_workspace(id)` | mirror an entry to S3 for full-speed reprocessing (RELION/EMAN2) |
+
+## Why parallel range reads
+
+EBI throttles per connection (~1.5 MB/s) and past ~8 concurrent connections. `pread` splits a read into ~8 concurrent range requests, which aggregates to ~5–10 MB/s — enough to *look* at any entry interactively. For heavy reprocessing of a whole multi-hundred-GB dataset, mirror it to fast storage first (`add_to_fast_workspace`); streaming a full entry at 1.5 MB/s isn't practical.
+
+## Existing work
+
+The job splits in two: parse MRC, and read bytes from a remote file. Both have existing libraries; neither covers the specific case here.
+
+- [`mrcfile`](https://github.com/ccpem/mrcfile) (CCP-EM) is the standard MRC reader. Its lazy mode is a numpy `memmap`, which needs a local filesystem path — it does not issue HTTP range requests. `scigantic_empiar` parses the 1024-byte header directly (`parse_mrc_header`) to seek to one frame of a remote file without a local copy.
+- [`fsspec`](https://filesystem-spec.readthedocs.io/) `HTTPFileSystem` turns byte reads into HTTP range requests and can fetch many ranges concurrently ([`cat_ranges`](https://filesystem-spec.readthedocs.io/en/latest/async.html)). `pread` is a small equivalent, kept dependency-free and tuned to EBI's ~8-connection throttle; moving the transport onto `fsspec` is a reasonable later change.
+- [`copick`](https://github.com/copick/copick) (CZI, [Protein Science 2026](https://onlinelibrary.wiley.com/doi/10.1002/pro.70578)) is the closest cryo-EM analog: an fsspec-backed, server-less dataset API with lazy reads. It assumes data stored as OME-Zarr (chunked, multiscale). EMPIAR entries are raw MRC/TIFF, so copick needs a per-entry zarr conversion first — the conversion that MRC's flat layout lets `scigantic_empiar` skip.
+
+## Notes
+
+- MRC/MRCS (movies, micrographs, tomograms, particle stacks) and some TIFF. Files often nest a couple subdir levels down; `find_mrc` handles that.
+- Entry ids are opaque numbers — discover datasets by **metadata** (`EmpiarCatalog.search`, or the EMPIAR website), not by listing the tree.
+- Inside a [Scigantic](https://scigantic.com) cryo-EM notebook this is preinstalled and the archive is also FUSE-mounted at `$SCIGANTIC_MOUNT_PATH`; standalone, it streams straight from EBI.
+
+## License
+
+MIT.

scigantic_empiar-0.1.0/scigantic_empiar.egg-info/SOURCES.txt

@@ -0,0 +1,19 @@
+LICENSE
+README.md
+pyproject.toml
+scigantic_empiar/__init__.py
+scigantic_empiar/catalog.py
+scigantic_empiar/config.py
+scigantic_empiar/mrc.py
+scigantic_empiar/reader.py
+scigantic_empiar/render.py
+scigantic_empiar/workspace.py
+scigantic_empiar.egg-info/PKG-INFO
+scigantic_empiar.egg-info/SOURCES.txt
+scigantic_empiar.egg-info/dependency_links.txt
+scigantic_empiar.egg-info/requires.txt
+scigantic_empiar.egg-info/top_level.txt
+tests/test_catalog.py
+tests/test_mrc.py
+tests/test_preview.py
+tests/test_reader.py

scigantic_empiar-0.1.0/scigantic_empiar.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

scigantic_empiar-0.1.0/scigantic_empiar.egg-info/requires.txt

@@ -0,0 +1,13 @@
+numpy
+requests
+
+[dev]
+pytest
+matplotlib
+pandas
+
+[viz]
+matplotlib
+pandas
+pillow
+ipython

scigantic_empiar-0.1.0/scigantic_empiar.egg-info/top_level.txt

@@ -0,0 +1 @@
+scigantic_empiar

scigantic_empiar-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

scigantic_empiar-0.1.0/tests/test_catalog.py

@@ -0,0 +1,49 @@
+"""catalog.py — metadata client + searchable catalog (no network)."""
+import pandas as pd
+
+import scigantic_empiar as se
+from scigantic_empiar.catalog import EmpiarClient
+
+
+def _catalog():
+    cat = se.EmpiarCatalog()
+    cat._df = pd.DataFrame([
+        {"id": "10002", "title": "80S ribosome", "method": "SPA", "size_gb": 260},
+        {"id": "10406", "title": "HIV-1 tomogram", "method": "tomography", "size_gb": 40},
+        {"id": "11000", "title": "ribosome subunit", "method": "SPA", "size_gb": 5},
+    ])
+    return cat
+
+
+def test_search_by_title():
+    hits = _catalog().search(query="ribosome")
+    assert set(hits["id"]) == {"10002", "11000"}
+
+
+def test_search_by_method():
+    hits = _catalog().search(method="tomography")
+    assert list(hits["id"]) == ["10406"]
+
+
+def test_search_by_size():
+    hits = _catalog().search(max_gb=50)
+    assert set(hits["id"]) == {"10406", "11000"}          # excludes the 260 GB entry
+
+
+def test_search_limit():
+    assert len(_catalog().search(limit=1)) == 1
+
+
+def test_client_summary_shape(monkeypatch):
+    fake = {
+        "title": "80S ribosome",
+        "dataset_size": "260 GB",
+        "release_date": "2016-01-01",
+        "entry_doi": "10.6019/EMPIAR-10002",
+        "imagesets": [{"data_format": "MRC", "category": "micrographs"}],
+    }
+    monkeypatch.setattr(EmpiarClient, "entry", lambda self, eid: fake)
+    s = EmpiarClient().summary("EMPIAR-10002")
+    assert s["id"] == "10002"                              # prefix stripped
+    assert s["format"] == "MRC"
+    assert s["title"] == "80S ribosome"

scigantic_empiar-0.1.0/tests/test_mrc.py

@@ -0,0 +1,64 @@
+"""mrc.py — header parsing, file discovery, and NumPy readers."""
+import numpy as np
+
+from conftest import make_mrc_header
+
+import scigantic_empiar as se
+from scigantic_empiar import mrc
+
+
+def test_parse_mrc_header():
+    h = se.parse_mrc_header(make_mrc_header(nx=4, ny=3, nz=5, mode=2, mx=200, xlen=210.0))
+    assert (h["nx"], h["ny"], h["nz"], h["mode"]) == (4, 3, 5, 2)
+    assert h["dtype"] == np.float32
+    assert h["apix"] == 1.05
+    assert h["frame_bytes"] == 4 * 3 * 4          # nx*ny*itemsize
+    assert h["data0"] == 1024                     # nsymbt == 0
+
+
+def test_parse_mrc_header_extended_offset():
+    h = se.parse_mrc_header(make_mrc_header(mode=1, nsymbt=512))
+    assert h["dtype"] == np.int16                 # mode 1
+    assert h["data0"] == 1024 + 512               # extended header shifts data start
+
+
+def test_find_mrc_recurses_subdirs(monkeypatch):
+    listings = {
+        "data": ["notes.txt", "micrographs/"],           # no MRC at top level
+        "data/micrographs": ["log.txt", "stack_0001.mrc"],  # found one level down
+    }
+    monkeypatch.setattr(mrc, "list_files", lambda eid, subdir="data": listings[subdir])
+    assert se.find_mrc(10406) == "data/micrographs/stack_0001.mrc"
+
+
+def test_find_mrc_returns_none_when_absent(monkeypatch):
+    monkeypatch.setattr(mrc, "list_files", lambda eid, subdir="data": ["readme.txt"])
+    assert se.find_mrc(1) is None
+
+
+def test_read_mrc_frame_shapes_and_values(monkeypatch):
+    header = make_mrc_header(nx=4, ny=3, nz=5, mode=2)
+
+    def fake_pread(url, off, length, nthreads=8):
+        if length == 1024:
+            return header                                   # header read
+        return np.arange(length // 4, dtype=np.float32).tobytes()  # frame read
+
+    monkeypatch.setattr(mrc, "pread", fake_pread)
+    monkeypatch.setattr(mrc, "fast_path", lambda eid: None)
+
+    arr, h = se.read_mrc_frame(10002, filename="m.mrc", frame=0)
+    assert arr.shape == (3, 4)                               # (ny, nx)
+    assert arr.dtype == np.float32
+    assert np.allclose(arr.ravel(), np.arange(12))
+    assert h["file"] == "m.mrc"
+
+
+def test_power_spectrum_and_downsample():
+    g = np.outer(np.sin(np.linspace(0, 12, 128)), np.sin(np.linspace(0, 9, 128)))
+    ps = se.power_spectrum(g.astype(np.float32))
+    assert ps.shape == (128, 128)
+    assert ps.min() >= 0.0 and ps.max() <= 1.0              # normalised to [0,1]
+
+    ds = mrc.downsample(g, target=64)
+    assert ds.shape == (64, 64)                             # stride-2 decimation

scigantic_empiar-0.1.0/tests/test_preview.py

@@ -0,0 +1,35 @@
+"""render.py — rendering, and the promise that preview() never raises."""
+import matplotlib
+
+matplotlib.use("Agg")  # headless
+import numpy as np
+
+import scigantic_empiar as se
+from scigantic_empiar import render
+
+
+def test_preview_renders_two_panels(monkeypatch):
+    img = np.random.default_rng(0).standard_normal((256, 256)).astype(np.float32)
+    header = {"nx": 256, "ny": 256, "apix": 1.05, "file": "m.mrc"}
+    monkeypatch.setattr(render, "read_mrc_frame", lambda *a, **k: (img, header))
+
+    fig = se.preview(10002, filename="m.mrc")
+    assert fig is not None
+    assert len(fig.axes) == 2          # micrograph + power spectrum
+
+
+def test_preview_never_raises_on_read_error(monkeypatch, capsys):
+    def boom(*a, **k):
+        raise OSError("404 from EBI")
+
+    monkeypatch.setattr(render, "read_mrc_frame", boom)
+    # metadata lookup also unavailable -> still must not raise
+    monkeypatch.setattr(
+        "scigantic_empiar.catalog.EmpiarClient.summary",
+        lambda self, eid: (_ for _ in ()).throw(RuntimeError("no api")),
+    )
+
+    result = se.preview(99999)
+    assert result is None                          # graceful, returns None
+    out = capsys.readouterr().out
+    assert "list_files" in out                     # prints how to drill in

scigantic_empiar-0.1.0/tests/test_reader.py

@@ -0,0 +1,68 @@
+"""reader.py — URL/path helpers and the parallel range reader."""
+import numpy as np
+import pytest
+
+from conftest import BUFFER
+
+import scigantic_empiar as se
+from scigantic_empiar import reader
+
+
+def test_entry_url_normalises_ids():
+    assert reader.entry_url(10002).endswith("/10002/")
+    # EMPIAR- prefix stripped, leading zeros trimmed
+    assert reader.entry_url("EMPIAR-00123").endswith("/123/")
+    # extra parts joined, no trailing slash
+    assert reader.entry_url(10002, "data", "m.mrc").endswith("/10002/data/m.mrc")
+
+
+def test_fast_path(tmp_path, monkeypatch):
+    monkeypatch.setattr(reader, "FAST_MNT", str(tmp_path))
+    (tmp_path / "10002").mkdir()
+    assert reader.fast_path("10002") == str(tmp_path / "10002")
+    assert reader.fast_path("99999") is None
+
+
+def test_pread_local_file(tmp_path):
+    p = tmp_path / "blob.bin"
+    p.write_bytes(BUFFER[:10000])
+    got = se.pread(str(p), 100, 250)          # local path branch (no threads)
+    assert got == BUFFER[100:350]
+
+
+def test_pread_splits_into_parallel_ranges(monkeypatch):
+    calls = []
+
+    def fake_get_range(url, start, end, retries=2):
+        calls.append((start, end))
+        return BUFFER[start:end + 1]
+
+    monkeypatch.setattr(reader, "_get_range", fake_get_range)
+
+    offset, length = 500, 2_500_000        # > 2 MB -> must fan out
+    got = se.pread("http://ebi.example/x.mrc", offset, length, nthreads=8)
+
+    assert got == BUFFER[offset:offset + length]     # correct + correctly ordered
+    assert len(calls) > 1                             # actually parallelised
+    spans = sorted(calls)
+    assert spans[0][0] == offset                      # covers the whole request...
+    assert spans[-1][1] == offset + length - 1
+    for (_, e), (s2, _) in zip(spans, spans[1:]):     # ...contiguously, no gaps/overlap
+        assert s2 == e + 1
+
+
+def test_pread_zero_length():
+    assert se.pread("http://x/y", 0, 0) == b""
+
+
+def test_get_range_retries_then_raises(monkeypatch):
+    attempts = {"n": 0}
+
+    def always_fail(*a, **k):
+        attempts["n"] += 1
+        raise RuntimeError("boom")
+
+    monkeypatch.setattr(reader.session, "get", always_fail)
+    with pytest.raises(RuntimeError):
+        reader._get_range("http://x/y", 0, 10, retries=2)
+    assert attempts["n"] == 3          # initial try + 2 retries