scigantic-facebase 0.1.0__tar.gz

scigantic_facebase-0.1.0/LICENSE

@@ -0,0 +1,16 @@
+MIT No Attribution
+
+Copyright 2026 Scigantic
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this
+software and associated documentation files (the "Software"), to deal in the Software
+without restriction, including without limitation the rights to use, copy, modify,
+merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
+PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
+CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE
+OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

scigantic_facebase-0.1.0/PKG-INFO

@@ -0,0 +1,88 @@
+Metadata-Version: 2.4
+Name: scigantic-facebase
+Version: 0.1.0
+Summary: Search FaceBase, the craniofacial research data hub, and read its open-access micro-CT, imaging and sequencing data from Python: a typed catalog over FaceBase's open metadata API, citations in the form its Terms of Use ask for, ranged reads of remote files, and single-slice reads from zipped micro-CT stacks without downloading them.
+Author: Scigantic
+License: MIT-0
+Project-URL: Homepage, https://scigantic.com
+Project-URL: Repository, https://github.com/Scigantic/scigantic-facebase
+Project-URL: Issues, https://github.com/Scigantic/scigantic-facebase/issues
+Project-URL: FaceBase, https://www.facebase.org
+Keywords: facebase,craniofacial,micro-ct,zebrafish,deriva,ermrest,bioinformatics
+Classifier: License :: OSI Approved :: MIT No Attribution License (MIT-0)
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Typing :: Typed
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Classifier: Topic :: Scientific/Engineering :: Image Processing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests<3,>=2.28
+Requires-Dist: numpy<3,>=1.24
+Requires-Dist: tifffile<2027,>=2023.7
+Requires-Dist: imagecodecs>=2023.9
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: mypy>=1.10; extra == "dev"
+Requires-Dist: types-requests; extra == "dev"
+Dynamic: license-file
+
+# scigantic-facebase
+
+Search [FaceBase](https://www.facebase.org), the craniofacial research data hub, and read its open-access data from Python.
+
+FaceBase holds about 1,070 open datasets: micro-CT and light-sheet imaging, ChIP-seq and RNA-seq tracks, expression tables and more, from mouse, zebrafish, human and other models. This package is a typed layer over FaceBase's open metadata API and file store. No key or login is needed.
+
+```
+pip install scigantic-facebase
+```
+
+## Use
+
+```python
+import scigantic_facebase as fb
+
+hits = fb.search("zebrafish thyroid microCT")     # newest first
+ds = hits[0]
+print(ds.title, ds.doi_url)
+print(fb.cite(ds))
+
+files = fb.files(ds)                               # size, MD5, download URL
+fb.download(files[0], "data/")                     # MD5-checked
+```
+
+Search matches every word against the whole record, and treats "zebrafish" and "Danio rerio", "mouse" and "Mus musculus" as the same thing.
+
+## Read a micro-CT scan without downloading it
+
+Many micro-CT datasets are a zip of TIFF slices. A zip keeps its directory at the end, so one slice costs one HTTP range request.
+
+```python
+stack = fb.microct.open_stack(files[0])
+len(stack)                  # 596 slices
+stack.voxel_size_um         # 10.5, from the scan log
+img = stack.read_slice(300)
+vol = stack.volume(step=10) # every 10th slice, (z, y, x)
+stack.bytes_fetched         # a few MB of a 49 MB file
+```
+
+`fb.microct.thyroid_atlas()` lists the zebrafish thyroid-hormone atlas: 81 head scans from 12 to 25 mm standard length, raised hypothyroid, euthyroid or hyperthyroid.
+
+`fb.open_remote(file)` gives any FaceBase file as a seekable file object backed by range requests, which `zipfile`, `tifffile` and `h5py` accept.
+
+## What is not here
+
+Protected human-subjects datasets sit behind FaceBase's data access request. Their metadata is public and `fb.search(include_protected=True)` lists them, but reading their files raises `FacebaseAccessError`.
+
+## Terms of use
+
+Copyright to FaceBase data belongs to the contributing investigators. FaceBase's [Terms of Use](https://www.facebase.org/policies/tou/) ask users to cite the dataset, acknowledge FaceBase, and send other people to FaceBase instead of circulating downloaded files. `fb.cite()` produces the citation. This package fetches from FaceBase on request and does not mirror anything.
+
+## License
+
+MIT-0 for this code. The data carries its own terms.

scigantic_facebase-0.1.0/README.md

@@ -0,0 +1,54 @@
+# scigantic-facebase
+
+Search [FaceBase](https://www.facebase.org), the craniofacial research data hub, and read its open-access data from Python.
+
+FaceBase holds about 1,070 open datasets: micro-CT and light-sheet imaging, ChIP-seq and RNA-seq tracks, expression tables and more, from mouse, zebrafish, human and other models. This package is a typed layer over FaceBase's open metadata API and file store. No key or login is needed.
+
+```
+pip install scigantic-facebase
+```
+
+## Use
+
+```python
+import scigantic_facebase as fb
+
+hits = fb.search("zebrafish thyroid microCT")     # newest first
+ds = hits[0]
+print(ds.title, ds.doi_url)
+print(fb.cite(ds))
+
+files = fb.files(ds)                               # size, MD5, download URL
+fb.download(files[0], "data/")                     # MD5-checked
+```
+
+Search matches every word against the whole record, and treats "zebrafish" and "Danio rerio", "mouse" and "Mus musculus" as the same thing.
+
+## Read a micro-CT scan without downloading it
+
+Many micro-CT datasets are a zip of TIFF slices. A zip keeps its directory at the end, so one slice costs one HTTP range request.
+
+```python
+stack = fb.microct.open_stack(files[0])
+len(stack)                  # 596 slices
+stack.voxel_size_um         # 10.5, from the scan log
+img = stack.read_slice(300)
+vol = stack.volume(step=10) # every 10th slice, (z, y, x)
+stack.bytes_fetched         # a few MB of a 49 MB file
+```
+
+`fb.microct.thyroid_atlas()` lists the zebrafish thyroid-hormone atlas: 81 head scans from 12 to 25 mm standard length, raised hypothyroid, euthyroid or hyperthyroid.
+
+`fb.open_remote(file)` gives any FaceBase file as a seekable file object backed by range requests, which `zipfile`, `tifffile` and `h5py` accept.
+
+## What is not here
+
+Protected human-subjects datasets sit behind FaceBase's data access request. Their metadata is public and `fb.search(include_protected=True)` lists them, but reading their files raises `FacebaseAccessError`.
+
+## Terms of use
+
+Copyright to FaceBase data belongs to the contributing investigators. FaceBase's [Terms of Use](https://www.facebase.org/policies/tou/) ask users to cite the dataset, acknowledge FaceBase, and send other people to FaceBase instead of circulating downloaded files. `fb.cite()` produces the citation. This package fetches from FaceBase on request and does not mirror anything.
+
+## License
+
+MIT-0 for this code. The data carries its own terms.

scigantic_facebase-0.1.0/pyproject.toml

@@ -0,0 +1,58 @@
+[build-system]
+requires = ["setuptools>=61", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "scigantic-facebase"
+version = "0.1.0"
+description = "Search FaceBase, the craniofacial research data hub, and read its open-access micro-CT, imaging and sequencing data from Python: a typed catalog over FaceBase's open metadata API, citations in the form its Terms of Use ask for, ranged reads of remote files, and single-slice reads from zipped micro-CT stacks without downloading them."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT-0" }
+authors = [{ name = "Scigantic" }]
+keywords = ["facebase", "craniofacial", "micro-ct", "zebrafish", "deriva", "ermrest", "bioinformatics"]
+classifiers = [
+  "License :: OSI Approved :: MIT No Attribution License (MIT-0)",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: 3.14",
+  "Typing :: Typed",
+  "Topic :: Scientific/Engineering :: Bio-Informatics",
+  "Topic :: Scientific/Engineering :: Image Processing",
+]
+
+dependencies = [
+  "requests>=2.28,<3",
+  "numpy>=1.24,<3",
+  "tifffile>=2023.7,<2027",
+  "imagecodecs>=2023.9",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=7", "mypy>=1.10", "types-requests"]
+
+[project.urls]
+Homepage = "https://scigantic.com"
+Repository = "https://github.com/Scigantic/scigantic-facebase"
+Issues = "https://github.com/Scigantic/scigantic-facebase/issues"
+FaceBase = "https://www.facebase.org"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+scigantic_facebase = ["py.typed"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = ["live: hits facebase.org"]
+
+[tool.mypy]
+strict = true
+
+[[tool.mypy.overrides]]
+module = ["tifffile", "tifffile.*"]
+ignore_missing_imports = true

scigantic_facebase-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

scigantic_facebase-0.1.0/src/scigantic_facebase/__init__.py

@@ -0,0 +1,42 @@
+"""Search FaceBase and read its open-access craniofacial data from Python.
+
+    import scigantic_facebase as fb
+
+    hits = fb.search("zebrafish thyroid microCT")
+    fb.cite(hits[0])
+    stack = fb.microct.open_stack(fb.files(hits[0])[0])
+    stack.read_slice(300)
+
+Metadata comes from FaceBase's open ERMrest API and files from its open-access
+file store, both anonymous. Copyright for the data stays with the contributing
+investigators and FaceBase's Terms of Use apply: cite the dataset DOI
+(`cite()`), and send other people to FaceBase rather than redistributing files.
+"""
+
+from . import microct
+from ._client import FacebaseAccessError, FacebaseError, FacebaseNotFoundError
+from ._version import __version__
+from .catalog import cite, contributors, count, files, get, project, search
+from .models import Dataset, File, Project
+from .remote import RangeFile, download, open_remote
+
+__all__ = [
+    "Dataset",
+    "FacebaseAccessError",
+    "FacebaseError",
+    "FacebaseNotFoundError",
+    "File",
+    "Project",
+    "RangeFile",
+    "__version__",
+    "cite",
+    "contributors",
+    "count",
+    "download",
+    "files",
+    "get",
+    "microct",
+    "open_remote",
+    "project",
+    "search",
+]

scigantic_facebase-0.1.0/src/scigantic_facebase/_client.py

@@ -0,0 +1,107 @@
+"""Shared HTTP plumbing for FaceBase: one lazily-built requests.Session,
+retry with backoff on transient failures, and the two FaceBase endpoints this
+package talks to.
+
+FaceBase publishes no rate limit. Its metadata (ERMrest) and open-access file
+store (Hatrac) answer anonymous requests, so there is no key or login here.
+Retries cover 429/5xx and connection errors only. A 404 is a real answer and
+raises FacebaseNotFoundError at once. A 401 or 403 means the data is behind
+FaceBase's data access request process and raises FacebaseAccessError.
+"""
+
+from __future__ import annotations
+
+import threading
+import time
+from typing import Any
+
+import requests
+
+from ._version import __version__
+
+SITE = "https://www.facebase.org"
+ERMREST = f"{SITE}/ermrest/catalog/1"
+
+_USER_AGENT = f"scigantic-facebase/{__version__} (+https://scigantic.com; mailto:support@scigantic.com)"
+
+_MAX_RETRIES = 4
+_RETRY_STATUS_CODES = {429, 500, 502, 503, 504}
+
+_session: requests.Session | None = None
+_session_lock = threading.Lock()
+
+
+class FacebaseError(Exception):
+    """An HTTP error after retries were exhausted, or a malformed response."""
+
+
+class FacebaseNotFoundError(FacebaseError):
+    """A dataset, file or path that does not exist (HTTP 404)."""
+
+
+class FacebaseAccessError(FacebaseError):
+    """The data needs authentication (HTTP 401 or 403). FaceBase holds
+    protected human-subjects data behind a data access request, and this
+    package only reads the open-access part."""
+
+
+def get_session() -> requests.Session:
+    global _session
+    if _session is None:
+        with _session_lock:
+            if _session is None:
+                _session = requests.Session()
+                _session.headers["User-Agent"] = _USER_AGENT
+    return _session
+
+
+def send(
+    method: str,
+    url: str,
+    params: dict[str, Any] | None = None,
+    headers: dict[str, str] | None = None,
+    stream: bool = False,
+    timeout: float = 60.0,
+) -> requests.Response:
+    """Issue one request with retry on 429/5xx and connection errors."""
+    session = get_session()
+    last_error: Exception | None = None
+    for attempt in range(_MAX_RETRIES + 1):
+        try:
+            resp = session.request(
+                method, url, params=params, headers=headers, stream=stream, timeout=timeout
+            )
+        except requests.RequestException as exc:
+            last_error = exc
+            if attempt == _MAX_RETRIES:
+                break
+            time.sleep(1.5 * (2**attempt))
+            continue
+        if resp.status_code == 404:
+            resp.close()
+            raise FacebaseNotFoundError(f"404 for {resp.url}")
+        if resp.status_code in (401, 403):
+            resp.close()
+            raise FacebaseAccessError(
+                f"HTTP {resp.status_code} for {resp.url}: this data is not open access. "
+                "FaceBase releases protected human-subjects data only through a data access request."
+            )
+        if resp.status_code in _RETRY_STATUS_CODES and attempt < _MAX_RETRIES:
+            resp.close()
+            time.sleep(1.5 * (2**attempt))
+            continue
+        if resp.status_code >= 400:
+            body = resp.text[:300]
+            resp.close()
+            raise FacebaseError(f"HTTP {resp.status_code} for {resp.url}: {body}")
+        return resp
+    raise FacebaseError(f"request to {url} failed after {_MAX_RETRIES + 1} attempts: {last_error}")
+
+
+def ermrest(path: str) -> Any:
+    """GET an ERMrest path (everything after /catalog/1) and return parsed JSON."""
+    resp = send("GET", f"{ERMREST}/{path.lstrip('/')}")
+    try:
+        return resp.json()
+    except ValueError as exc:
+        raise FacebaseError(f"non-JSON response from {resp.url}") from exc

scigantic_facebase-0.1.0/src/scigantic_facebase/_version.py

@@ -0,0 +1 @@
+__version__ = "0.1.0"

scigantic_facebase-0.1.0/src/scigantic_facebase/catalog.py

@@ -0,0 +1,149 @@
+"""Search FaceBase datasets and list their files through the open ERMrest
+metadata API. Everything here is a metadata call; no file bytes move."""
+
+from __future__ import annotations
+
+import re
+from typing import Any
+from functools import lru_cache
+from urllib.parse import quote
+
+from ._client import SITE, FacebaseNotFoundError, ermrest
+from .models import Dataset, File, Project
+
+_OPEN = "released=true&protected_human_subjects=false"
+
+
+def _q(value: str) -> str:
+    return quote(value, safe="")
+
+
+# FaceBase records name species inconsistently ("zebrafish" in some titles,
+# "Danio rerio" in most). A search word that is a common name also matches
+# the Latin name, and the other way round.
+_SYNONYMS = {
+    "zebrafish": ("zebrafish", "danio rerio"),
+    "danio": ("zebrafish", "danio rerio"),
+    "mouse": ("mouse", "mice", "mus musculus"),
+    "mice": ("mouse", "mice", "mus musculus"),
+    "human": ("human", "homo sapiens"),
+    "chick": ("chick", "gallus gallus"),
+    "microct": ("microct", "micro-ct", "micro ct", "\u00b5ct"),
+}
+
+
+def _filter(text: str | None, include_protected: bool) -> str:
+    parts = [_OPEN if not include_protected else "released=true"]
+    for word in (text or "").split():
+        options = _SYNONYMS.get(word.lower(), (word,))
+        pattern = "|".join(re.escape(o) for o in options)
+        parts.append(f"*::ciregexp::{_q(pattern)}")
+    return "&".join(parts)
+
+
+def _str(row: dict[str, Any], key: str) -> str | None:
+    val = row.get(key)
+    return val if isinstance(val, str) else None
+
+
+def _dataset(row: dict[str, Any]) -> Dataset:
+    project_id = row.get("project")
+    return Dataset(
+        rid=str(row["RID"]),
+        accession=_str(row, "accession"),
+        title=str(row.get("title") or ""),
+        description=str(row.get("description") or ""),
+        release_date=_str(row, "release_date"),
+        doi=_str(row, "DOI"),
+        project_id=project_id if isinstance(project_id, int) else None,
+        protected=bool(row.get("protected_human_subjects")),
+        internal_id=int(row["id"]),
+    )
+
+
+def search(text: str | None = None, *, limit: int = 50, include_protected: bool = False) -> list[Dataset]:
+    """Find datasets, newest first. Every word in `text` must appear somewhere
+    in the dataset's title, description or keywords (case-insensitive).
+    Protected human-subjects datasets are left out unless include_protected=True;
+    their metadata is public but their files need a data access request."""
+    path = f"entity/isa:dataset/{_filter(text, include_protected)}@sort(release_date::desc::,RID)"
+    rows = ermrest(f"{path}?limit={int(limit)}")
+    return [_dataset(r) for r in rows]
+
+
+def count(text: str | None = None, *, include_protected: bool = False) -> int:
+    """How many datasets match `text` (same rules as search)."""
+    rows = ermrest(f"aggregate/isa:dataset/{_filter(text, include_protected)}/n:=cnt(RID)")
+    return int(rows[0]["n"])
+
+
+def get(rid: str) -> Dataset:
+    """One dataset by FaceBase record id (e.g. '2E-YW20')."""
+    rows = ermrest(f"entity/isa:dataset/RID={_q(rid)}")
+    if not rows:
+        raise FacebaseNotFoundError(f"no FaceBase dataset {rid!r}")
+    return _dataset(rows[0])
+
+
+@lru_cache(maxsize=1)
+def _formats() -> dict[str, str]:
+    out: dict[str, str] = {}
+    for r in ermrest("entity/vocab:file_format?limit=500"):
+        key = r.get("id") or r.get("ID")
+        name = r.get("name") or r.get("Name")
+        if key and name:
+            out[str(key)] = str(name)
+    return out
+
+
+def files(dataset: Dataset | str) -> list[File]:
+    """Every file in a dataset, with size, MD5 and an absolute download URL."""
+    ds = get(dataset) if isinstance(dataset, str) else dataset
+    rows = ermrest(f"entity/isa:file/dataset={_q(ds.rid)}@sort(filename)?limit=20000")
+    fmt = _formats()
+    return [
+        File(
+            rid=str(r["RID"]),
+            dataset_rid=ds.rid,
+            filename=str(r["filename"]),
+            size=int(r.get("byte_count") or 0),
+            md5=_str(r, "md5"),
+            url=SITE + str(r["url"]),
+            format=fmt.get(str(r["file_format"])) if r.get("file_format") else None,
+            relative_path=_str(r, "relative_path"),
+            description=_str(r, "description"),
+            protected=ds.protected,
+        )
+        for r in rows
+    ]
+
+
+def project(dataset: Dataset) -> Project | None:
+    """The project (grant and investigator group) a dataset belongs to."""
+    if dataset.project_id is None:
+        return None
+    rows = ermrest(f"entity/isa:project/id={dataset.project_id}")
+    if not rows:
+        return None
+    r = rows[0]
+    return Project(
+        id=int(r["id"]),
+        name=str(r.get("name") or ""),
+        doi=_str(r, "DOI"),
+        funding=_str(r, "funding"),
+    )
+
+
+def contributors(dataset: Dataset) -> list[str]:
+    rows = ermrest(f"attribute/isa:dataset_contributor/dataset_id={dataset.internal_id}/order,full_name@sort(order)")
+    return [str(r["full_name"]) for r in rows if r.get("full_name")]
+
+
+def cite(dataset: Dataset | str) -> str:
+    """The citation FaceBase's Terms of Use ask for: authors, title,
+    FaceBase Consortium, DOI link and release year."""
+    ds = get(dataset) if isinstance(dataset, str) else dataset
+    authors = ", ".join(contributors(ds)) or "FaceBase Consortium contributors"
+    year = (ds.release_date or "")[:4]
+    link = ds.doi_url or ds.url
+    return f"{authors}. {ds.title}. FaceBase Consortium {link}" + (f" ({year})." if year else ".")

scigantic_facebase-0.1.0/src/scigantic_facebase/microct.py

@@ -0,0 +1,147 @@
+"""Zip-of-TIFF micro-CT stacks, read in place.
+
+Many FaceBase micro-CT datasets ship each scan as one .zip holding a stack of
+reconstructed TIFF slices plus a SkyScan scan log. A zip keeps a directory at
+its end, so a single slice can be read with one HTTP range request; a 50 MB
+scan is never downloaded just to look at one cross-section.
+"""
+
+from __future__ import annotations
+
+import io
+import re
+import threading
+import zipfile
+from concurrent.futures import ThreadPoolExecutor
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+import numpy.typing as npt
+
+from . import catalog
+from .models import Dataset, File
+from .remote import RangeFile, open_remote
+
+_TIF = re.compile(r"\.tiff?$", re.IGNORECASE)
+
+
+class Stack:
+    """A remote zip of TIFF slices. Create with open_stack()."""
+
+    def __init__(self, source: RangeFile, name: str = "") -> None:
+        self.name = name
+        self._source = source
+        self._zip = zipfile.ZipFile(source)
+        self._lock = threading.Lock()
+        self.slice_names: list[str] = sorted(
+            i.filename for i in self._zip.infolist() if _TIF.search(i.filename) and not i.is_dir()
+        )
+        self._logs = [i.filename for i in self._zip.infolist() if i.filename.lower().endswith(".log")]
+
+    def __len__(self) -> int:
+        return len(self.slice_names)
+
+    @property
+    def requests(self) -> int:
+        """HTTP requests made so far, to see how little was fetched."""
+        return self._source.requests
+
+    @property
+    def bytes_fetched(self) -> int:
+        return self._source.bytes_fetched
+
+    def read_slice(self, index: int) -> npt.NDArray[Any]:
+        """One slice as a 2-D array (decoded with tifffile)."""
+        import tifffile
+
+        with self._lock:
+            raw = self._zip.read(self.slice_names[index])
+        return np.asarray(tifffile.imread(io.BytesIO(raw)))
+
+    def volume(self, *, step: int = 1, start: int = 0, stop: int | None = None, workers: int = 4) -> npt.NDArray[Any]:
+        """Slices start:stop:step stacked into a (z, y, x) array. Slices are
+        read in parallel; each worker reuses the same cached connection."""
+        idx = list(range(start, len(self) if stop is None else stop, step))
+        if not idx:
+            raise ValueError("no slices selected")
+        with ThreadPoolExecutor(max_workers=max(1, workers)) as pool:
+            slices = list(pool.map(self.read_slice, idx))
+        return np.stack(slices)
+
+    def scan_log(self) -> dict[str, dict[str, str]]:
+        """The SkyScan scan log as {section: {key: value}}; empty when the
+        zip has none."""
+        if not self._logs:
+            return {}
+        with self._lock:
+            text = self._zip.read(self._logs[0]).decode("latin-1")
+        out: dict[str, dict[str, str]] = {}
+        section = ""
+        for line in text.splitlines():
+            line = line.strip()
+            if line.startswith("[") and line.endswith("]"):
+                section = line[1:-1]
+                out.setdefault(section, {})
+            elif "=" in line and section:
+                key, _, val = line.partition("=")
+                out[section][key.strip()] = val.strip()
+        return out
+
+    @property
+    def voxel_size_um(self) -> float | None:
+        """Isotropic voxel edge in micrometres, from the scan log."""
+        log = self.scan_log()
+        for section in ("Reconstruction", "Acquisition"):
+            for key in ("Pixel Size (um)", "Image Pixel Size (um)", "Scaled Image Pixel Size (um)"):
+                val = log.get(section, {}).get(key)
+                if val:
+                    try:
+                        return float(val)
+                    except ValueError:
+                        continue
+        return None
+
+    def close(self) -> None:
+        self._zip.close()
+
+    def __enter__(self) -> Stack:
+        return self
+
+    def __exit__(self, *exc: object) -> None:
+        self.close()
+
+
+def open_stack(file: File) -> Stack:
+    """Open a zip-of-TIFFs file from FaceBase without downloading it."""
+    return Stack(open_remote(file), name=file.filename)
+
+
+@dataclass(frozen=True)
+class ThyroidScan:
+    """One scan in the zebrafish thyroid-hormone craniofacial atlas."""
+
+    dataset: Dataset
+    condition: str  # hypothyroid, euthyroid or hyperthyroid
+    standard_length_mm: float
+
+
+_TITLE = re.compile(r"microCT scan of ([\d.]+) mm (hypothyroid|euthyroid|hyperthyroid) Danio rerio head", re.IGNORECASE)
+
+
+def parse_thyroid_title(title: str) -> tuple[float, str] | None:
+    """(standard length in mm, condition) from an atlas dataset title."""
+    m = _TITLE.search(title)
+    return (float(m.group(1)), m.group(2).lower()) if m else None
+
+
+def thyroid_atlas() -> list[ThyroidScan]:
+    """The zebrafish head micro-CT series: fish from 12 to 25 mm standard
+    length raised hypothyroid, euthyroid (normal) or hyperthyroid, one dataset
+    per fish. Sorted by condition then length."""
+    scans = []
+    for ds in catalog.search("microCT Danio rerio head thyroid", limit=300):
+        parsed = parse_thyroid_title(ds.title)
+        if parsed:
+            scans.append(ThyroidScan(ds, parsed[1], parsed[0]))
+    return sorted(scans, key=lambda s: (s.condition, s.standard_length_mm, s.dataset.rid))

scigantic_facebase-0.1.0/src/scigantic_facebase/models.py

@@ -0,0 +1,66 @@
+"""Typed records for FaceBase datasets and files."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+from ._client import SITE
+
+
+@dataclass(frozen=True)
+class Dataset:
+    """One FaceBase dataset. `rid` is FaceBase's record id, the same string
+    that appears in the dataset's landing page and DOI."""
+
+    rid: str
+    accession: str | None
+    title: str
+    description: str
+    release_date: str | None
+    doi: str | None
+    project_id: int | None
+    protected: bool
+    internal_id: int
+
+    @property
+    def url(self) -> str:
+        return f"{SITE}/id/{self.rid}"
+
+    @property
+    def doi_url(self) -> str | None:
+        return f"https://doi.org/{self.doi}" if self.doi else None
+
+    @property
+    def open_access(self) -> bool:
+        """False for datasets behind FaceBase's data access request."""
+        return not self.protected
+
+
+@dataclass(frozen=True)
+class Project:
+    id: int
+    name: str
+    doi: str | None
+    funding: str | None
+
+
+@dataclass(frozen=True)
+class File:
+    """One file in a FaceBase dataset. `url` is absolute and anonymous for
+    open-access datasets; byte-range requests are honoured."""
+
+    rid: str
+    dataset_rid: str
+    filename: str
+    size: int
+    md5: str | None
+    url: str
+    format: str | None = None
+    relative_path: str | None = None
+    description: str | None = None
+    protected: bool = field(default=False, compare=False)
+
+    @property
+    def path(self) -> str:
+        """Path within the dataset, folders included."""
+        return f"{self.relative_path or ''}{self.filename}"

scigantic_facebase-0.1.0/src/scigantic_facebase/remote.py

@@ -0,0 +1,171 @@
+"""Read FaceBase files over HTTP without downloading them first.
+
+`RangeFile` is a seekable, read-only file object backed by HTTP Range
+requests with a small block cache. Hand it to zipfile.ZipFile or tifffile and
+only the bytes they actually touch cross the network. For a 50 MB zip of
+600 TIFF slices, opening the archive reads the central directory (one small
+request) and each slice costs one request of its own size.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import io
+import os
+import threading
+from collections import OrderedDict
+from pathlib import Path
+from typing import Any
+
+from ._client import FacebaseError, send
+from .models import File
+
+_BLOCK = 1 << 20  # 1 MiB
+_CACHE_BLOCKS = 32
+
+
+class RangeFile(io.RawIOBase):
+    """Seekable read-only view of one remote file."""
+
+    def __init__(self, url: str, size: int | None = None) -> None:
+        super().__init__()
+        self.url = url
+        self._pos = 0
+        self._lock = threading.Lock()
+        self._cache: OrderedDict[int, bytes] = OrderedDict()
+        self.requests = 0
+        self.bytes_fetched = 0
+        self._size = size if size is not None else self._probe_size()
+
+    def _probe_size(self) -> int:
+        resp = send("GET", self.url, headers={"Range": "bytes=0-0"}, stream=True)
+        try:
+            cr = resp.headers.get("Content-Range", "")
+            if "/" in cr and cr.rsplit("/", 1)[1].isdigit():
+                return int(cr.rsplit("/", 1)[1])
+            length = resp.headers.get("Content-Length")
+            if resp.status_code == 200 and length and length.isdigit():
+                return int(length)
+        finally:
+            resp.close()
+        raise FacebaseError(f"cannot determine size of {self.url}")
+
+    def __len__(self) -> int:
+        return self._size
+
+    def readable(self) -> bool:
+        return True
+
+    def seekable(self) -> bool:
+        return True
+
+    def tell(self) -> int:
+        return self._pos
+
+    def seek(self, offset: int, whence: int = os.SEEK_SET) -> int:
+        if whence == os.SEEK_SET:
+            self._pos = offset
+        elif whence == os.SEEK_CUR:
+            self._pos += offset
+        elif whence == os.SEEK_END:
+            self._pos = self._size + offset
+        else:
+            raise ValueError(f"invalid whence: {whence}")
+        self._pos = max(self._pos, 0)
+        return self._pos
+
+    def _block(self, index: int) -> bytes:
+        with self._lock:
+            hit = self._cache.get(index)
+            if hit is not None:
+                self._cache.move_to_end(index)
+                return hit
+        start = index * _BLOCK
+        end = min(start + _BLOCK, self._size) - 1
+        resp = send("GET", self.url, headers={"Range": f"bytes={start}-{end}"})
+        data = resp.content
+        if resp.status_code == 200 and len(data) != end - start + 1:
+            raise FacebaseError(f"{self.url} ignored the Range header")
+        with self._lock:
+            self.requests += 1
+            self.bytes_fetched += len(data)
+            self._cache[index] = data
+            while len(self._cache) > _CACHE_BLOCKS:
+                self._cache.popitem(last=False)
+        return data
+
+    def read(self, size: int = -1) -> bytes:
+        if self._pos >= self._size:
+            return b""
+        if size is None or size < 0:
+            size = self._size - self._pos
+        stop = min(self._pos + size, self._size)
+        # A request larger than the cache would thrash it; fetch it in one go.
+        if stop - self._pos > _BLOCK * 4:
+            resp = send("GET", self.url, headers={"Range": f"bytes={self._pos}-{stop - 1}"})
+            data = resp.content
+            with self._lock:
+                self.requests += 1
+                self.bytes_fetched += len(data)
+            self._pos += len(data)
+            return data
+        out = bytearray()
+        pos = self._pos
+        while pos < stop:
+            block = self._block(pos // _BLOCK)
+            off = pos % _BLOCK
+            take = block[off : off + (stop - pos)]
+            if not take:
+                break
+            out += take
+            pos += len(take)
+        self._pos = pos
+        return bytes(out)
+
+    def readinto(self, b: Any) -> int:
+        data = self.read(len(b))
+        b[: len(data)] = data
+        return len(data)
+
+
+def open_remote(file: File) -> RangeFile:
+    """Open a FaceBase file as a seekable file object (nothing downloaded)."""
+    return RangeFile(file.url, size=file.size or None)
+
+
+def download(file: File, dest: str | os.PathLike[str], *, verify: bool = True, chunk: int = 1 << 20) -> Path:
+    """Download a file to `dest` (a directory or a file path) and return the
+    path. Checks size and MD5 against FaceBase's record unless verify=False.
+    An existing file with the right size and MD5 is kept, so reruns are free."""
+    target = Path(dest)
+    if target.is_dir():
+        target = target / file.filename
+    target.parent.mkdir(parents=True, exist_ok=True)
+    if target.exists() and target.stat().st_size == file.size and (not verify or not file.md5 or _md5(target) == file.md5):
+        return target
+    tmp = target.with_suffix(target.suffix + ".part")
+    md5 = hashlib.md5()
+    resp = send("GET", file.url, stream=True, timeout=120.0)
+    try:
+        with open(tmp, "wb") as fh:
+            for part in resp.iter_content(chunk):
+                fh.write(part)
+                md5.update(part)
+    finally:
+        resp.close()
+    if file.size and tmp.stat().st_size != file.size:
+        tmp.unlink(missing_ok=True)
+        raise FacebaseError(f"{file.filename}: got {tmp.stat().st_size if tmp.exists() else 0} bytes, expected {file.size}")
+    if verify and file.md5 and md5.hexdigest() != file.md5:
+        tmp.unlink(missing_ok=True)
+        raise FacebaseError(f"{file.filename}: MD5 mismatch")
+    tmp.replace(target)
+    return target
+
+
+def _md5(path: Path) -> str:
+    h = hashlib.md5()
+    with open(path, "rb") as fh:
+        for part in iter(lambda: fh.read(1 << 20), b""):
+            h.update(part)
+    return h.hexdigest()

scigantic_facebase-0.1.0/src/scigantic_facebase.egg-info/PKG-INFO

@@ -0,0 +1,88 @@
+Metadata-Version: 2.4
+Name: scigantic-facebase
+Version: 0.1.0
+Summary: Search FaceBase, the craniofacial research data hub, and read its open-access micro-CT, imaging and sequencing data from Python: a typed catalog over FaceBase's open metadata API, citations in the form its Terms of Use ask for, ranged reads of remote files, and single-slice reads from zipped micro-CT stacks without downloading them.
+Author: Scigantic
+License: MIT-0
+Project-URL: Homepage, https://scigantic.com
+Project-URL: Repository, https://github.com/Scigantic/scigantic-facebase
+Project-URL: Issues, https://github.com/Scigantic/scigantic-facebase/issues
+Project-URL: FaceBase, https://www.facebase.org
+Keywords: facebase,craniofacial,micro-ct,zebrafish,deriva,ermrest,bioinformatics
+Classifier: License :: OSI Approved :: MIT No Attribution License (MIT-0)
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Typing :: Typed
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Classifier: Topic :: Scientific/Engineering :: Image Processing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests<3,>=2.28
+Requires-Dist: numpy<3,>=1.24
+Requires-Dist: tifffile<2027,>=2023.7
+Requires-Dist: imagecodecs>=2023.9
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: mypy>=1.10; extra == "dev"
+Requires-Dist: types-requests; extra == "dev"
+Dynamic: license-file
+
+# scigantic-facebase
+
+Search [FaceBase](https://www.facebase.org), the craniofacial research data hub, and read its open-access data from Python.
+
+FaceBase holds about 1,070 open datasets: micro-CT and light-sheet imaging, ChIP-seq and RNA-seq tracks, expression tables and more, from mouse, zebrafish, human and other models. This package is a typed layer over FaceBase's open metadata API and file store. No key or login is needed.
+
+```
+pip install scigantic-facebase
+```
+
+## Use
+
+```python
+import scigantic_facebase as fb
+
+hits = fb.search("zebrafish thyroid microCT")     # newest first
+ds = hits[0]
+print(ds.title, ds.doi_url)
+print(fb.cite(ds))
+
+files = fb.files(ds)                               # size, MD5, download URL
+fb.download(files[0], "data/")                     # MD5-checked
+```
+
+Search matches every word against the whole record, and treats "zebrafish" and "Danio rerio", "mouse" and "Mus musculus" as the same thing.
+
+## Read a micro-CT scan without downloading it
+
+Many micro-CT datasets are a zip of TIFF slices. A zip keeps its directory at the end, so one slice costs one HTTP range request.
+
+```python
+stack = fb.microct.open_stack(files[0])
+len(stack)                  # 596 slices
+stack.voxel_size_um         # 10.5, from the scan log
+img = stack.read_slice(300)
+vol = stack.volume(step=10) # every 10th slice, (z, y, x)
+stack.bytes_fetched         # a few MB of a 49 MB file
+```
+
+`fb.microct.thyroid_atlas()` lists the zebrafish thyroid-hormone atlas: 81 head scans from 12 to 25 mm standard length, raised hypothyroid, euthyroid or hyperthyroid.
+
+`fb.open_remote(file)` gives any FaceBase file as a seekable file object backed by range requests, which `zipfile`, `tifffile` and `h5py` accept.
+
+## What is not here
+
+Protected human-subjects datasets sit behind FaceBase's data access request. Their metadata is public and `fb.search(include_protected=True)` lists them, but reading their files raises `FacebaseAccessError`.
+
+## Terms of use
+
+Copyright to FaceBase data belongs to the contributing investigators. FaceBase's [Terms of Use](https://www.facebase.org/policies/tou/) ask users to cite the dataset, acknowledge FaceBase, and send other people to FaceBase instead of circulating downloaded files. `fb.cite()` produces the citation. This package fetches from FaceBase on request and does not mirror anything.
+
+## License
+
+MIT-0 for this code. The data carries its own terms.

scigantic_facebase-0.1.0/src/scigantic_facebase.egg-info/SOURCES.txt

@@ -0,0 +1,18 @@
+LICENSE
+README.md
+pyproject.toml
+src/scigantic_facebase/__init__.py
+src/scigantic_facebase/_client.py
+src/scigantic_facebase/_version.py
+src/scigantic_facebase/catalog.py
+src/scigantic_facebase/microct.py
+src/scigantic_facebase/models.py
+src/scigantic_facebase/py.typed
+src/scigantic_facebase/remote.py
+src/scigantic_facebase.egg-info/PKG-INFO
+src/scigantic_facebase.egg-info/SOURCES.txt
+src/scigantic_facebase.egg-info/dependency_links.txt
+src/scigantic_facebase.egg-info/requires.txt
+src/scigantic_facebase.egg-info/top_level.txt
+tests/test_live.py
+tests/test_remote.py

scigantic_facebase-0.1.0/src/scigantic_facebase.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

scigantic_facebase-0.1.0/src/scigantic_facebase.egg-info/requires.txt

@@ -0,0 +1,9 @@
+requests<3,>=2.28
+numpy<3,>=1.24
+tifffile<2027,>=2023.7
+imagecodecs>=2023.9
+
+[dev]
+pytest>=7
+mypy>=1.10
+types-requests

scigantic_facebase-0.1.0/src/scigantic_facebase.egg-info/top_level.txt

@@ -0,0 +1 @@
+scigantic_facebase

scigantic_facebase-0.1.0/tests/test_live.py

@@ -0,0 +1,45 @@
+"""Live checks against facebase.org. Skipped with `-m 'not live'`."""
+
+from __future__ import annotations
+
+import pytest
+
+import scigantic_facebase as fb
+
+pytestmark = pytest.mark.live
+
+
+def test_search_and_get() -> None:
+    hits = fb.search("zebrafish thyroid microCT", limit=5)
+    assert hits and all(h.open_access for h in hits)
+    assert fb.count("zebrafish thyroid microCT") >= 80
+    assert fb.get("2E-YW20").doi == "10.25550/2E-YW20"
+
+
+def test_citation_has_doi_and_year() -> None:
+    c = fb.cite("2E-YW20")
+    assert "https://doi.org/10.25550/2E-YW20" in c and "(2024)" in c and "FaceBase Consortium" in c
+
+
+def test_stack_read_in_place() -> None:
+    f = fb.files("2E-YW20")[0]
+    with fb.microct.open_stack(f) as stack:
+        assert len(stack) > 500
+        assert stack.voxel_size_um and 10 < stack.voxel_size_um < 11
+        img = stack.read_slice(len(stack) // 2)
+        assert img.ndim == 2
+        assert stack.bytes_fetched < f.size // 4
+
+
+def test_atlas_is_three_conditions() -> None:
+    atlas = fb.microct.thyroid_atlas()
+    assert {s.condition for s in atlas} == {"hypothyroid", "euthyroid", "hyperthyroid"}
+    assert len(atlas) >= 80
+
+
+def test_protected_dataset_files_raise() -> None:
+    ds = next(d for d in fb.search(include_protected=True, limit=400) if d.protected)
+    files = fb.files(ds)
+    assert files, "protected dataset lists its files (metadata is public)"
+    with pytest.raises(fb.FacebaseAccessError):
+        fb.open_remote(files[0]).read(1)

scigantic_facebase-0.1.0/tests/test_remote.py

@@ -0,0 +1,103 @@
+"""RangeFile and download against a local Range-capable server (offline)."""
+
+from __future__ import annotations
+
+import hashlib
+import http.server
+import io
+import threading
+import zipfile
+from collections.abc import Iterator
+from pathlib import Path
+
+import pytest
+
+from scigantic_facebase import File, RangeFile, download
+from scigantic_facebase.microct import Stack
+
+PAYLOAD = bytes(range(256)) * 20000  # ~5 MB
+
+
+class _Handler(http.server.BaseHTTPRequestHandler):
+    body = PAYLOAD
+    hits: list[str] = []
+
+    def log_message(self, *a: object) -> None:  # silence
+        pass
+
+    def do_GET(self) -> None:
+        rng = self.headers.get("Range")
+        data = self.body
+        if rng:
+            a, _, b = rng.removeprefix("bytes=").partition("-")
+            lo, hi = int(a), min(int(b), len(data) - 1)
+            self.hits.append(rng)
+            self.send_response(206)
+            self.send_header("Content-Range", f"bytes {lo}-{hi}/{len(data)}")
+            chunk = data[lo : hi + 1]
+        else:
+            self.send_response(200)
+            chunk = data
+        self.send_header("Content-Length", str(len(chunk)))
+        self.end_headers()
+        self.wfile.write(chunk)
+
+
+@pytest.fixture()
+def server() -> Iterator[str]:
+    _Handler.hits = []
+    srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), _Handler)
+    threading.Thread(target=srv.serve_forever, daemon=True).start()
+    yield f"http://127.0.0.1:{srv.server_port}/f.bin"
+    srv.shutdown()
+
+
+def test_random_access_reads_only_needed_blocks(server: str) -> None:
+    f = RangeFile(server)
+    assert len(f) == len(PAYLOAD)
+    f.seek(3_000_000)
+    assert f.read(10) == PAYLOAD[3_000_000:3_000_010]
+    assert f.requests == 1  # the size probe is not counted; one block read here
+    assert f.bytes_fetched <= 1 << 20
+
+
+def test_read_past_end_and_seek_end(server: str) -> None:
+    f = RangeFile(server)
+    f.seek(-5, 2)
+    assert f.read() == PAYLOAD[-5:]
+    assert f.read(10) == b""
+
+
+def test_download_verifies_md5(server: str, tmp_path: Path) -> None:
+    good = File("r", "d", "f.bin", len(PAYLOAD), hashlib.md5(PAYLOAD).hexdigest(), server)
+    out = download(good, tmp_path)
+    assert out.read_bytes() == PAYLOAD
+    bad = File("r", "d", "g.bin", len(PAYLOAD), "0" * 32, server)
+    with pytest.raises(Exception, match="MD5"):
+        download(bad, tmp_path)
+    assert not (tmp_path / "g.bin").exists()
+
+
+def test_stack_reads_one_slice_from_a_remote_zip(tmp_path: Path) -> None:
+    import numpy as np
+    import tifffile
+
+    buf = io.BytesIO()
+    with zipfile.ZipFile(buf, "w", zipfile.ZIP_STORED) as z:
+        for i in range(5):
+            t = io.BytesIO()
+            tifffile.imwrite(t, np.full((8, 9), i, dtype=np.uint16))
+            z.writestr(f"scan/s{i:04d}.tif", t.getvalue())
+        z.writestr("scan/scan.log", "[Acquisition]\nImage Pixel Size (um)=12.5\n")
+    _Handler.body = buf.getvalue()
+    srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), _Handler)
+    threading.Thread(target=srv.serve_forever, daemon=True).start()
+    try:
+        stack = Stack(RangeFile(f"http://127.0.0.1:{srv.server_port}/z.zip"))
+        assert len(stack) == 5
+        assert int(stack.read_slice(3).mean()) == 3
+        assert stack.volume(step=2).shape == (3, 8, 9)
+        assert stack.voxel_size_um == 12.5
+    finally:
+        srv.shutdown()
+        _Handler.body = PAYLOAD