scigantic-headers 0.1.0__tar.gz

scigantic_headers-0.1.0/LICENSE

@@ -0,0 +1,4 @@
+Proprietary — Evaluation Use Only.
+Copyright (c) Scigantic. All rights reserved.
+Provided for evaluation; not licensed for redistribution or production use
+without a separate written agreement.

scigantic_headers-0.1.0/PKG-INFO

@@ -0,0 +1,203 @@
+Metadata-Version: 2.4
+Name: scigantic-headers
+Version: 0.1.0
+Summary: Decode scientific file headers (MRC cryo-EM, extensible) into typed context for agents — pure, zero-dependency, with bounded parallel readers.
+Author: Scigantic
+License: Proprietary — Evaluation Use Only. See LICENSE.
+Keywords: cryo-em,mrc,header,schema-card,instrument,context,edge
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Dynamic: license-file
+
+# scigantic-headers
+
+Read the metadata of a scientific file and return its fields (dimensions, data
+type, pixel size, columns, row count) as a dict, without reading the rest of the
+file. Decodes MRC, NPY, NIfTI, CryoSPARC `.cs`, and Parquet. The decode
+functions have no dependencies; separate reader functions fetch the bytes from a
+file or a URL (the leading bytes for most formats, the trailing bytes for
+Parquet, whose schema is in a footer).
+
+```python
+from scigantic_headers import decode_file
+hdr = decode_file("session/frame.mrc")
+hdr.summary            # "MRC stack 4096x4096x16, float32"
+hdr.fields["dtype"]    # "float32"
+hdr.fields["nz"]       # 16   (frames)
+```
+
+One implementation, shared. The cloud probe's TypeScript twin
+(`backend/src/services/headerDecoders.ts`), the notebook, and the edge
+context-producer describe a file the same way. Same byte offsets, same
+`mode -> dtype` table, held to the same test fixtures so they cannot drift.
+
+Edge use: the context-producer (`infrastructure/edge/context-producer`) runs
+this next to an instrument, with no network, on amd64 or arm64. The zero
+dependencies and header-only reads are what let it run there.
+
+## Install
+
+Published to PyPI on a `scigantic-headers-vX.Y.Z` tag (see
+`.github/workflows/scigantic-headers.yml`):
+
+    pip install scigantic-headers
+
+It has zero runtime dependencies, so it also installs from source into an
+air-gapped image with no package index at build time. The notebook and
+edge-producer images bundle it this way:
+
+    COPY infrastructure/python/scigantic-headers /tmp/scigantic-headers
+    RUN pip install /tmp/scigantic-headers
+
+Both paths install the same version pinned in `pyproject.toml`. Consumers that
+install from an index should pin it (`scigantic-headers==X.Y.Z`).
+
+## Why it exists
+
+A schema card or context record wants to say what a file is without opening it
+over a FUSE mount. A scientific file's first ~1 KiB is a structured header that
+already says so. This reads that, and nothing more.
+
+Decoders today:
+
+- **MRC / MRCS**: cryo-EM micrographs, movie stacks, tilt series, EMDB maps.
+- **NPY**: NumPy arrays (ML, genomics, materials, not cryo-EM). Cross-checked
+  against numpy.
+- **NIfTI-1**: neuroimaging volumes, `.nii` and `.nii.gz` (not cryo-EM).
+  Cross-checked against nibabel. Big- and little-endian.
+- **CryoSPARC `.cs`**: a structured-array dataset. Reports record count and the
+  field schema.
+- **Parquet**: reads the footer, not a leading header, and returns the column
+  names, physical types, and row count. Cross-checked against pyarrow.
+
+`.gz` is transparent. The reader decompresses just the leading block, so a
+gzipped `.nii.gz` or `.mrc.gz` decodes without inflating the whole file, and
+dispatch sees the inner format.
+
+Adding a format is a pure `bytes -> DecodedHeader | None` function plus one
+`register_decoder` call. Nothing about the dispatch, the readers, or the batch
+path is cryo-EM-specific. NPY and NIfTI landed with zero plumbing changes.
+
+## Pixel size for raw movies
+
+A raw MRC movie header has no pixel size: CELLA is 0, so `decode_*` returns
+`pixelSizeA` as `None`. The value is in the RELION STAR or CryoSPARC `.cs` file
+the workflow writes in the session, and the library reads it. `read_session_optics`
+finds that file next to (or above) the data file and returns the optics:
+
+```python
+from scigantic_headers import decode_file, read_session_optics
+
+hdr = decode_file("Movies/frame.mrc")       # header fields; pixelSizeA is None
+opt = read_session_optics("Movies/frame.mrc")
+# {'pixelSizeA': 1.05, 'voltageKv': 300.0, 'source': 'relion-star:particles.star'}
+```
+
+`read_star_optics(path)` and `read_cryosparc_optics(path)` read a specific optics
+file when you already know which one. The RELION 3.1 `data_optics` loop, the
+older key-value form, optics columns in the main data loop, and the legacy
+detector-pixel-size / magnification pair are all handled.
+
+## Layout
+
+    src/scigantic_headers/
+      decoders.py   pure core: registry + MRC / NPY / NIfTI / .cs decoders
+      parquet.py    Parquet footer decode (small Thrift-compact reader)
+      sources.py    read leading bytes from file or URL; bounded parallel batch
+      star.py       RELION STAR optics reader
+      cryosparc.py  CryoSPARC .cs optics reader
+      optics.py     read_session_optics: find a data file's optics file
+      cli.py        scigantic-headers <file|url|--dir>
+      benchmark.py  scigantic-headers-bench, measures the speed levers
+    tests/          pytest (117 tests, incl. fuzz + golden fixtures)
+
+## Robustness
+
+A decoder is handed the leading bytes of arbitrary files, so it must be total:
+for any input it returns None or a valid, JSON-safe DecodedHeader. It never
+raises and never emits NaN or inf. This is enforced, not assumed.
+
+- **Fuzz** (`test_fuzz.py`): thousands of random and magic-seeded byte buffers
+  through every decoder. Each result must serialize with `allow_nan=False`.
+- **Golden fixtures** (`fixtures/mrc-cases.json`): hex input to exact decode.
+  This is the single source of truth the three MRC decoders (this library, the
+  TypeScript twin, scigantic_empiar) are all held to, so none can silently
+  diverge.
+- Non-finite floats (a garbage NIfTI pixdim, an inf MRC cell) sanitize to null.
+
+## Speed
+
+The decode is microseconds: a 1 KiB read and a few `struct` unpacks. Optimizing
+that is a rounding error. All the speed is in I/O, and two things carry it. Both
+are measured, not asserted (run `scigantic-headers-bench`).
+
+**Read the header, not the file.** The read is exactly `HEADER_BYTES` (1 KiB),
+so its cost is independent of file size and a multi-GB movie decodes as fast as
+a small one.
+
+    header read + decode (1 KiB):        15 us
+    full-file read + decode (200 MB):  22.7 ms      (about 1,500x cheaper)
+
+**Parallelize the I/O, bounded.** Header reads across files are independent and
+I/O-bound, so a thread pool overlaps their latency (the read releases the GIL).
+Measured over 8 real EMPIAR headers pulled by HTTP Range:
+
+    serial   (1 worker):   25.7 s
+    parallel (8 workers):   0.7 s        (about 35x)
+
+That 35x is large because EBI's per-request latency is about 3 s, so overlapping
+eight requests recovers a lot of dead wait. On low-latency local storage the
+speedup is smaller. The win scales with per-read latency, which is why the pool
+is bounded and tunable (default 8, the EMPIAR sweet spot; past a point more
+connections hit server throttling or disk-queue thrash and get slower).
+
+Two more, by construction:
+
+- **Filter before I/O.** `has_decoder_for` is a cheap extension check. The batch
+  and walk paths use it to skip a file before any syscall.
+- **Zero deps, fast import.** The dtype table is a plain dict, not numpy. There
+  is nothing to install and nothing to import, and it runs air-gapped.
+
+The biggest system-level lever lives in the caller, not here: decode once when a
+file lands, write the context record, and answer every later query from the tiny
+record without re-decoding. The edge context-producer and the cloud probe both
+do this.
+
+## Use
+
+    from scigantic_headers import decode_file, decode_paths, decode_urls, iter_decodable_files
+
+    decode_file("s/frame.mrc")                              # one local file
+    decode_paths(iter_decodable_files("/mnt/sessions"))     # a tree, parallel
+    decode_urls([...], workers=8)                           # remote, by Range
+
+    # CLI
+    scigantic-headers session/frame.mrc
+    scigantic-headers https://ftp.ebi.ac.uk/empiar/.../img.mrcs
+    scigantic-headers --dir /mnt/flashblade/sessions --workers 8
+
+    pytest
+    scigantic-headers-bench --empiar   # reproduce the numbers above
+
+## Limitations
+
+- **Pointer-based formats are not supported.** HDF5 and TIFF store their layout
+  behind internal pointers, so reading the header or footer alone is not enough.
+  Parquet (a footer) is supported; formats that need to chase offsets through the
+  file are not. Plain gzip is handled (the reader inflates the leading block),
+  but BAM's BGZF framing and per-record structure are not.
+- **Output is not a standard.** The fields are an ad-hoc dict, not an
+  interchange format such as Allotrope ASM. Map the dict to a standard if you
+  need one.
+
+## Repository
+
+This is the standalone home of the library. It is also vendored into the
+Scigantic monorepo, where the paths above (`backend/...`, `infrastructure/...`)
+point at the TypeScript twin, the notebook, and the edge context-producer that
+use it.

scigantic_headers-0.1.0/README.md

@@ -0,0 +1,187 @@
+# scigantic-headers
+
+Read the metadata of a scientific file and return its fields (dimensions, data
+type, pixel size, columns, row count) as a dict, without reading the rest of the
+file. Decodes MRC, NPY, NIfTI, CryoSPARC `.cs`, and Parquet. The decode
+functions have no dependencies; separate reader functions fetch the bytes from a
+file or a URL (the leading bytes for most formats, the trailing bytes for
+Parquet, whose schema is in a footer).
+
+```python
+from scigantic_headers import decode_file
+hdr = decode_file("session/frame.mrc")
+hdr.summary            # "MRC stack 4096x4096x16, float32"
+hdr.fields["dtype"]    # "float32"
+hdr.fields["nz"]       # 16   (frames)
+```
+
+One implementation, shared. The cloud probe's TypeScript twin
+(`backend/src/services/headerDecoders.ts`), the notebook, and the edge
+context-producer describe a file the same way. Same byte offsets, same
+`mode -> dtype` table, held to the same test fixtures so they cannot drift.
+
+Edge use: the context-producer (`infrastructure/edge/context-producer`) runs
+this next to an instrument, with no network, on amd64 or arm64. The zero
+dependencies and header-only reads are what let it run there.
+
+## Install
+
+Published to PyPI on a `scigantic-headers-vX.Y.Z` tag (see
+`.github/workflows/scigantic-headers.yml`):
+
+    pip install scigantic-headers
+
+It has zero runtime dependencies, so it also installs from source into an
+air-gapped image with no package index at build time. The notebook and
+edge-producer images bundle it this way:
+
+    COPY infrastructure/python/scigantic-headers /tmp/scigantic-headers
+    RUN pip install /tmp/scigantic-headers
+
+Both paths install the same version pinned in `pyproject.toml`. Consumers that
+install from an index should pin it (`scigantic-headers==X.Y.Z`).
+
+## Why it exists
+
+A schema card or context record wants to say what a file is without opening it
+over a FUSE mount. A scientific file's first ~1 KiB is a structured header that
+already says so. This reads that, and nothing more.
+
+Decoders today:
+
+- **MRC / MRCS**: cryo-EM micrographs, movie stacks, tilt series, EMDB maps.
+- **NPY**: NumPy arrays (ML, genomics, materials, not cryo-EM). Cross-checked
+  against numpy.
+- **NIfTI-1**: neuroimaging volumes, `.nii` and `.nii.gz` (not cryo-EM).
+  Cross-checked against nibabel. Big- and little-endian.
+- **CryoSPARC `.cs`**: a structured-array dataset. Reports record count and the
+  field schema.
+- **Parquet**: reads the footer, not a leading header, and returns the column
+  names, physical types, and row count. Cross-checked against pyarrow.
+
+`.gz` is transparent. The reader decompresses just the leading block, so a
+gzipped `.nii.gz` or `.mrc.gz` decodes without inflating the whole file, and
+dispatch sees the inner format.
+
+Adding a format is a pure `bytes -> DecodedHeader | None` function plus one
+`register_decoder` call. Nothing about the dispatch, the readers, or the batch
+path is cryo-EM-specific. NPY and NIfTI landed with zero plumbing changes.
+
+## Pixel size for raw movies
+
+A raw MRC movie header has no pixel size: CELLA is 0, so `decode_*` returns
+`pixelSizeA` as `None`. The value is in the RELION STAR or CryoSPARC `.cs` file
+the workflow writes in the session, and the library reads it. `read_session_optics`
+finds that file next to (or above) the data file and returns the optics:
+
+```python
+from scigantic_headers import decode_file, read_session_optics
+
+hdr = decode_file("Movies/frame.mrc")       # header fields; pixelSizeA is None
+opt = read_session_optics("Movies/frame.mrc")
+# {'pixelSizeA': 1.05, 'voltageKv': 300.0, 'source': 'relion-star:particles.star'}
+```
+
+`read_star_optics(path)` and `read_cryosparc_optics(path)` read a specific optics
+file when you already know which one. The RELION 3.1 `data_optics` loop, the
+older key-value form, optics columns in the main data loop, and the legacy
+detector-pixel-size / magnification pair are all handled.
+
+## Layout
+
+    src/scigantic_headers/
+      decoders.py   pure core: registry + MRC / NPY / NIfTI / .cs decoders
+      parquet.py    Parquet footer decode (small Thrift-compact reader)
+      sources.py    read leading bytes from file or URL; bounded parallel batch
+      star.py       RELION STAR optics reader
+      cryosparc.py  CryoSPARC .cs optics reader
+      optics.py     read_session_optics: find a data file's optics file
+      cli.py        scigantic-headers <file|url|--dir>
+      benchmark.py  scigantic-headers-bench, measures the speed levers
+    tests/          pytest (117 tests, incl. fuzz + golden fixtures)
+
+## Robustness
+
+A decoder is handed the leading bytes of arbitrary files, so it must be total:
+for any input it returns None or a valid, JSON-safe DecodedHeader. It never
+raises and never emits NaN or inf. This is enforced, not assumed.
+
+- **Fuzz** (`test_fuzz.py`): thousands of random and magic-seeded byte buffers
+  through every decoder. Each result must serialize with `allow_nan=False`.
+- **Golden fixtures** (`fixtures/mrc-cases.json`): hex input to exact decode.
+  This is the single source of truth the three MRC decoders (this library, the
+  TypeScript twin, scigantic_empiar) are all held to, so none can silently
+  diverge.
+- Non-finite floats (a garbage NIfTI pixdim, an inf MRC cell) sanitize to null.
+
+## Speed
+
+The decode is microseconds: a 1 KiB read and a few `struct` unpacks. Optimizing
+that is a rounding error. All the speed is in I/O, and two things carry it. Both
+are measured, not asserted (run `scigantic-headers-bench`).
+
+**Read the header, not the file.** The read is exactly `HEADER_BYTES` (1 KiB),
+so its cost is independent of file size and a multi-GB movie decodes as fast as
+a small one.
+
+    header read + decode (1 KiB):        15 us
+    full-file read + decode (200 MB):  22.7 ms      (about 1,500x cheaper)
+
+**Parallelize the I/O, bounded.** Header reads across files are independent and
+I/O-bound, so a thread pool overlaps their latency (the read releases the GIL).
+Measured over 8 real EMPIAR headers pulled by HTTP Range:
+
+    serial   (1 worker):   25.7 s
+    parallel (8 workers):   0.7 s        (about 35x)
+
+That 35x is large because EBI's per-request latency is about 3 s, so overlapping
+eight requests recovers a lot of dead wait. On low-latency local storage the
+speedup is smaller. The win scales with per-read latency, which is why the pool
+is bounded and tunable (default 8, the EMPIAR sweet spot; past a point more
+connections hit server throttling or disk-queue thrash and get slower).
+
+Two more, by construction:
+
+- **Filter before I/O.** `has_decoder_for` is a cheap extension check. The batch
+  and walk paths use it to skip a file before any syscall.
+- **Zero deps, fast import.** The dtype table is a plain dict, not numpy. There
+  is nothing to install and nothing to import, and it runs air-gapped.
+
+The biggest system-level lever lives in the caller, not here: decode once when a
+file lands, write the context record, and answer every later query from the tiny
+record without re-decoding. The edge context-producer and the cloud probe both
+do this.
+
+## Use
+
+    from scigantic_headers import decode_file, decode_paths, decode_urls, iter_decodable_files
+
+    decode_file("s/frame.mrc")                              # one local file
+    decode_paths(iter_decodable_files("/mnt/sessions"))     # a tree, parallel
+    decode_urls([...], workers=8)                           # remote, by Range
+
+    # CLI
+    scigantic-headers session/frame.mrc
+    scigantic-headers https://ftp.ebi.ac.uk/empiar/.../img.mrcs
+    scigantic-headers --dir /mnt/flashblade/sessions --workers 8
+
+    pytest
+    scigantic-headers-bench --empiar   # reproduce the numbers above
+
+## Limitations
+
+- **Pointer-based formats are not supported.** HDF5 and TIFF store their layout
+  behind internal pointers, so reading the header or footer alone is not enough.
+  Parquet (a footer) is supported; formats that need to chase offsets through the
+  file are not. Plain gzip is handled (the reader inflates the leading block),
+  but BAM's BGZF framing and per-record structure are not.
+- **Output is not a standard.** The fields are an ad-hoc dict, not an
+  interchange format such as Allotrope ASM. Map the dict to a standard if you
+  need one.
+
+## Repository
+
+This is the standalone home of the library. It is also vendored into the
+Scigantic monorepo, where the paths above (`backend/...`, `infrastructure/...`)
+point at the TypeScript twin, the notebook, and the edge context-producer that
+use it.

scigantic_headers-0.1.0/pyproject.toml

@@ -0,0 +1,32 @@
+[build-system]
+requires = ["setuptools>=61", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "scigantic-headers"
+version = "0.1.0"
+description = "Decode scientific file headers (MRC cryo-EM, extensible) into typed context for agents — pure, zero-dependency, with bounded parallel readers."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "Proprietary — Evaluation Use Only. See LICENSE." }
+authors = [{ name = "Scigantic" }]
+keywords = ["cryo-em", "mrc", "header", "schema-card", "instrument", "context", "edge"]
+classifiers = [
+  "License :: Other/Proprietary License",
+  "Programming Language :: Python :: 3",
+]
+# Zero runtime dependencies is a feature: installs and runs on an air-gapped box.
+dependencies = []
+
+[project.optional-dependencies]
+dev = ["pytest>=7"]
+
+[project.scripts]
+scigantic-headers = "scigantic_headers.cli:main"
+scigantic-headers-bench = "scigantic_headers.benchmark:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

scigantic_headers-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

scigantic_headers-0.1.0/src/scigantic_headers/__init__.py

@@ -0,0 +1,52 @@
+"""scigantic-headers, decode scientific file headers into typed context.
+
+Bytes in, typed fields out. Pure, zero-dependency core (decoders) plus bounded
+parallel I/O readers (sources). One implementation shared by the cloud probe's
+twin, the notebook, and the edge box, so a file is described the same way
+everywhere.
+
+    from scigantic_headers import decode_file, decode_paths
+    hdr = decode_file("session/frame.mrc")
+    print(hdr.summary)  # "MRC stack 4096x4096x16, float32"
+"""
+
+from .cryosparc import parse_cryosparc_optics, read_cryosparc_optics
+from .optics import read_session_optics
+from .parquet import decode_parquet
+from .decoders import (
+    HEADER_BYTES,
+    DecodedHeader,
+    decode_bytes,
+    decode_cryosparc_header,
+    decode_mrc_header,
+    decode_nifti_header,
+    decode_npy_header,
+    extension_of,
+    has_decoder_for,
+    register_decoder,
+)
+from .sources import (
+    DEFAULT_WORKERS,
+    decode_file,
+    decode_paths,
+    decode_url,
+    decode_urls,
+    is_decodable,
+    iter_decodable_files,
+    read_leading_bytes,
+    read_leading_bytes_url,
+)
+from .star import parse_relion_optics, read_star_optics
+
+__version__ = "0.1.0"
+__all__ = [
+    "HEADER_BYTES", "DecodedHeader", "decode_bytes", "decode_mrc_header",
+    "decode_nifti_header", "decode_npy_header", "decode_cryosparc_header",
+    "parse_cryosparc_optics", "read_cryosparc_optics", "read_session_optics",
+    "decode_parquet",
+    "extension_of", "has_decoder_for", "register_decoder",
+    "DEFAULT_WORKERS", "decode_file", "decode_paths", "decode_url",
+    "decode_urls", "is_decodable", "iter_decodable_files", "read_leading_bytes",
+    "read_leading_bytes_url", "parse_relion_optics", "read_star_optics",
+    "__version__",
+]

scigantic_headers-0.1.0/src/scigantic_headers/benchmark.py

@@ -0,0 +1,107 @@
+"""Measure where the time actually goes, honestly, not by assertion.
+
+Two things worth measuring, matching the two real speed levers:
+
+  1. Bounded read vs full read. Decoding reads HEADER_BYTES; it does not read
+     the file. This shows the header read is ~constant and tiny while a full
+     read scales with file size, the single biggest win, and the reason a
+     multi-GB movie costs the same to decode as a 1 KB one.
+
+  2. Serial vs bounded-parallel over many *remote* files. Header reads are
+     independent and I/O-bound, so a thread pool overlaps their latency. Run
+     against real EMPIAR URLs so the numbers are real network latency, not a
+     simulation, this reproduces the EMPIAR range-reader's finding at the
+     file-set level.
+
+    scigantic-headers-bench                 # local read-size bench (offline)
+    scigantic-headers-bench --empiar        # + remote serial-vs-parallel (network)
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import struct
+import tempfile
+import time
+
+from .decoders import decode_bytes
+from .sources import DEFAULT_WORKERS, decode_file, decode_urls, read_leading_bytes
+
+# A handful of real EMPIAR-10002 movie stacks (public, no auth). Header pulled
+# via Range, nothing is downloaded.
+_EMPIAR_BASE = "https://ftp.ebi.ac.uk/empiar/world_availability/10002/data"
+_EMPIAR_FILES = [
+    "15_movie_gc.mrcs", "16_movie_gc.mrcs", "17_movie_gc.mrcs", "18_movie_gc.mrcs",
+    "20_movie_gc.mrcs", "21_movie_gc.mrcs", "22_movie_gc.mrcs", "23_movie_gc.mrcs",
+]
+
+
+def _make_mrc_file(path: str, size_mb: int) -> None:
+    """Write a valid-header MRC file of `size_mb` megabytes so we can time a full
+    read against a bounded read at a realistic size."""
+    buf = bytearray(1024)
+    struct.pack_into("<4i", buf, 0, 4096, 4096, 16, 2)
+    struct.pack_into("<i", buf, 28, 4096)
+    buf[208:212] = b"MAP "
+    with open(path, "wb") as fh:
+        fh.write(buf)
+        fh.write(b"\0" * (size_mb * 1024 * 1024))
+
+
+def _time(fn, repeat=5):
+    best = float("inf")
+    for _ in range(repeat):
+        t0 = time.perf_counter()
+        fn()
+        best = min(best, time.perf_counter() - t0)
+    return best
+
+
+def bench_read_size(size_mb: int = 200) -> None:
+    print(f"\n[1] bounded read vs full read ({size_mb} MB file)")
+    with tempfile.TemporaryDirectory() as d:
+        path = os.path.join(d, "big.mrc")
+        _make_mrc_file(path, size_mb)
+
+        def bounded():
+            decode_bytes(path, read_leading_bytes(path))
+
+        def full():
+            with open(path, "rb") as fh:
+                decode_bytes(path, fh.read())
+
+        tb = _time(bounded)
+        tf = _time(full)
+        print(f"    header read + decode (1 KiB): {tb * 1e6:8.1f} us")
+        print(f"    full file  read + decode ({size_mb} MB): {tf * 1e3:8.1f} ms")
+        print(f"    bounded read is ~{tf / tb:,.0f}x cheaper, and flat in file size")
+
+
+def bench_parallel_remote(workers: int = DEFAULT_WORKERS) -> None:
+    urls = [f"{_EMPIAR_BASE}/{name}" for name in _EMPIAR_FILES]
+    print(f"\n[2] serial vs {workers}-way parallel over {len(urls)} real EMPIAR headers (network)")
+    t_serial = _time(lambda: decode_urls(urls, workers=1), repeat=1)
+    t_par = _time(lambda: decode_urls(urls, workers=workers), repeat=1)
+    got = len(decode_urls(urls, workers=workers))
+    print(f"    serial (1 worker):        {t_serial:6.2f} s")
+    print(f"    parallel ({workers} workers):     {t_par:6.2f} s")
+    if t_par > 0:
+        print(f"    speedup: {t_serial / t_par:.1f}x   ({got}/{len(urls)} decoded)")
+
+
+def main(argv=None) -> int:
+    ap = argparse.ArgumentParser(prog="scigantic-headers-bench", description=__doc__)
+    ap.add_argument("--size-mb", type=int, default=200, help="file size for the read-size bench")
+    ap.add_argument("--empiar", action="store_true", help="also run the remote parallel bench (needs network)")
+    ap.add_argument("--workers", type=int, default=DEFAULT_WORKERS)
+    args = ap.parse_args(argv)
+
+    bench_read_size(args.size_mb)
+    if args.empiar:
+        bench_parallel_remote(args.workers)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scigantic_headers-0.1.0/src/scigantic_headers/cli.py

@@ -0,0 +1,50 @@
+"""scigantic-headers, decode a file or URL header into typed fields.
+
+    scigantic-headers session/frame.mrc
+    scigantic-headers https://ftp.ebi.ac.uk/empiar/.../img.mrcs
+    scigantic-headers --dir /mnt/flashblade/sessions --workers 8
+
+A URL is read with an HTTP Range request, so a header is pulled from a remote
+archive without downloading the file.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+
+from .decoders import has_decoder_for
+from .sources import decode_file, decode_paths, decode_url, iter_decodable_files
+
+
+def main(argv=None) -> int:
+    ap = argparse.ArgumentParser(prog="scigantic-headers", description=__doc__)
+    ap.add_argument("source", nargs="?", help="a file path or http(s) URL")
+    ap.add_argument("--dir", help="decode every recognized file under this directory")
+    ap.add_argument("--workers", type=int, default=8, help="parallel workers for --dir")
+    args = ap.parse_args(argv)
+
+    if args.dir:
+        results = decode_paths(iter_decodable_files(args.dir), workers=args.workers)
+        print(json.dumps(
+            {p: h.to_dict() for p, h in results.items()}, indent=2, default=str))
+        print(f"# {len(results)} decoded", file=sys.stderr)
+        return 0
+
+    if not args.source:
+        ap.error("give a file/URL, or --dir")
+    if not has_decoder_for(args.source):
+        print(f"no decoder for '{args.source}' (extension not recognized)", file=sys.stderr)
+        return 1
+
+    hdr = decode_url(args.source) if args.source.startswith(("http://", "https://")) else decode_file(args.source)
+    if hdr is None:
+        print(f"header did not decode for {args.source} (invalid or unreadable)", file=sys.stderr)
+        return 1
+    print(json.dumps({"source": args.source, **hdr.to_dict()}, indent=2, default=str))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())