lfpack 0.2.0__tar.gz → 0.4.0__tar.gz

{lfpack-0.2.0 → lfpack-0.4.0}/.gitignore

@@ -1,4 +1,5 @@
 TODO.md
+.DS_Store
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

lfpack-0.4.0/CHANGELOG.md

@@ -0,0 +1,62 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.4.0] - 2026-08-23
+
+### Added
+- Non-linear sync support: `sync_samples`/`sync_times` — the raw ALF sync knot pairs
+  (sample index <-> time), stored verbatim per scale — alongside the existing
+  `t0_sync`/`fs_sync` scalars, which are now always a least-squares affine derived from
+  the knots (or the legacy scalar, for archives predating this change). `LFPackReader`
+  interpolates through the knots for `.times`/`saturation_times()` inside their range and
+  falls back to the derived affine outside it (`interp1d` with no extrapolation, rather
+  than the less stable slope of the two outermost knots). New `lfpack.write_sync_attrs` /
+  `lfpack.clear_sync_attrs` (`src/lfpack/_sync.py`) write/clear both tiers together and
+  validate the knots (>=2 points, strictly increasing, all-finite). `sync_samples`/
+  `sync_times` are stored as gzip+shuffle child datasets of `meta`, not attrs — a
+  `type='exact'` fit's knots (every raw pulse, verbatim) can run to tens of thousands of
+  points, well past HDF5's per-attribute object-header-message size limit.
+- Added `docs/how-to/write-your-own-file.qmd`, documenting the two sync styles (linear
+  affine vs. piecewise knots) and moving/fixing the multi-recording write example that
+  used to live in `how-to/multi-recording.qmd`.
+
+### Fixed
+- `saturation_times()` now calls the same sample→time conversion as `.times` instead of
+  duplicating the affine formula inline, so the two could never silently diverge on a
+  scale with real non-linear sync structure.
+
+## [0.3.0] - 2026-07-25
+
+### Added
+- `subset_h5` — copy a subset of recordings out of a multi-recording HDF5 archive
+  (inverse of `merge_h5`), for carving a smaller release (e.g. BWM) out of a larger
+  superset (e.g. ephys-atlas) without re-compression.
+- Bad-channel labels (0=good, 1=dead, 2=noisy, 3=outside brain) persisted through
+  compression and exposed via `LFPackReader.channels` / `channels_full`.
+- **Saturation detection and muting**: ADC-clipped spans are detected on the raw LFP
+  band, stored as a per-recording interval table, and muted (cosine-tapered zero) on
+  the decimated output after Cadzow denoising. Exposed via `LFPackReader.saturation`
+  (raw-rate sample indices, recording-aligned) and `.saturation_summary`.
+  `.saturation_mask` is a property, indexed like the reader itself (`sr.saturation_mask[a:b]`);
+  `.saturation_times()` converts samples to session-clock seconds (matching `times`).
+
+### Fixed
+- `compress` no longer decompresses low-SNR chunks to exact zero; new `floor_k=64`
+  survival floor keeps the dominant mode's largest coefficients. Closes #2.
+
+## [0.2.0] - 2026-06-30
+
+### Added
+- `LFPackReader.channels` and `LFPackReader.channels_full` — per-channel probe geometry
+  and brain location annotations (`x/y/z` MNI coordinates, `atlas_id`, `acronym`).
+  Brain location fields are optional; the properties work on any existing file.
+  `channels` aggregates over binned-channel groups (mean for coordinates, mode for
+  brain region); `channels_full` always returns the raw per-electrode data.
+- `compress_to_h5` accepts an optional `channels` dict to embed brain location
+  annotations at compression time.

lfpack-0.4.0/CLAUDE.md

@@ -0,0 +1,97 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this project is
+
+**lfpack** is a lossy LFP (local-field-potential) codec for Neuropixels recordings. It compresses raw `.cbin` LFP binaries into HDF5 files at >100× compression with RMSE < 25 µV, using an 8-stage pipeline followed by adaptive SVD + wavelet-packet thresholding.
+
+## Commands
+
+```bash
+# Dev setup
+uv sync --group dev
+git config core.hooksPath .githooks   # installs ruff pre-commit hook
+
+# Tests
+uv run pytest                          # all tests
+uv run pytest tests/test_lfpack.py::TestLfpack::test_compress_output_shapes  # single test
+
+# Lint / format
+uv run ruff check src/ tests/
+uv run ruff format src/ tests/
+
+# Docs — regenerate API reference from docstrings, then preview
+uv run quartodoc build --config docs/_quarto.yml
+quarto preview docs/
+```
+
+## Architecture
+
+Nearly all code lives in `src/lfpack/_core.py` (~1700 lines). Non-linear sync knot storage (`write_sync_attrs`/`clear_sync_attrs`) lives in the small `src/lfpack/_sync.py` module. The public API is re-exported from `src/lfpack/__init__.py`.
+
+### Compression pipeline (`compress_bin_to_h5`)
+
+0. Saturation detection — flag ADC-clipped samples on the raw LFP band *before* any step below obscures them (`ibldsp.voltage.saturation_cbin`, parallel over `n_jobs`). Detection is **amplitude-only** for LFP (`v_per_sec=None`): the derivative criterion is tuned for the 30 kHz AP band and mislabels normal LFP dynamics as saturation, so it is disabled here. Stored as an insertion-level interval table (see HDF5 layout) and used to mute the clipped stretches; toggle with `detect_saturation` / `saturation_kwargs`.
+1. Bad-channel detection (`ibldsp.voltage.detect_bad_channels_cbin`) — labels saved to the scale-`00` `meta` as a `labels` int8 attr; read via `LFPackReader.channels`/`channels_full` under the `labels` key.
+2. Dephasing — sample-shift correction (NP1 only, via `ibldsp.fourier.fshift`)
+3. Highpass filter — 0.5 Hz zero-phase 3rd-order Butterworth (keeps delta/infra-slow; ibldsp warmup padding scales with the corner). The true recording start/end are cosine-apodized (~3 filter time-constants) before this filter so the zero-phase transient cannot ring against the data-boundary step (interior chunk edges are handled by the warmup padding).
+4. Bad-channel interpolation — distance-weighted neighbors
+5. CAR — median subtraction across channels
+6. Decimation — 2500 → 250 Hz (Q=10)
+7. Cadzow denoising — spatial rank reduction (`ibldsp.cadzow`)
+8. **Adaptive SVD + wavelet-packet thresholding** — the actual codec
+
+Steps 6–7 are checkpointed to a `.npy` file so the expensive decimation can be skipped on resume. Steps 7–8 are parallelised with `joblib`. Muting (step 0): a raw-rate boolean saturation mask is passed to `resample_denoise_lfp_cbin` as `saturation_file`; each worker downsamples its slice of the mask onto the decimated grid (block-max over each Q-sample block), cosine-tapers it (`mute_window_samples`, now in **decimated** output samples) and multiplies the **decimated output *after* Cadzow** by it. This *late* muting replaced the earlier pre-filter approach: muting the raw traces before the highpass/CAR/decimate/Cadzow did not keep the span clean — those stages leaked energy back into it (residual noise, spurious CSD sources/sinks); a single re-mute of the final output is cleaner and cheaper. Muting is skipped when resuming from an existing checkpoint (data already decimated); the interval table is still written.
+
+### Codec (`compress` / `decompress`)
+
+`compress(data, epsilon=150, alpha=28)` → `LFPCompressed` dataclass:
+- **SVD rank selection**: keep singular values > `epsilon × sigma_noise`
+- **Wavelet-packet thresholding**: db4, level 5; per-component threshold `alpha × sigma_noise / sv[k]`
+- Sparse Vh stored as `(vh_indices, vh_values)` in HDF5; `U_scaled` with shuffle+gzip
+
+Guard bands: 64-sample Cadzow halos, 128-sample SVD/WP overlap to prevent edge transients.
+
+### HDF5 layout (multi-recording, pyramidal)
+
+```
+<file>.h5
+└─ <recording>/
+   ├─ saturation   # (n_intervals, 2) int64 [start_sample, stop_sample] at raw LFP fs;
+   │               # written once per recording (scale-independent). attrs: fs, ns_total,
+   │               # n_saturated_samples, saturated_fraction, detection params, muted
+   └─ <scale_2digit>/
+      ├─ meta         # attrs: nc, ns_total, fs, fs_sync, t0_sync, epsilon, alpha, geometry, …
+      │               # optional child datasets: sync_samples, sync_times (gzip+shuffle)
+      └─ chunks/
+         └─ <i>/      # U_scaled, vh_indices, vh_values + attrs
+```
+
+The `saturation` node sits at recording level (not under a scale) because it describes the raw recording, not a codec pyramid level; `merge_h5` copies it automatically. Legacy flat layout (meta at root) is still readable for backwards compatibility.
+
+`fs_sync`/`t0_sync` are always plain scalar attrs — a least-squares affine (derived from
+`sync_samples`/`sync_times` when present, else the legacy scalar sync, else the nominal
+rate) — so downstream consumers reading `meta.attrs` directly (e.g. `qc_report.py`) and
+`LFPackReader.t0`/`.fs` never see anything but a scalar. `sync_samples`/`sync_times`
+(float64, optional) are the raw ALF sync knot pairs (sample index <-> time) in this
+scale's own native sample-index units, written verbatim with no re-fitting; `LFPackReader`
+interpolates through them for `.times`/`saturation_times()` inside their range and falls
+back to the derived affine outside it. Stored as **child datasets** of `meta`, not attrs
+like everything else here — a `type='exact'` fit keeps every raw pulse verbatim, which can
+run to tens of thousands of knots and overflows HDF5's per-attribute object-header-message
+size limit (`OSError: object header message is too large`, hit at ~38k knots in production).
+Write/clear both tiers together via `lfpack.write_sync_attrs(meta_group, sample_knots,
+time_knots)` / `lfpack.clear_sync_attrs(meta_group)` (in `src/lfpack/_sync.py`) rather than
+setting them by hand.
+
+### `LFPackReader`
+
+Drop-in replacement for `spikeglx.Reader`. Wraps an HDF5 file and decompresses chunks on demand. Supports multi-recording files via `recording=` kwarg and pyramidal scales via `scale=` kwarg. Saturation access: `.saturation` (interval DataFrame with raw samples + seconds), `.saturation_summary` (fraction/count/muted from attrs), `saturation_mask(first, last)` (boolean at the reader's decimated rate, interval edges rounded outward). All three degrade gracefully to empty/default on files written without saturation detection.
+
+## Ecosystem interconnections
+
+lfpack sits in a tight three-way dependency with two sibling IBL packages:
+
+- **ibl-neuropixel** (`/Users/olivier/PycharmProjects/ephys-atlas/ibl-neuropixel`) — provides the low-level Neuropixels binary I/O (`spikeglx.Reader`), destriping, dephasing (`ibldsp.fourier.fshift`), decimation (`ibldsp.voltage.resample_denoise_lfp_cbin`), bad-channel detection (`ibldsp.voltage.detect_bad_channels_cbin`), saturation detection (`ibldsp.voltage.saturation_cbin` / `saturation` / `saturation_samples_to_intervals`), and Cadzow denoising (`ibldsp.cadzow`). lfpack wraps these directly; changes to ibldsp can break the pipeline. `resample_denoise_lfp_cbin` gained a `saturation_file` / `mute_window_samples` parameter (added for this codec) to late-mute saturated stretches on the decimated output after Cadzow.
+- **viewephys** (`/Users/olivier/PycharmProjects/ephys-atlas/viewephys`) — Qt-based interactive viewer for raw Neuropixels traces. Since [PR #49](https://github.com/int-brain-lab/viewephys/pull/49) it has a native lfpack backend (`LFPackDataModel` / `LFPackBinViewer`, optional `viewephys[lfpack]` extra): `viewephys -f file.h5` opens a `.h5` file directly, with automatic brain-region colouring from embedded `atlas_id`/`acronym` annotations, a searchable multi-recording selector, and a CSD step. No manual transpose or `BrainRegions` wiring needed.

{lfpack-0.2.0 → lfpack-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: lfpack
-Version: 0.2.0
+Version: 0.4.0
 Summary: LFP codec — lossy compression of local-field-potential recordings via adaptive SVD and wavelet-packet thresholding
 Project-URL: Homepage, https://github.com/int-brain-lab/lfpack
 Project-URL: Bug Tracker, https://github.com/int-brain-lab/lfpack/issues
@@ -19,9 +19,6 @@ Description-Content-Type: text/markdown
 
 # lfpack — LFP codec for Neuropixels recordings
 
-> **IBL Brain-Wide Map LFP dataset** — 699 recordings, 384 channels, session-clock aligned.
-> [How to access →](https://int-brain-lab.github.io/lfpack/how-to/bwm-dataset.html)
-
 <p align="center">
   <img src="docs/figures/logo.png" alt="lfpack logo" width="300"/>
 </p>
@@ -30,10 +27,25 @@ Lossy codec for local-field-potential (LFP) recordings from Neuropixels probes.
 Achieves **>100× compression** with median RMSE < 25 µV via an 8-stage pipeline
 (bad-channel detection → dephasing → highpass → interpolation → CAR → decimation → Cadzow → adaptive SVD + wavelet-packet thresholding).
 
+> **IBL Brain-Wide Map LFP dataset** — 699 recordings, 384 channels, session-clock aligned.
+> [How to access →](https://int-brain-lab.github.io/lfpack/how-to/bwm-dataset.html)
+
 ```bash
 pip install lfpack
 ```
 
+## Visualise with viewephys
+
+[`viewephys`](https://github.com/int-brain-lab/viewephys) opens lfpack `.h5` files natively —
+`viewephys -f recording.h5` gets you a browsable, brain-region-coloured view with no
+manual decompression step:
+
+<p align="center">
+  <img src="docs/figures/viewephys_screenshot.jpg" alt="viewephys opened directly on a lfpack HDF5 file, showing brain-region-coloured LFP traces" width="700"/>
+</p>
+
+See the [BWM how-to](https://int-brain-lab.github.io/lfpack/how-to/bwm-dataset.html#visualise-with-viewephys) for details.
+
 ## Documentation
 
 Full documentation is at **https://int-brain-lab.github.io/lfpack/**.
@@ -46,4 +58,10 @@ Full documentation is at **https://int-brain-lab.github.io/lfpack/**.
 | [API reference](https://int-brain-lab.github.io/lfpack/reference/) | Full public API (`compress_bin_to_h5`, `LFPackReader`, …) |
 | [HDF5 format](https://int-brain-lab.github.io/lfpack/reference/hdf5-layout.html) | On-disk layout specification |
 | [Pipeline explanation](https://int-brain-lab.github.io/lfpack/explanation/pipeline.html) | Stage-by-stage description of the compression pipeline |
-| [SVD+WP benchmark](https://int-brain-lab.github.io/lfpack/explanation/benchmark.html) | RMSE, SNR, and compression-ratio results across 11 insertions |
+| [SVD+WP benchmark](https://int-brain-lab.github.io/lfpack/explanation/benchmark.html) | RMSE, SNR, and compression-ratio results across 11 insertions |
+
+## Help and Feedback
+
+Have a question, found an issue, or want to share feedback? Please open an issue on
+[GitHub](https://github.com/int-brain-lab/lfpack/issues). We welcome bug reports, questions,
+suggestions, and other feedback.

{lfpack-0.2.0 → lfpack-0.4.0}/README.md

@@ -1,8 +1,5 @@
 # lfpack — LFP codec for Neuropixels recordings
 
-> **IBL Brain-Wide Map LFP dataset** — 699 recordings, 384 channels, session-clock aligned.
-> [How to access →](https://int-brain-lab.github.io/lfpack/how-to/bwm-dataset.html)
-
 <p align="center">
   <img src="docs/figures/logo.png" alt="lfpack logo" width="300"/>
 </p>
@@ -11,10 +8,25 @@ Lossy codec for local-field-potential (LFP) recordings from Neuropixels probes.
 Achieves **>100× compression** with median RMSE < 25 µV via an 8-stage pipeline
 (bad-channel detection → dephasing → highpass → interpolation → CAR → decimation → Cadzow → adaptive SVD + wavelet-packet thresholding).
 
+> **IBL Brain-Wide Map LFP dataset** — 699 recordings, 384 channels, session-clock aligned.
+> [How to access →](https://int-brain-lab.github.io/lfpack/how-to/bwm-dataset.html)
+
 ```bash
 pip install lfpack
 ```
 
+## Visualise with viewephys
+
+[`viewephys`](https://github.com/int-brain-lab/viewephys) opens lfpack `.h5` files natively —
+`viewephys -f recording.h5` gets you a browsable, brain-region-coloured view with no
+manual decompression step:
+
+<p align="center">
+  <img src="docs/figures/viewephys_screenshot.jpg" alt="viewephys opened directly on a lfpack HDF5 file, showing brain-region-coloured LFP traces" width="700"/>
+</p>
+
+See the [BWM how-to](https://int-brain-lab.github.io/lfpack/how-to/bwm-dataset.html#visualise-with-viewephys) for details.
+
 ## Documentation
 
 Full documentation is at **https://int-brain-lab.github.io/lfpack/**.
@@ -27,4 +39,10 @@ Full documentation is at **https://int-brain-lab.github.io/lfpack/**.
 | [API reference](https://int-brain-lab.github.io/lfpack/reference/) | Full public API (`compress_bin_to_h5`, `LFPackReader`, …) |
 | [HDF5 format](https://int-brain-lab.github.io/lfpack/reference/hdf5-layout.html) | On-disk layout specification |
 | [Pipeline explanation](https://int-brain-lab.github.io/lfpack/explanation/pipeline.html) | Stage-by-stage description of the compression pipeline |
-| [SVD+WP benchmark](https://int-brain-lab.github.io/lfpack/explanation/benchmark.html) | RMSE, SNR, and compression-ratio results across 11 insertions |
+| [SVD+WP benchmark](https://int-brain-lab.github.io/lfpack/explanation/benchmark.html) | RMSE, SNR, and compression-ratio results across 11 insertions |
+
+## Help and Feedback
+
+Have a question, found an issue, or want to share feedback? Please open an issue on
+[GitHub](https://github.com/int-brain-lab/lfpack/issues). We welcome bug reports, questions,
+suggestions, and other feedback.

{lfpack-0.2.0 → lfpack-0.4.0}/docs/_quarto.yml

@@ -16,6 +16,8 @@ website:
         file: tutorials/first-compression.qmd
       - text: "How-To"
         menu:
+          - text: "Write your own file"
+            file: how-to/write-your-own-file.qmd
           - text: "Binned-channel reads"
             file: how-to/binned-reads.qmd
           - text: "Multi-recording files"
@@ -33,6 +35,9 @@ website:
           - text: "Benchmark gallery"
             file: explanation/gallery.qmd
     right:
+      - text: "Help"
+        icon: question-circle
+        href: https://github.com/int-brain-lab/lfpack/issues
       - icon: github
         href: https://github.com/int-brain-lab/lfpack
 
@@ -65,4 +70,9 @@ quartodoc:
     - title: "Reader"
       desc: "Random-access decompression."
       contents:
-        - LFPackReader
+        - LFPackReader
+    - title: "Multi-recording archives"
+      desc: "Combine or subset recordings across HDF5 files without re-compression."
+      contents:
+        - merge_h5
+        - subset_h5

{lfpack-0.2.0 → lfpack-0.4.0}/docs/explanation/pipeline.qmd

@@ -139,7 +139,9 @@ This ensures dominant spatial modes receive finer temporal detail than weak mode
 
 Non-zero coefficients are stored as sparse index/value pairs in HDF5.
 
-**Parameters:** `epsilon` — SVD noise threshold multiplier (default `150`); `alpha` — WP threshold multiplier (default `28`)
+**Survival floor.** Because each row of $V_r^\top$ has unit norm, its wavelet-packet coefficients have a fixed magnitude ceiling ($\max_j|w_{kj}| \approx 0.2\text{–}0.4$). On low-SNR chunks — where even the dominant mode is weak, $s_0/\sigma_\text{noise} \lesssim \alpha/\max_j|w_{0j}| \approx 140$ — the threshold exceeds every coefficient and the whole chunk reconstructs to exact zero, silently discarding real low-amplitude LFP. To prevent this the dominant mode ($k=0$) always keeps its `floor_k` largest coefficients. The floor is a no-op on high-SNR recordings (their dominant row keeps far more than `floor_k`, so their output is bit-for-bit unchanged) and on saturation-muted spans, which enter as all-zero and still reconstruct to zero ($s_0 = 0 \Rightarrow$ `U_scaled` $= 0$).
+
+**Parameters:** `epsilon` — SVD noise threshold multiplier (default `150`); `alpha` — WP threshold multiplier (default `28`); `floor_k` — minimum WP coefficients kept on the dominant mode (default `64`)
 
 ### Guard bands
 

lfpack-0.4.0/docs/how-to/bwm-dataset.qmd

@@ -0,0 +1,272 @@
+---
+title: "How to access the IBL Brain-Wide Map LFP dataset"
+---
+
+The IBL Brain-Wide Map (BWM) LFP dataset is distributed as a 16GB `.h5` file
+containing 699 individual recordings of 384 channels each.
+
+
+::: {.callout-warning}
+## Known Dataset limitations
+
+- **Saturation events** — ADC-clipped stretches are flagged and muted in the decompressed
+  output; check whether your window of interest overlaps one before trusting amplitude.
+  See [Quality Control - Saturation](#quality-control---saturation).
+- **Missing sync for 7 probes** — `t0_sync`/`fs_sync` could not be computed for 7 probes
+  (spanning 4 sessions) due to a session-level sync data-quality issue upstream; `sr.t0`
+  returns `NaN` for these. See [lfpack#8](https://github.com/int-brain-lab/lfpack/issues/8).
+- **High-frequency roll-off** — the SVD + wavelet-packet codec trades off power above
+  roughly 20-30 Hz for compression ratio, more so at the aggressive tier. See the PSD
+  comparison below.
+- **Low-frequency channel-to-channel mismatch** — per-channel impedance/analog-filter
+  differences show up as amplitude outliers at low frequencies (below 1.5 Hz) and are not fully
+  corrected by the current pre-processing pipeline.
+
+![Average PSD across compression tiers for a representative recording — high frequencies are increasingly attenuated as compression ratio (CR) rises from cadzow-only to default (119×) to aggressive (254×).](../figures/bwm_compression_hf_rolloff.png){.lightbox}
+:::
+
+::: {.callout-important}
+## Before you start
+
+- **Align with trial events via `sr.times`** — each recording stores a sync-corrected
+  session clock; always use `sr.times` rather than a manual `np.arange / sr.fs` when
+  matching LFP samples to spike times or trial events.
+  See [Session-clock times](#session-clock-times).
+
+- **Use binned reads for brainwide sweeps** — looping over all 699 recordings at full
+  384-channel resolution is memory-intensive. Pass `bin_channels=4` to reduce each
+  recording to 96 spatial bins with acceptable loss of depth coverage.
+  See [Binned-channel reads](#binned-channel-reads).
+:::
+
+## Visualise with viewephys
+
+`viewephys` has a native lfpack backend — point it at the `.h5` file directly,
+no manual transpose or `BrainRegions` wiring needed:
+
+```bash
+viewephys -f lf_compressed_all_bwm.h5
+```
+
+The `.h5` extension is auto-detected and opens the lfpack-aware viewer, which:
+
+- shows a searchable recording selector (type any `pid` substring) when the
+  file holds multiple recordings, preserving window size, position and zoom
+  when you switch
+- colours the depth axis by brain region automatically, reading the
+  `atlas_id` / `acronym` annotations embedded in the file — no separate
+  `iblatlas` call required
+- adds a **CSD** (current-source density) step alongside the raw signal
+
+![Recording selector (top right) and raw/CSD checkboxes after opening a multi-recording lfpack file — the dataset info panel confirms the native "Neuropixels LFP (lfpack)" backend](../figures/viewephys_recording_selector.jpg){fig-alt="Ephys Bin Viewer window showing the searchable recording dropdown and raw/CSD checkboxes for an lfpack HDF5 file" .lightbox}
+
+The same class is usable from Python:
+
+```python
+from viewephys.gui import LFPackBinViewer
+from viewephys.viewer.qt import create_app
+
+app = create_app()
+win = LFPackBinViewer("lf_compressed_all_bwm.h5")
+app.exec()
+```
+
+![viewephys opened directly on a BWM lfpack file — brain regions colour the depth axis automatically and the searchable selector switches between the file's 699 recordings](../figures/viewephys_screenshot.jpg){fig-alt="viewephys density view showing 384-channel LFP data with brain region colour bar and a searchable recording selector" .lightbox}
+
+See [viewephys PR #49](https://github.com/int-brain-lab/viewephys/pull/49) for implementation details.
+
+## Prerequisites
+
+```bash
+uv pip install "viewephys[lfpack]" one-api
+```
+
+## Download
+
+From inside the `ibl-ai-agent` repository:
+
+```bash
+uv run python scripts/download_datasets.py --lfp
+```
+
+This downloads `lf_compressed_all_bwm.h5` (~14 GB) into `reports/datasets/bwm_lfp/`
+and writes the path into `data_locations.local.yaml` automatically.
+
+Or, from Python (no AWS credentials needed — uses the ONE public S3 helper):
+
+```python
+from one.remote.aws import s3_download_file
+
+s3_download_file(
+    source="resources/ibl-agent-data/lf_compressed_all_bwm.h5",
+    destination="lf_compressed_all_bwm.h5",
+)
+```
+
+`s3_download_file` defaults to the IBL public bucket, shows a tqdm progress bar,
+and skips the download if the local file already has the correct size.
+
+## Session-clock times
+
+`sr.times` returns a `(ns,)` array of session-clock timestamps (seconds) for every
+LFP sample, derived from the sync signal stored during compression.
+Use it to align LFP traces with trial events or spike times:
+
+```python
+from lfpack import LFPackReader
+import numpy as np
+
+file_lfpack = "lf_compressed_all_bwm.h5"
+pid = 'dab512bd-a02d-4c1f-8dbc-9155a163efc0'
+
+sr = LFPackReader(file_lfpack, recording=pid)
+
+n = int(10 * sr.fs)
+traces = sr[:n, :]      # (n_samples, nc) — first 10 s
+t = sr.times[:n]        # (n_samples,)    — session-clock seconds
+
+# Align with trial events loaded via ONE
+# stim_on = trials['stimOn_times']   # already in session-clock seconds
+# idx = np.searchsorted(t, stim_on)  # nearest LFP sample for each stimulus onset
+```
+
+## List recordings
+
+Each file contains one entry per BWM probe. Recording identifiers are named after the probe identifier UUID `pid`.
+
+```python
+from lfpack import LFPackReader
+
+recordings = LFPackReader.recordings("lf_compressed_all_bwm.h5")
+print(f"{len(recordings)} recordings")
+print(recordings[:3])
+# 699 recordings
+# ['00a824c0-e060-495f-9ebc-79c82fef4c67', '00a96dee-1e8b-44cc-9cc3-aca704d2b594', '00c425fd-ec3e-4cd2-b8af-c0bc0c4bdd44']
+```
+
+## Read a single recording
+
+```python
+from lfpack import LFPackReader
+
+file_lfpack = "lf_compressed_all_bwm.h5"
+pid = 'dab512bd-a02d-4c1f-8dbc-9155a163efc0'
+
+sr = LFPackReader(file_lfpack, recording=pid)
+
+# First 10 seconds — shape (2500, nc), float32, volts
+traces = sr[: int(10 * sr.fs)].T
+
+print(f"Duration:    {sr.ns / sr.fs:.1f} s")
+print(f"Channels:    {sr.nc}")
+print(f"Sample rate: {sr.fs} Hz")   # 250 Hz (decimated from 2500 Hz)
+print(f"Shape:       {traces.shape}")
+
+# Duration:    3668.9 s
+# Channels:    384
+# Sample rate: 250.00252518761928 Hz
+# Shape:       (384, 2500)
+```
+
+`LFPackReader` is a drop-in for `spikeglx.Reader`.
+Slicing decompresses only the requested chunks — the full file is never loaded into memory.
+
+## Quality Control - Saturation
+
+Each recording carries an insertion-level table of ADC-saturated (clipped) spans,
+detected on the raw LFP band before any filtering. The saturated stretches are muted
+(zeroed) in the decompressed output, so it's worth checking whether a window of
+interest overlaps one before trusting the amplitude.
+
+```python
+from lfpack import LFPackReader
+
+sr = LFPackReader("lf_compressed_all_bwm.h5", recording=pid)
+
+# Interval table: raw-rate sample indices, recording-aligned
+sr.saturation
+#    start_sample  stop_sample
+# 0        123456       125000
+
+# Recording-level summary (fraction/count/whether muting was applied)
+sr.saturation_summary
+# {'saturated_fraction': 0.0052, 'n_intervals': 3, 'total_saturated_sec': 12.9, 'muted': True, ...}
+```
+
+`saturation_mask` gives a boolean array at the reader's own — decimated — sampling rate,
+indexed exactly like the reader itself rather than called with a window:
+
+```python
+n = int(10 * sr.fs)
+traces = sr[:n, :]                  # (n, nc)
+mask = sr.saturation_mask[:n]       # (n,) bool, True where saturated
+
+traces_clean = traces[~mask]
+```
+
+For session-clock timestamps of saturated intervals (e.g. to overlay against trial events),
+use `saturation_times` rather than converting `sr.saturation`'s samples to seconds by hand —
+it applies the same sync correction as [`sr.times`](#session-clock-times):
+
+```python
+sr.saturation_times()
+#    start_sample  stop_sample  start_time  stop_time
+# 0        123456       125000    1234.382    1235.000
+```
+
+All four (`saturation`, `saturation_summary`, `saturation_mask`, `saturation_times`) degrade
+gracefully to empty/default values on recordings without saturation detection, so the same
+code works across the whole dataset without a try/except.
+
+
+![Saturation detection across the 699-recording release: per-recording saturated-fraction distribution (left) and the 15 most-affected recordings (right).](../figures/bwm_saturation_detection.png){.lightbox}
+
+
+## Channel brain locations
+
+Each recording stores per-channel brain location annotations embedded in the file —
+no network call needed.  Access them via `sr.channels`:
+
+```python
+from lfpack import LFPackReader
+
+sr = LFPackReader("lf_compressed_all_bwm.h5", recording=pid)
+ch = sr.channels
+
+print(ch["acronym"][:5])   # ['VISp', 'VISp', 'CA1', 'CA1', 'DG']
+print(ch["atlas_id"][:5])  # Allen CCF structure IDs, int32
+print(ch["x"][:5])         # mediolateral MNI coordinates, metres
+print(ch["y"][:5])         # anteroposterior
+print(ch["z"][:5])         # dorsoventral
+```
+
+When using `bin_channels`, `channels` automatically aggregates: coordinates are
+averaged and brain region is the within-group mode:
+
+```python
+sr = LFPackReader("lf_compressed_all_bwm.h5", recording=pid, bin_channels=4)
+ch = sr.channels           # shapes (96,) — one entry per spatial bin
+ch_full = sr.channels_full # shapes (384,) — always raw per-electrode
+```
+
+## Binned-channel reads
+
+Adjacent channels are spatially correlated at LFP frequencies.
+Passing `bin_channels` sums neighbouring channels during decompression, reducing
+memory use. A factor of 4 takes 384 channels down to 96 while keeping depth resolution useful for LFP.
+
+```python
+from lfpack import LFPackReader
+
+file_lfpack = "lf_compressed_all_bwm.h5"
+pid = 'dab512bd-a02d-4c1f-8dbc-9155a163efc0'
+
+sr = LFPackReader(file_lfpack, recording=pid, bin_channels=4)
+
+traces = sr[:, :]      # (ns, 96) — full recording, 96 spatial bins
+print(sr.nc)           # 96
+print(sr.geometry["y"])           # mean depth of each 4-channel group
+print(sr.geometry_full['binned_channel_index'])  # spatial mapping of the original 384 channels into the 96 binned ones
+```
+
+![Full-resolution (384 ch) vs binned ×4 (96 ch) LFP density for the same 4-second window](../figures/lfp_binned_comparison.jpg){fig-alt="Side-by-side LFP density plots at full and 4x binned channel resolution" .lightbox}

{lfpack-0.2.0 → lfpack-0.4.0}/docs/how-to/multi-recording.qmd

@@ -40,19 +40,4 @@ Higher scale indices correspond to progressively coarser spatial representations
 
 ## Write a multi-recording file
 
-Use `compress_to_h5` (not `compress_bin_to_h5`) and call it once per recording, passing the same output path each time:
-
-```python
-from lfpack import compress_to_h5
-import numpy as np
-
-# First probe
-data_a = np.load("probe00_lfp.npy")   # (nc, ns) float32
-compress_to_h5(data_a, fs=250.0, h5_file="session.lf.h5", recording="probe00")
-
-# Second probe — appended to the same file
-data_b = np.load("probe01_lfp.npy")
-compress_to_h5(data_b, fs=250.0, h5_file="session.lf.h5", recording="probe01")
-```
-
-See [`compress_to_h5`](../reference/compress_to_h5.qmd) for the full parameter list.
+See [how to write your own lfpack file](write-your-own-file.qmd#write-a-multi-recording-file).