lfpack 0.2.0__tar.gz → 0.3.0__tar.gz

{lfpack-0.2.0 → lfpack-0.3.0}/.gitignore

@@ -1,4 +1,5 @@
 TODO.md
+.DS_Store
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

lfpack-0.3.0/CHANGELOG.md

@@ -0,0 +1,38 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.3.0] - 2026-07-25
+
+### Added
+- `subset_h5` — copy a subset of recordings out of a multi-recording HDF5 archive
+  (inverse of `merge_h5`), for carving a smaller release (e.g. BWM) out of a larger
+  superset (e.g. ephys-atlas) without re-compression.
+- Bad-channel labels (0=good, 1=dead, 2=noisy, 3=outside brain) persisted through
+  compression and exposed via `LFPackReader.channels` / `channels_full`.
+- **Saturation detection and muting**: ADC-clipped spans are detected on the raw LFP
+  band, stored as a per-recording interval table, and muted (cosine-tapered zero) on
+  the decimated output after Cadzow denoising. Exposed via `LFPackReader.saturation`
+  (raw-rate sample indices, recording-aligned) and `.saturation_summary`.
+  `.saturation_mask` is a property, indexed like the reader itself (`sr.saturation_mask[a:b]`);
+  `.saturation_times()` converts samples to session-clock seconds (matching `times`).
+
+### Fixed
+- `compress` no longer decompresses low-SNR chunks to exact zero; new `floor_k=64`
+  survival floor keeps the dominant mode's largest coefficients. Closes #2.
+
+## [0.2.0] - 2026-06-30
+
+### Added
+- `LFPackReader.channels` and `LFPackReader.channels_full` — per-channel probe geometry
+  and brain location annotations (`x/y/z` MNI coordinates, `atlas_id`, `acronym`).
+  Brain location fields are optional; the properties work on any existing file.
+  `channels` aggregates over binned-channel groups (mean for coordinates, mode for
+  brain region); `channels_full` always returns the raw per-electrode data.
+- `compress_to_h5` accepts an optional `channels` dict to embed brain location
+  annotations at compression time.

lfpack-0.3.0/CLAUDE.md

@@ -0,0 +1,81 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this project is
+
+**lfpack** is a lossy LFP (local-field-potential) codec for Neuropixels recordings. It compresses raw `.cbin` LFP binaries into HDF5 files at >100× compression with RMSE < 25 µV, using an 8-stage pipeline followed by adaptive SVD + wavelet-packet thresholding.
+
+## Commands
+
+```bash
+# Dev setup
+uv sync --group dev
+git config core.hooksPath .githooks   # installs ruff pre-commit hook
+
+# Tests
+uv run pytest                          # all tests
+uv run pytest tests/test_lfpack.py::TestLfpack::test_compress_output_shapes  # single test
+
+# Lint / format
+uv run ruff check src/ tests/
+uv run ruff format src/ tests/
+
+# Docs — regenerate API reference from docstrings, then preview
+uv run quartodoc build --config docs/_quarto.yml
+quarto preview docs/
+```
+
+## Architecture
+
+Nearly all code lives in `src/lfpack/_core.py` (~1000 lines). The public API is re-exported from `src/lfpack/__init__.py`.
+
+### Compression pipeline (`compress_bin_to_h5`)
+
+0. Saturation detection — flag ADC-clipped samples on the raw LFP band *before* any step below obscures them (`ibldsp.voltage.saturation_cbin`, parallel over `n_jobs`). Detection is **amplitude-only** for LFP (`v_per_sec=None`): the derivative criterion is tuned for the 30 kHz AP band and mislabels normal LFP dynamics as saturation, so it is disabled here. Stored as an insertion-level interval table (see HDF5 layout) and used to mute the clipped stretches; toggle with `detect_saturation` / `saturation_kwargs`.
+1. Bad-channel detection (`ibldsp.voltage.detect_bad_channels_cbin`) — labels saved to the scale-`00` `meta` as a `labels` int8 attr; read via `LFPackReader.channels`/`channels_full` under the `labels` key.
+2. Dephasing — sample-shift correction (NP1 only, via `ibldsp.fourier.fshift`)
+3. Highpass filter — 0.5 Hz zero-phase 3rd-order Butterworth (keeps delta/infra-slow; ibldsp warmup padding scales with the corner). The true recording start/end are cosine-apodized (~3 filter time-constants) before this filter so the zero-phase transient cannot ring against the data-boundary step (interior chunk edges are handled by the warmup padding).
+4. Bad-channel interpolation — distance-weighted neighbors
+5. CAR — median subtraction across channels
+6. Decimation — 2500 → 250 Hz (Q=10)
+7. Cadzow denoising — spatial rank reduction (`ibldsp.cadzow`)
+8. **Adaptive SVD + wavelet-packet thresholding** — the actual codec
+
+Steps 6–7 are checkpointed to a `.npy` file so the expensive decimation can be skipped on resume. Steps 7–8 are parallelised with `joblib`. Muting (step 0): a raw-rate boolean saturation mask is passed to `resample_denoise_lfp_cbin` as `saturation_file`; each worker downsamples its slice of the mask onto the decimated grid (block-max over each Q-sample block), cosine-tapers it (`mute_window_samples`, now in **decimated** output samples) and multiplies the **decimated output *after* Cadzow** by it. This *late* muting replaced the earlier pre-filter approach: muting the raw traces before the highpass/CAR/decimate/Cadzow did not keep the span clean — those stages leaked energy back into it (residual noise, spurious CSD sources/sinks); a single re-mute of the final output is cleaner and cheaper. Muting is skipped when resuming from an existing checkpoint (data already decimated); the interval table is still written.
+
+### Codec (`compress` / `decompress`)
+
+`compress(data, epsilon=150, alpha=28)` → `LFPCompressed` dataclass:
+- **SVD rank selection**: keep singular values > `epsilon × sigma_noise`
+- **Wavelet-packet thresholding**: db4, level 5; per-component threshold `alpha × sigma_noise / sv[k]`
+- Sparse Vh stored as `(vh_indices, vh_values)` in HDF5; `U_scaled` with shuffle+gzip
+
+Guard bands: 64-sample Cadzow halos, 128-sample SVD/WP overlap to prevent edge transients.
+
+### HDF5 layout (multi-recording, pyramidal)
+
+```
+<file>.h5
+└─ <recording>/
+   ├─ saturation   # (n_intervals, 2) int64 [start_sample, stop_sample] at raw LFP fs;
+   │               # written once per recording (scale-independent). attrs: fs, ns_total,
+   │               # n_saturated_samples, saturated_fraction, detection params, muted
+   └─ <scale_2digit>/
+      ├─ meta         # nc, ns_total, fs, fs_sync, t0_sync, epsilon, alpha, geometry, …
+      └─ chunks/
+         └─ <i>/      # U_scaled, vh_indices, vh_values + attrs
+```
+
+The `saturation` node sits at recording level (not under a scale) because it describes the raw recording, not a codec pyramid level; `merge_h5` copies it automatically. Legacy flat layout (meta at root) is still readable for backwards compatibility.
+
+### `LFPackReader`
+
+Drop-in replacement for `spikeglx.Reader`. Wraps an HDF5 file and decompresses chunks on demand. Supports multi-recording files via `recording=` kwarg and pyramidal scales via `scale=` kwarg. Saturation access: `.saturation` (interval DataFrame with raw samples + seconds), `.saturation_summary` (fraction/count/muted from attrs), `saturation_mask(first, last)` (boolean at the reader's decimated rate, interval edges rounded outward). All three degrade gracefully to empty/default on files written without saturation detection.
+
+## Ecosystem interconnections
+
+lfpack sits in a tight three-way dependency with two sibling IBL packages:
+
+- **ibl-neuropixel** (`/Users/olivier/PycharmProjects/ephys-atlas/ibl-neuropixel`) — provides the low-level Neuropixels binary I/O (`spikeglx.Reader`), destriping, dephasing (`ibldsp.fourier.fshift`), decimation (`ibldsp.voltage.resample_denoise_lfp_cbin`), bad-channel detection (`ibldsp.voltage.detect_bad_channels_cbin`), saturation detection (`ibldsp.voltage.saturation_cbin` / `saturation` / `saturation_samples_to_intervals`), and Cadzow denoising (`ibldsp.cadzow`). lfpack wraps these directly; changes to ibldsp can break the pipeline. `resample_denoise_lfp_cbin` gained a `saturation_file` / `mute_window_samples` parameter (added for this codec) to late-mute saturated stretches on the decimated output after Cadzow.
+- **viewephys** (`/Users/olivier/PycharmProjects/ephys-atlas/viewephys`) — Qt-based interactive viewer for raw Neuropixels traces. Since [PR #49](https://github.com/int-brain-lab/viewephys/pull/49) it has a native lfpack backend (`LFPackDataModel` / `LFPackBinViewer`, optional `viewephys[lfpack]` extra): `viewephys -f file.h5` opens a `.h5` file directly, with automatic brain-region colouring from embedded `atlas_id`/`acronym` annotations, a searchable multi-recording selector, and a CSD step. No manual transpose or `BrainRegions` wiring needed.

{lfpack-0.2.0 → lfpack-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: lfpack
-Version: 0.2.0
+Version: 0.3.0
 Summary: LFP codec — lossy compression of local-field-potential recordings via adaptive SVD and wavelet-packet thresholding
 Project-URL: Homepage, https://github.com/int-brain-lab/lfpack
 Project-URL: Bug Tracker, https://github.com/int-brain-lab/lfpack/issues
@@ -19,9 +19,6 @@ Description-Content-Type: text/markdown
 
 # lfpack — LFP codec for Neuropixels recordings
 
-> **IBL Brain-Wide Map LFP dataset** — 699 recordings, 384 channels, session-clock aligned.
-> [How to access →](https://int-brain-lab.github.io/lfpack/how-to/bwm-dataset.html)
-
 <p align="center">
   <img src="docs/figures/logo.png" alt="lfpack logo" width="300"/>
 </p>
@@ -30,10 +27,25 @@ Lossy codec for local-field-potential (LFP) recordings from Neuropixels probes.
 Achieves **>100× compression** with median RMSE < 25 µV via an 8-stage pipeline
 (bad-channel detection → dephasing → highpass → interpolation → CAR → decimation → Cadzow → adaptive SVD + wavelet-packet thresholding).
 
+> **IBL Brain-Wide Map LFP dataset** — 699 recordings, 384 channels, session-clock aligned.
+> [How to access →](https://int-brain-lab.github.io/lfpack/how-to/bwm-dataset.html)
+
 ```bash
 pip install lfpack
 ```
 
+## Visualise with viewephys
+
+[`viewephys`](https://github.com/int-brain-lab/viewephys) opens lfpack `.h5` files natively —
+`viewephys -f recording.h5` gets you a browsable, brain-region-coloured view with no
+manual decompression step:
+
+<p align="center">
+  <img src="docs/figures/viewephys_screenshot.jpg" alt="viewephys opened directly on a lfpack HDF5 file, showing brain-region-coloured LFP traces" width="700"/>
+</p>
+
+See the [BWM how-to](https://int-brain-lab.github.io/lfpack/how-to/bwm-dataset.html#visualise-with-viewephys) for details.
+
 ## Documentation
 
 Full documentation is at **https://int-brain-lab.github.io/lfpack/**.

{lfpack-0.2.0 → lfpack-0.3.0}/README.md

@@ -1,8 +1,5 @@
 # lfpack — LFP codec for Neuropixels recordings
 
-> **IBL Brain-Wide Map LFP dataset** — 699 recordings, 384 channels, session-clock aligned.
-> [How to access →](https://int-brain-lab.github.io/lfpack/how-to/bwm-dataset.html)
-
 <p align="center">
   <img src="docs/figures/logo.png" alt="lfpack logo" width="300"/>
 </p>
@@ -11,10 +8,25 @@ Lossy codec for local-field-potential (LFP) recordings from Neuropixels probes.
 Achieves **>100× compression** with median RMSE < 25 µV via an 8-stage pipeline
 (bad-channel detection → dephasing → highpass → interpolation → CAR → decimation → Cadzow → adaptive SVD + wavelet-packet thresholding).
 
+> **IBL Brain-Wide Map LFP dataset** — 699 recordings, 384 channels, session-clock aligned.
+> [How to access →](https://int-brain-lab.github.io/lfpack/how-to/bwm-dataset.html)
+
 ```bash
 pip install lfpack
 ```
 
+## Visualise with viewephys
+
+[`viewephys`](https://github.com/int-brain-lab/viewephys) opens lfpack `.h5` files natively —
+`viewephys -f recording.h5` gets you a browsable, brain-region-coloured view with no
+manual decompression step:
+
+<p align="center">
+  <img src="docs/figures/viewephys_screenshot.jpg" alt="viewephys opened directly on a lfpack HDF5 file, showing brain-region-coloured LFP traces" width="700"/>
+</p>
+
+See the [BWM how-to](https://int-brain-lab.github.io/lfpack/how-to/bwm-dataset.html#visualise-with-viewephys) for details.
+
 ## Documentation
 
 Full documentation is at **https://int-brain-lab.github.io/lfpack/**.

{lfpack-0.2.0 → lfpack-0.3.0}/docs/_quarto.yml

@@ -65,4 +65,9 @@ quartodoc:
     - title: "Reader"
       desc: "Random-access decompression."
       contents:
-        - LFPackReader
+        - LFPackReader
+    - title: "Multi-recording archives"
+      desc: "Combine or subset recordings across HDF5 files without re-compression."
+      contents:
+        - merge_h5
+        - subset_h5

{lfpack-0.2.0 → lfpack-0.3.0}/docs/explanation/pipeline.qmd

@@ -139,7 +139,9 @@ This ensures dominant spatial modes receive finer temporal detail than weak mode
 
 Non-zero coefficients are stored as sparse index/value pairs in HDF5.
 
-**Parameters:** `epsilon` — SVD noise threshold multiplier (default `150`); `alpha` — WP threshold multiplier (default `28`)
+**Survival floor.** Because each row of $V_r^\top$ has unit norm, its wavelet-packet coefficients have a fixed magnitude ceiling ($\max_j|w_{kj}| \approx 0.2\text{–}0.4$). On low-SNR chunks — where even the dominant mode is weak, $s_0/\sigma_\text{noise} \lesssim \alpha/\max_j|w_{0j}| \approx 140$ — the threshold exceeds every coefficient and the whole chunk reconstructs to exact zero, silently discarding real low-amplitude LFP. To prevent this the dominant mode ($k=0$) always keeps its `floor_k` largest coefficients. The floor is a no-op on high-SNR recordings (their dominant row keeps far more than `floor_k`, so their output is bit-for-bit unchanged) and on saturation-muted spans, which enter as all-zero and still reconstruct to zero ($s_0 = 0 \Rightarrow$ `U_scaled` $= 0$).
+
+**Parameters:** `epsilon` — SVD noise threshold multiplier (default `150`); `alpha` — WP threshold multiplier (default `28`); `floor_k` — minimum WP coefficients kept on the dominant mode (default `64`)
 
 ### Guard bands
 

lfpack-0.3.0/docs/how-to/bwm-dataset.qmd

@@ -0,0 +1,253 @@
+---
+title: "How to access the IBL Brain-Wide Map LFP dataset"
+---
+
+The IBL Brain-Wide Map (BWM) LFP dataset is distributed as a 16GB `.h5` file
+containing 699 individual recordings of 384 channels each.
+
+
+::: {.callout-important}
+## Before you start
+
+- **Align with trial events via `sr.times`** — each recording stores a sync-corrected
+  session clock; always use `sr.times` rather than a manual `np.arange / sr.fs` when
+  matching LFP samples to spike times or trial events.
+  See [Session-clock times](#session-clock-times).
+
+- **Use binned reads for brainwide sweeps** — looping over all 699 recordings at full
+  384-channel resolution is memory-intensive. Pass `bin_channels=4` to reduce each
+  recording to 96 spatial bins with acceptable loss of depth coverage.
+  See [Binned-channel reads](#binned-channel-reads).
+:::
+
+## Visualise with viewephys
+
+`viewephys` has a native lfpack backend — point it at the `.h5` file directly,
+no manual transpose or `BrainRegions` wiring needed:
+
+```bash
+viewephys -f lf_compressed_all_bwm.h5
+```
+
+The `.h5` extension is auto-detected and opens the lfpack-aware viewer, which:
+
+- shows a searchable recording selector (type any `pid` substring) when the
+  file holds multiple recordings, preserving window size, position and zoom
+  when you switch
+- colours the depth axis by brain region automatically, reading the
+  `atlas_id` / `acronym` annotations embedded in the file — no separate
+  `iblatlas` call required
+- adds a **CSD** (current-source density) step alongside the raw signal
+
+![Recording selector (top right) and raw/CSD checkboxes after opening a multi-recording lfpack file — the dataset info panel confirms the native "Neuropixels LFP (lfpack)" backend](../figures/viewephys_recording_selector.jpg){fig-alt="Ephys Bin Viewer window showing the searchable recording dropdown and raw/CSD checkboxes for an lfpack HDF5 file" .lightbox}
+
+The same class is usable from Python:
+
+```python
+from viewephys.gui import LFPackBinViewer
+from viewephys.viewer.qt import create_app
+
+app = create_app()
+win = LFPackBinViewer("lf_compressed_all_bwm.h5")
+app.exec()
+```
+
+![viewephys opened directly on a BWM lfpack file — brain regions colour the depth axis automatically and the searchable selector switches between the file's 699 recordings](../figures/viewephys_screenshot.jpg){fig-alt="viewephys density view showing 384-channel LFP data with brain region colour bar and a searchable recording selector" .lightbox}
+
+See [viewephys PR #49](https://github.com/int-brain-lab/viewephys/pull/49) for implementation details.
+
+## Prerequisites
+
+```bash
+uv pip install "viewephys[lfpack]" one-api
+```
+
+## Download
+
+From inside the `ibl-ai-agent` repository:
+
+```bash
+uv run python scripts/download_datasets.py --lfp
+```
+
+This downloads `lf_compressed_all_bwm.h5` (~14 GB) into `reports/datasets/bwm_lfp/`
+and writes the path into `data_locations.local.yaml` automatically.
+
+Or, from Python (no AWS credentials needed — uses the ONE public S3 helper):
+
+```python
+from one.remote.aws import s3_download_file
+
+s3_download_file(
+    source="resources/ibl-agent-data/lf_compressed_all_bwm.h5",
+    destination="lf_compressed_all_bwm.h5",
+)
+```
+
+`s3_download_file` defaults to the IBL public bucket, shows a tqdm progress bar,
+and skips the download if the local file already has the correct size.
+
+## Session-clock times
+
+`sr.times` returns a `(ns,)` array of session-clock timestamps (seconds) for every
+LFP sample, derived from the sync signal stored during compression.
+Use it to align LFP traces with trial events or spike times:
+
+```python
+from lfpack import LFPackReader
+import numpy as np
+
+file_lfpack = "lf_compressed_all_bwm.h5"
+pid = 'dab512bd-a02d-4c1f-8dbc-9155a163efc0'
+
+sr = LFPackReader(file_lfpack, recording=pid)
+
+n = int(10 * sr.fs)
+traces = sr[:n, :]      # (n_samples, nc) — first 10 s
+t = sr.times[:n]        # (n_samples,)    — session-clock seconds
+
+# Align with trial events loaded via ONE
+# stim_on = trials['stimOn_times']   # already in session-clock seconds
+# idx = np.searchsorted(t, stim_on)  # nearest LFP sample for each stimulus onset
+```
+
+## List recordings
+
+Each file contains one entry per BWM probe. Recording identifiers are named after the probe identifier UUID `pid`.
+
+```python
+from lfpack import LFPackReader
+
+recordings = LFPackReader.recordings("lf_compressed_all_bwm.h5")
+print(f"{len(recordings)} recordings")
+print(recordings[:3])
+# 699 recordings
+# ['00a824c0-e060-495f-9ebc-79c82fef4c67', '00a96dee-1e8b-44cc-9cc3-aca704d2b594', '00c425fd-ec3e-4cd2-b8af-c0bc0c4bdd44']
+```
+
+## Read a single recording
+
+```python
+from lfpack import LFPackReader
+
+file_lfpack = "lf_compressed_all_bwm.h5"
+pid = 'dab512bd-a02d-4c1f-8dbc-9155a163efc0'
+
+sr = LFPackReader(file_lfpack, recording=pid)
+
+# First 10 seconds — shape (2500, nc), float32, volts
+traces = sr[: int(10 * sr.fs)].T
+
+print(f"Duration:    {sr.ns / sr.fs:.1f} s")
+print(f"Channels:    {sr.nc}")
+print(f"Sample rate: {sr.fs} Hz")   # 250 Hz (decimated from 2500 Hz)
+print(f"Shape:       {traces.shape}")
+
+# Duration:    3668.9 s
+# Channels:    384
+# Sample rate: 250.00252518761928 Hz
+# Shape:       (384, 2500)
+```
+
+`LFPackReader` is a drop-in for `spikeglx.Reader`.
+Slicing decompresses only the requested chunks — the full file is never loaded into memory.
+
+## Quality Control - Saturation
+
+Each recording carries an insertion-level table of ADC-saturated (clipped) spans,
+detected on the raw LFP band before any filtering. The saturated stretches are muted
+(zeroed) in the decompressed output, so it's worth checking whether a window of
+interest overlaps one before trusting the amplitude.
+
+```python
+from lfpack import LFPackReader
+
+sr = LFPackReader("lf_compressed_all_bwm.h5", recording=pid)
+
+# Interval table: raw-rate sample indices, recording-aligned
+sr.saturation
+#    start_sample  stop_sample
+# 0        123456       125000
+
+# Recording-level summary (fraction/count/whether muting was applied)
+sr.saturation_summary
+# {'saturated_fraction': 0.0052, 'n_intervals': 3, 'total_saturated_sec': 12.9, 'muted': True, ...}
+```
+
+`saturation_mask` gives a boolean array at the reader's own — decimated — sampling rate,
+indexed exactly like the reader itself rather than called with a window:
+
+```python
+n = int(10 * sr.fs)
+traces = sr[:n, :]                  # (n, nc)
+mask = sr.saturation_mask[:n]       # (n,) bool, True where saturated
+
+traces_clean = traces[~mask]
+```
+
+For session-clock timestamps of saturated intervals (e.g. to overlay against trial events),
+use `saturation_times` rather than converting `sr.saturation`'s samples to seconds by hand —
+it applies the same sync correction as [`sr.times`](#session-clock-times):
+
+```python
+sr.saturation_times()
+#    start_sample  stop_sample  start_time  stop_time
+# 0        123456       125000    1234.382    1235.000
+```
+
+All four (`saturation`, `saturation_summary`, `saturation_mask`, `saturation_times`) degrade
+gracefully to empty/default values on recordings without saturation detection, so the same
+code works across the whole dataset without a try/except.
+
+
+![Saturation detection across the 699-recording release: per-recording saturated-fraction distribution (left) and the 15 most-affected recordings (right).](../figures/bwm_saturation_detection.png){.lightbox}
+
+
+## Channel brain locations
+
+Each recording stores per-channel brain location annotations embedded in the file —
+no network call needed.  Access them via `sr.channels`:
+
+```python
+from lfpack import LFPackReader
+
+sr = LFPackReader("lf_compressed_all_bwm.h5", recording=pid)
+ch = sr.channels
+
+print(ch["acronym"][:5])   # ['VISp', 'VISp', 'CA1', 'CA1', 'DG']
+print(ch["atlas_id"][:5])  # Allen CCF structure IDs, int32
+print(ch["x"][:5])         # mediolateral MNI coordinates, metres
+print(ch["y"][:5])         # anteroposterior
+print(ch["z"][:5])         # dorsoventral
+```
+
+When using `bin_channels`, `channels` automatically aggregates: coordinates are
+averaged and brain region is the within-group mode:
+
+```python
+sr = LFPackReader("lf_compressed_all_bwm.h5", recording=pid, bin_channels=4)
+ch = sr.channels           # shapes (96,) — one entry per spatial bin
+ch_full = sr.channels_full # shapes (384,) — always raw per-electrode
+```
+
+## Binned-channel reads
+
+Adjacent channels are spatially correlated at LFP frequencies.
+Passing `bin_channels` sums neighbouring channels during decompression, reducing
+memory use. A factor of 4 takes 384 channels down to 96 while keeping depth resolution useful for LFP.
+
+```python
+from lfpack import LFPackReader
+
+file_lfpack = "lf_compressed_all_bwm.h5"
+pid = 'dab512bd-a02d-4c1f-8dbc-9155a163efc0'
+
+sr = LFPackReader(file_lfpack, recording=pid, bin_channels=4)
+
+traces = sr[:, :]      # (ns, 96) — full recording, 96 spatial bins
+print(sr.nc)           # 96
+print(sr.geometry["y"])           # mean depth of each 4-channel group
+print(sr.geometry_full['binned_channel_index'])  # spatial mapping of the original 384 channels into the 96 binned ones
+```
+
+![Full-resolution (384 ch) vs binned ×4 (96 ch) LFP density for the same 4-second window](../figures/lfp_binned_comparison.jpg){fig-alt="Side-by-side LFP density plots at full and 4x binned channel resolution" .lightbox}

lfpack-0.3.0/docs/objects.json

@@ -0,0 +1 @@
+{"project": "lfpack", "version": "0.0.9999", "count": 54, "items": [{"name": "lfpack.compress_bin_to_h5", "domain": "py", "role": "function", "priority": "1", "uri": "reference/compress_bin_to_h5.html#lfpack.compress_bin_to_h5", "dispname": "-"}, {"name": "lfpack._core.compress_bin_to_h5", "domain": "py", "role": "function", "priority": "1", "uri": "reference/compress_bin_to_h5.html#lfpack.compress_bin_to_h5", "dispname": "lfpack.compress_bin_to_h5"}, {"name": "lfpack.compress_to_h5", "domain": "py", "role": "function", "priority": "1", "uri": "reference/compress_to_h5.html#lfpack.compress_to_h5", "dispname": "-"}, {"name": "lfpack._core.compress_to_h5", "domain": "py", "role": "function", "priority": "1", "uri": "reference/compress_to_h5.html#lfpack.compress_to_h5", "dispname": "lfpack.compress_to_h5"}, {"name": "lfpack.compress_pipeline", "domain": "py", "role": "function", "priority": "1", "uri": "reference/compress_pipeline.html#lfpack.compress_pipeline", "dispname": "-"}, {"name": "lfpack._core.compress_pipeline", "domain": "py", "role": "function", "priority": "1", "uri": "reference/compress_pipeline.html#lfpack.compress_pipeline", "dispname": "lfpack.compress_pipeline"}, {"name": "lfpack.compress", "domain": "py", "role": "function", "priority": "1", "uri": "reference/compress.html#lfpack.compress", "dispname": "-"}, {"name": "lfpack._core.compress", "domain": "py", "role": "function", "priority": "1", "uri": "reference/compress.html#lfpack.compress", "dispname": "lfpack.compress"}, {"name": "lfpack.decompress", "domain": "py", "role": "function", "priority": "1", "uri": "reference/decompress.html#lfpack.decompress", "dispname": "-"}, {"name": "lfpack._core.decompress", "domain": "py", "role": "function", "priority": "1", "uri": "reference/decompress.html#lfpack.decompress", "dispname": "lfpack.decompress"}, {"name": "lfpack.run_cadzow_checkpoint", "domain": "py", "role": "function", "priority": "1", "uri": "reference/run_cadzow_checkpoint.html#lfpack.run_cadzow_checkpoint", "dispname": "-"}, {"name": "lfpack._core.run_cadzow_checkpoint", "domain": "py", "role": "function", "priority": "1", "uri": "reference/run_cadzow_checkpoint.html#lfpack.run_cadzow_checkpoint", "dispname": "lfpack.run_cadzow_checkpoint"}, {"name": "lfpack.LFPCompressed", "domain": "py", "role": "class", "priority": "1", "uri": "reference/LFPCompressed.html#lfpack.LFPCompressed", "dispname": "-"}, {"name": "lfpack._core.LFPCompressed", "domain": "py", "role": "class", "priority": "1", "uri": "reference/LFPCompressed.html#lfpack.LFPCompressed", "dispname": "lfpack.LFPCompressed"}, {"name": "lfpack.LFPackReader.bin_channels", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.bin_channels", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.bin_channels", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.bin_channels", "dispname": "lfpack.LFPackReader.bin_channels"}, {"name": "lfpack.LFPackReader.channels", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.channels", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.channels", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.channels", "dispname": "lfpack.LFPackReader.channels"}, {"name": "lfpack.LFPackReader.channels_full", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.channels_full", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.channels_full", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.channels_full", "dispname": "lfpack.LFPackReader.channels_full"}, {"name": "lfpack.LFPackReader.fs", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.fs", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.fs", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.fs", "dispname": "lfpack.LFPackReader.fs"}, {"name": "lfpack.LFPackReader.geometry", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.geometry", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.geometry", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.geometry", "dispname": "lfpack.LFPackReader.geometry"}, {"name": "lfpack.LFPackReader.geometry_full", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.geometry_full", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.geometry_full", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.geometry_full", "dispname": "lfpack.LFPackReader.geometry_full"}, {"name": "lfpack.LFPackReader.nc", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.nc", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.nc", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.nc", "dispname": "lfpack.LFPackReader.nc"}, {"name": "lfpack.LFPackReader.read", "domain": "py", "role": "function", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.read", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.read", "domain": "py", "role": "function", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.read", "dispname": "lfpack.LFPackReader.read"}, {"name": "lfpack.LFPackReader.read_samples", "domain": "py", "role": "function", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.read_samples", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.read_samples", "domain": "py", "role": "function", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.read_samples", "dispname": "lfpack.LFPackReader.read_samples"}, {"name": "lfpack.LFPackReader.recordings", "domain": "py", "role": "function", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.recordings", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.recordings", "domain": "py", "role": "function", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.recordings", "dispname": "lfpack.LFPackReader.recordings"}, {"name": "lfpack.LFPackReader.saturation", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.saturation", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.saturation", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.saturation", "dispname": "lfpack.LFPackReader.saturation"}, {"name": "lfpack.LFPackReader.saturation_mask", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.saturation_mask", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.saturation_mask", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.saturation_mask", "dispname": "lfpack.LFPackReader.saturation_mask"}, {"name": "lfpack.LFPackReader.saturation_summary", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.saturation_summary", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.saturation_summary", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.saturation_summary", "dispname": "lfpack.LFPackReader.saturation_summary"}, {"name": "lfpack.LFPackReader.saturation_times", "domain": "py", "role": "function", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.saturation_times", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.saturation_times", "domain": "py", "role": "function", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.saturation_times", "dispname": "lfpack.LFPackReader.saturation_times"}, {"name": "lfpack.LFPackReader.scales", "domain": "py", "role": "function", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.scales", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.scales", "domain": "py", "role": "function", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.scales", "dispname": "lfpack.LFPackReader.scales"}, {"name": "lfpack.LFPackReader.t0", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.t0", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.t0", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.t0", "dispname": "lfpack.LFPackReader.t0"}, {"name": "lfpack.LFPackReader.times", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.times", "dispname": "-"}, {"name": "lfpack._core.LFPackReader.times", "domain": "py", "role": "attribute", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader.times", "dispname": "lfpack.LFPackReader.times"}, {"name": "lfpack.LFPackReader", "domain": "py", "role": "class", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader", "dispname": "-"}, {"name": "lfpack._core.LFPackReader", "domain": "py", "role": "class", "priority": "1", "uri": "reference/LFPackReader.html#lfpack.LFPackReader", "dispname": "lfpack.LFPackReader"}, {"name": "lfpack.merge_h5", "domain": "py", "role": "function", "priority": "1", "uri": "reference/merge_h5.html#lfpack.merge_h5", "dispname": "-"}, {"name": "lfpack._core.merge_h5", "domain": "py", "role": "function", "priority": "1", "uri": "reference/merge_h5.html#lfpack.merge_h5", "dispname": "lfpack.merge_h5"}, {"name": "lfpack.subset_h5", "domain": "py", "role": "function", "priority": "1", "uri": "reference/subset_h5.html#lfpack.subset_h5", "dispname": "-"}, {"name": "lfpack._core.subset_h5", "domain": "py", "role": "function", "priority": "1", "uri": "reference/subset_h5.html#lfpack.subset_h5", "dispname": "lfpack.subset_h5"}]}

{lfpack-0.2.0 → lfpack-0.3.0}/docs/reference/LFPackReader.qmd

@@ -40,9 +40,17 @@ contains exactly one recording the key is auto-detected; otherwise pass recordin
 | Name | Description |
 | --- | --- |
 | [bin_channels](#lfpack.LFPackReader.bin_channels) | Number of adjacent channels summed on every read (1 = no binning). |
+| [channels](#lfpack.LFPackReader.channels) | Per-channel info aggregated over bin groups when ``bin_channels > 1``. |
+| [channels_full](#lfpack.LFPackReader.channels_full) | Full per-electrode channel info, independent of ``bin_channels``. |
+| [fs](#lfpack.LFPackReader.fs) | LFP sample rate in Hz, sync-corrected when sync data is present. |
 | [geometry](#lfpack.LFPackReader.geometry) | Probe geometry averaged over each bin group. |
 | [geometry_full](#lfpack.LFPackReader.geometry_full) | Full per-electrode probe geometry, independent of ``bin_channels``. |
 | [nc](#lfpack.LFPackReader.nc) | Number of output channels (raw nc // bin_channels). |
+| [saturation](#lfpack.LFPackReader.saturation) | Insertion-level table of ADC-saturated intervals. |
+| [saturation_mask](#lfpack.LFPackReader.saturation_mask) | Boolean saturation mask for the full recording, at this reader's sampling rate. |
+| [saturation_summary](#lfpack.LFPackReader.saturation_summary) | Summary of saturation over the whole recording, read from dataset attrs. |
+| [t0](#lfpack.LFPackReader.t0) | Session-clock time in seconds at LFP sample 0. NaN when no sync data. |
+| [times](#lfpack.LFPackReader.times) | Session-clock timestamps in seconds for every LFP sample. |
 
 ## Methods
 
@@ -51,6 +59,7 @@ contains exactly one recording the key is auto-detected; otherwise pass recordin
 | [read](#lfpack.LFPackReader.read) | Decompress and return a sample range. |
 | [read_samples](#lfpack.LFPackReader.read_samples) | Read and decompress a sample range with optional spatial binning. |
 | [recordings](#lfpack.LFPackReader.recordings) | List recording keys at the root of an H5 file written by compress_to_h5. |
+| [saturation_times](#lfpack.LFPackReader.saturation_times) | Saturation interval table with session-clock start/stop times. |
 | [scales](#lfpack.LFPackReader.scales) | List scale indices available for a recording. |
 
 ### read { #lfpack.LFPackReader.read }
@@ -130,6 +139,26 @@ List recording keys at the root of an H5 file written by compress_to_h5.
 |--------|-------------|---------------|
 |        | list of str |               |
 
+### saturation_times { #lfpack.LFPackReader.saturation_times }
+
+```python
+LFPackReader.saturation_times()
+```
+
+Saturation interval table with session-clock start/stop times.
+
+Converts ``start_sample``/``stop_sample`` (raw-rate, recording-aligned) to
+``start_time``/``stop_time`` (session-clock seconds, matching ``times``) via the
+same sample-rate ratio and ``t0``/``fs`` used everywhere else in the reader —
+never divide the raw samples by a raw sample rate by hand.
+
+#### Returns {.doc-section .doc-section-returns}
+
+| Name   | Type                                                                             | Description   |
+|--------|----------------------------------------------------------------------------------|---------------|
+|        | pandas.DataFrame with columns ``start_sample``, ``stop_sample``, ``start_time``, |               |
+|        | ``stop_time``.                                                                   |               |
+
 ### scales { #lfpack.LFPackReader.scales }
 
 ```python

{lfpack-0.2.0 → lfpack-0.3.0}/docs/reference/_sidebar.yml

@@ -16,5 +16,9 @@ website:
     - contents:
       - reference/LFPackReader.qmd
       section: Reader
+    - contents:
+      - reference/merge_h5.qmd
+      - reference/subset_h5.qmd
+      section: Multi-recording archives
     id: reference
   - id: dummy-sidebar

lfpack-0.3.0/docs/reference/compress.qmd

@@ -0,0 +1,29 @@
+# compress { #lfpack.compress }
+
+```python
+compress(data, epsilon=150.0, alpha=28.0, floor_k=64)
+```
+
+Compress an LFP snippet using adaptive SVD and wavelet-packet thresholding.
+
+## Parameters {.doc-section .doc-section-parameters}
+
+| Name    | Type                      | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | Default    |
+|---------|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|
+| data    | ndarray of shape (nc, ns) | LFP data matrix, float32 or float64.  Rows are channels, columns are time.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | _required_ |
+| epsilon | float                     | SVD threshold multiplier.  rank = #{k : sv[k] > epsilon × sigma_noise}. Default 150.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | `150.0`    |
+| alpha   | float                     | WP threshold multiplier per component: tau_k = alpha × sigma_noise / sv[k]. Set to 0 to skip wavelet-packet stage.  Default 28.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `28.0`     |
+| floor_k | int                       | Survival floor on the dominant mode: the top retained row (``k == 0``) keeps at least its ``floor_k`` largest-magnitude WP coefficients.  This guarantees the reconstruction is never identically zero.  Without it, low-SNR chunks (``sv[0] / sigma_noise`` below ~``alpha / max\|coeff\|``, empirically ~140) have every coefficient thresholded and decompress to exact zero — silently destroying real low-amplitude LFP.  High-SNR recordings are unaffected: their dominant row keeps far more than ``floor_k`` coefficients, so the floor never triggers and the output is bit-for-bit identical to ``floor_k = 0``.  Set to 0 to disable. Default 64 (kills the zeroing on affected recordings while leaving the benchmark set unchanged). | `64`       |
+
+## Returns {.doc-section .doc-section-returns}
+
+| Name   | Type          | Description   |
+|--------|---------------|---------------|
+|        | LFPCompressed |               |
+
+## Notes {.doc-section .doc-section-notes}
+
+The floor never resurrects saturation-muted spans: an all-zero input gives
+``sv = 0`` so ``U_scaled = 0`` and the output is exactly zero whatever ``floor_k``
+is.  Zero output over a muted span is correct; the floor only targets the
+pathological all-zero from over-thresholding genuine low-amplitude LFP.