ibl-neuropixel 1.8.1__tar.gz → 1.9.1__tar.gz

{ibl_neuropixel-1.8.1/src/ibl_neuropixel.egg-info → ibl_neuropixel-1.9.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ibl-neuropixel
-Version: 1.8.1
+Version: 1.9.1
 Summary: Collection of tools for Neuropixel 1.0 and 2.0 probes data
 Home-page: https://github.com/int-brain-lab/ibl-neuropixel
 Author: The International Brain Laboratory
@@ -41,6 +41,53 @@ Minimum Python version supported is 3.10
 
 ## Destriping
 ### Getting started
+
+#### Compress a binary file losslessly using `mtscomp`
+
+The mtscomp util implements fast chunked compression for neurophysiology data in a single shard.
+Package repository is [here](https://github.com/int-brain-lab/mtscomp).
+
+
+```python
+from pathlib import Path
+import spikeglx
+file_spikeglx = Path('/datadisk/neuropixel/file.imec0.ap.bin')
+sr = spikeglx.Reader(file_spikeglx)
+sr.compress_file()
+# note: you can use sr.compress_file(keep_original=False) to also remove the orginal bin file
+```
+
+#### Reading raw spikeglx file and manipulating arrays
+
+The mtscomp util implements fast chunked compression for neurophysiology data in a single shard.
+Package repository is [here](https://github.com/int-brain-lab/mtscomp).
+
+```python
+from pathlib import Path
+import spikeglx
+
+import ibldsp.voltage
+
+file_spikeglx = Path('/datadisk/Data/neuropixel/human/Pt01.imec0.ap.bin')
+sr = spikeglx.Reader(file_spikeglx)
+
+# reads in 300ms of data
+raw = sr[10_300_000:10_310_000, :sr.nc - sr.nsync].T
+destripe = ibldsp.voltage.destripe(raw, fs=sr.fs, neuropixel_version=1)
+
+# display with matplotlib backend
+import ibldsp.plots
+ibldsp.plots.voltageshow(raw, fs=sr.fs, title='raw')
+ibldsp.plots.voltageshow(destripe, fs=sr.fs, title='destripe')
+
+# display with QT backend
+from viewephys.gui import viewephys
+eqc = {}
+eqc['raw'] = viewephys(raw, fs=sr.fs, title='raw')
+eqc['destripe'] = viewephys(destripe, fs=sr.fs, title='destripe')
+```
+
+#### Destripe a binary file
 This relies on a fast fourier transform external library: `pip install pyfftw`.
 
 Minimal working example to destripe a neuropixel binary file. 
@@ -71,22 +118,4 @@ The following describes the methods implemented in this repository.
 https://doi.org/10.6084/m9.figshare.19705522
 
 ## Contribution
-Contribution checklist:
-- run tests
-- ruff format
-- PR to main
-
-
-Pypi Release checklist:
-- Edit the version number in `setup.py`
-- add release notes in `release_notes.md`
-
-
-```shell
-ruff format
-tag=X.Y.Z
-git tag -a $tag 
-git push origin $tag
-```
-
-Create new release with tag X.Y.Z (will automatically publish to PyPI)
+Please see our [contribution guidelines](CONTRIBUTING.md) for details on how to contribute to this project.

ibl_neuropixel-1.9.1/README.md

@@ -0,0 +1,89 @@
+# ibl-neuropixel
+Collection of tools to handle Neuropixel 1.0 and 2.0 data
+(documentation coming soon...)
+
+## Installation
+Minimum Python version supported is 3.10
+`pip install ibl-neuropixel`
+
+
+## Destriping
+### Getting started
+
+#### Compress a binary file losslessly using `mtscomp`
+
+The mtscomp util implements fast chunked compression for neurophysiology data in a single shard.
+Package repository is [here](https://github.com/int-brain-lab/mtscomp).
+
+
+```python
+from pathlib import Path
+import spikeglx
+file_spikeglx = Path('/datadisk/neuropixel/file.imec0.ap.bin')
+sr = spikeglx.Reader(file_spikeglx)
+sr.compress_file()
+# note: you can use sr.compress_file(keep_original=False) to also remove the orginal bin file
+```
+
+#### Reading raw spikeglx file and manipulating arrays
+
+The mtscomp util implements fast chunked compression for neurophysiology data in a single shard.
+Package repository is [here](https://github.com/int-brain-lab/mtscomp).
+
+```python
+from pathlib import Path
+import spikeglx
+
+import ibldsp.voltage
+
+file_spikeglx = Path('/datadisk/Data/neuropixel/human/Pt01.imec0.ap.bin')
+sr = spikeglx.Reader(file_spikeglx)
+
+# reads in 300ms of data
+raw = sr[10_300_000:10_310_000, :sr.nc - sr.nsync].T
+destripe = ibldsp.voltage.destripe(raw, fs=sr.fs, neuropixel_version=1)
+
+# display with matplotlib backend
+import ibldsp.plots
+ibldsp.plots.voltageshow(raw, fs=sr.fs, title='raw')
+ibldsp.plots.voltageshow(destripe, fs=sr.fs, title='destripe')
+
+# display with QT backend
+from viewephys.gui import viewephys
+eqc = {}
+eqc['raw'] = viewephys(raw, fs=sr.fs, title='raw')
+eqc['destripe'] = viewephys(destripe, fs=sr.fs, title='destripe')
+```
+
+#### Destripe a binary file
+This relies on a fast fourier transform external library: `pip install pyfftw`.
+
+Minimal working example to destripe a neuropixel binary file. 
+```python
+from pathlib import Path
+from ibldsp.voltage import decompress_destripe_cbin
+sr_file = Path('/datadisk/Data/spike_sorting/pykilosort_tests/imec_385_100s.ap.bin')
+out_file = Path('/datadisk/scratch/imec_385_100s.ap.bin')
+
+decompress_destripe_cbin(sr_file=sr_file, output_file=out_file, nprocesses=8)
+```
+
+### Viewer
+
+The best way to look at the results is to use [viewephys](https://github.com/oliche/viewephys),
+open an ephys viewer on the raw data.
+
+- tick the destripe box.
+- move to a desired location in the file
+- ctr+P will make the gain and axis the same on both windows
+
+![alt text](./docs/raw_bin_viewer_destripe.png "Ephys viewer")
+
+You can then move within the raw data file.
+
+### White Paper
+The following describes the methods implemented in this repository.
+https://doi.org/10.6084/m9.figshare.19705522
+
+## Contribution
+Please see our [contribution guidelines](CONTRIBUTING.md) for details on how to contribute to this project.

{ibl_neuropixel-1.8.1 → ibl_neuropixel-1.9.1}/setup.py

@@ -8,7 +8,7 @@ with open("requirements.txt") as f:
 
 setuptools.setup(
     name="ibl-neuropixel",
-    version="1.8.1",
+    version="1.9.1",
     author="The International Brain Laboratory",
     description="Collection of tools for Neuropixel 1.0 and 2.0 probes data",
     long_description=long_description,

{ibl_neuropixel-1.8.1 → ibl_neuropixel-1.9.1/src/ibl_neuropixel.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ibl-neuropixel
-Version: 1.8.1
+Version: 1.9.1
 Summary: Collection of tools for Neuropixel 1.0 and 2.0 probes data
 Home-page: https://github.com/int-brain-lab/ibl-neuropixel
 Author: The International Brain Laboratory
@@ -41,6 +41,53 @@ Minimum Python version supported is 3.10
 
 ## Destriping
 ### Getting started
+
+#### Compress a binary file losslessly using `mtscomp`
+
+The mtscomp util implements fast chunked compression for neurophysiology data in a single shard.
+Package repository is [here](https://github.com/int-brain-lab/mtscomp).
+
+
+```python
+from pathlib import Path
+import spikeglx
+file_spikeglx = Path('/datadisk/neuropixel/file.imec0.ap.bin')
+sr = spikeglx.Reader(file_spikeglx)
+sr.compress_file()
+# note: you can use sr.compress_file(keep_original=False) to also remove the orginal bin file
+```
+
+#### Reading raw spikeglx file and manipulating arrays
+
+The mtscomp util implements fast chunked compression for neurophysiology data in a single shard.
+Package repository is [here](https://github.com/int-brain-lab/mtscomp).
+
+```python
+from pathlib import Path
+import spikeglx
+
+import ibldsp.voltage
+
+file_spikeglx = Path('/datadisk/Data/neuropixel/human/Pt01.imec0.ap.bin')
+sr = spikeglx.Reader(file_spikeglx)
+
+# reads in 300ms of data
+raw = sr[10_300_000:10_310_000, :sr.nc - sr.nsync].T
+destripe = ibldsp.voltage.destripe(raw, fs=sr.fs, neuropixel_version=1)
+
+# display with matplotlib backend
+import ibldsp.plots
+ibldsp.plots.voltageshow(raw, fs=sr.fs, title='raw')
+ibldsp.plots.voltageshow(destripe, fs=sr.fs, title='destripe')
+
+# display with QT backend
+from viewephys.gui import viewephys
+eqc = {}
+eqc['raw'] = viewephys(raw, fs=sr.fs, title='raw')
+eqc['destripe'] = viewephys(destripe, fs=sr.fs, title='destripe')
+```
+
+#### Destripe a binary file
 This relies on a fast fourier transform external library: `pip install pyfftw`.
 
 Minimal working example to destripe a neuropixel binary file. 
@@ -71,22 +118,4 @@ The following describes the methods implemented in this repository.
 https://doi.org/10.6084/m9.figshare.19705522
 
 ## Contribution
-Contribution checklist:
-- run tests
-- ruff format
-- PR to main
-
-
-Pypi Release checklist:
-- Edit the version number in `setup.py`
-- add release notes in `release_notes.md`
-
-
-```shell
-ruff format
-tag=X.Y.Z
-git tag -a $tag 
-git push origin $tag
-```
-
-Create new release with tag X.Y.Z (will automatically publish to PyPI)
+Please see our [contribution guidelines](CONTRIBUTING.md) for details on how to contribute to this project.

{ibl_neuropixel-1.8.1 → ibl_neuropixel-1.9.1}/src/ibl_neuropixel.egg-info/SOURCES.txt

@@ -59,7 +59,9 @@ src/tests/integration/csd_experiments.py
 src/tests/integration/test_destripe.py
 src/tests/unit/__init__.py
 src/tests/unit/test_ephys_np2.py
-src/tests/unit/test_ibldsp.py
 src/tests/unit/test_neuropixel.py
+src/tests/unit/test_plots.py
 src/tests/unit/test_spikeglx.py
+src/tests/unit/test_utils.py
+src/tests/unit/test_voltage.py
 src/tests/unit/test_waveforms.py

ibl_neuropixel-1.9.1/src/ibldsp/plots.py

@@ -0,0 +1,135 @@
+import numpy as np
+import matplotlib.pyplot as plt
+
+AP_RANGE_UV = 75
+LF_RANGE_UV = 250
+
+
+def show_channels_labels(
+    raw,
+    fs,
+    channel_labels,
+    xfeats,
+    similarity_threshold=(-0.5, 1),
+    psd_hf_threshold=0.02,
+):
+    """
+    Shows the features side by side a snippet of raw data
+    :param sr:
+    :return:
+    """
+    nc, ns = raw.shape
+    raw = raw - np.mean(raw, axis=-1)[:, np.newaxis]  # removes DC offset
+    ns_plot = np.minimum(ns, 3000)
+    fig, ax = plt.subplots(
+        1, 5, figsize=(18, 6), gridspec_kw={"width_ratios": [1, 1, 1, 8, 0.2]}
+    )
+    ax[0].plot(xfeats["xcor_hf"], np.arange(nc))
+    ax[0].plot(  # plot channel below the similarity threshold as dead in black
+        xfeats["xcor_hf"][(iko := channel_labels == 1)], np.arange(nc)[iko], "k*"
+    )
+    ax[0].plot(  # plot the values above the similarity threshold as noisy in red
+        xfeats["xcor_hf"][
+            (iko := np.where(xfeats["xcor_hf"] > similarity_threshold[1]))
+        ],
+        np.arange(nc)[iko],
+        "r*",
+    )
+    ax[0].plot(similarity_threshold[0] * np.ones(2), [0, nc], "k--")
+    ax[0].plot(similarity_threshold[1] * np.ones(2), [0, nc], "r--")
+    ax[0].set(
+        ylabel="channel #",
+        xlabel="high coherence",
+        ylim=[0, nc],
+        title="a) dead channel",
+    )
+    ax[1].plot(xfeats["psd_hf"], np.arange(nc))
+    ax[1].plot(
+        xfeats["psd_hf"][(iko := xfeats["psd_hf"] > psd_hf_threshold)],
+        np.arange(nc)[iko],
+        "r*",
+    )
+    ax[1].plot(psd_hf_threshold * np.array([1, 1]), [0, nc], "r--")
+    ax[1].set(yticklabels=[], xlabel="PSD", ylim=[0, nc], title="b) noisy channel")
+    ax[1].sharey(ax[0])
+    ax[2].plot(xfeats["xcor_lf"], np.arange(nc))
+    ax[2].plot(
+        xfeats["xcor_lf"][(iko := channel_labels == 3)], np.arange(nc)[iko], "y*"
+    )
+    ax[2].plot([-0.75, -0.75], [0, nc], "y--")
+    ax[2].set(yticklabels=[], xlabel="LF coherence", ylim=[0, nc], title="c) outside")
+    ax[2].sharey(ax[0])
+    voltageshow(raw[:, :ns_plot], fs, ax=ax[3], cax=ax[4])
+    ax[3].sharey(ax[0])
+    fig.tight_layout()
+    return fig, ax
+
+
+def voltageshow(
+    raw,
+    fs,
+    cmap="PuOr",
+    ax=None,
+    cax=None,
+    cbar_label="Voltage (uV)",
+    scaling=1e6,
+    vrange=None,
+    **axis_kwargs,
+):
+    """
+    Visualizes electrophysiological voltage data as a heatmap.
+
+    This function displays raw voltage data as a color-coded image with appropriate
+    scaling based on the sampling frequency. It automatically selects voltage range
+    based on whether the data is low-frequency (LF) or action potential (AP) data.
+
+    Parameters
+    ----------
+    raw : numpy.ndarray
+        Raw voltage data array with shape (channels, samples), in Volts
+    fs : float
+        Sampling frequency in Hz, used to determine time axis scaling and voltage range.
+    cmap : str, optional
+        Matplotlib colormap name for the heatmap. Default is 'PuOr'.
+    ax : matplotlib.axes.Axes, optional
+        Axes object to plot on. If None, a new figure and axes are created.
+    cax : matplotlib.axes.Axes, optional
+        Axes object for the colorbar. If None and ax is None, a new colorbar axes is created.
+    cbar_label : str, optional
+        Label for the colorbar. Default is 'Voltage (uV)'.
+    vrange: float, optional
+        Voltage range for the colorbar. Defaults to +/- 75 uV for AP and +/- 250 uV for LF.
+    scaling: float, optional
+        Unit transform: default is 1e6: we expect Volts but plot uV.
+    **axis_kwargs: optional
+        Additional keyword arguments for the axis properties, fed to the ax.set() method.
+    Returns
+    -------
+    matplotlib.image.AxesImage
+        The image object created by imshow, which can be used for further customization.
+    """
+    if ax is None:
+        fig, axs = plt.subplots(1, 2, gridspec_kw={"width_ratios": [1, 0.05]})
+        ax, cax = axs
+    nc, ns = raw.shape
+    default_vrange = LF_RANGE_UV if fs < 2600 else AP_RANGE_UV
+    vrange = vrange if vrange is not None else default_vrange
+    im = ax.imshow(
+        raw * scaling,
+        origin="lower",
+        cmap=cmap,
+        aspect="auto",
+        vmin=-vrange,
+        vmax=vrange,
+        extent=[0, ns / fs, 0, nc],
+    )
+    # set the axis properties: we use defaults values that can be overridden by user-provided ones
+    axis_kwargs = (
+        dict(ylim=[0, nc], xlabel="Time (s)", ylabel="Depth (μm)") | axis_kwargs
+    )
+    ax.set(**axis_kwargs)
+    ax.grid(False)
+    if cax is not None:
+        plt.colorbar(im, cax=cax, shrink=0.8).ax.set(ylabel=cbar_label)
+
+    return im

{ibl_neuropixel-1.8.1 → ibl_neuropixel-1.9.1}/src/ibldsp/utils.py

@@ -89,7 +89,7 @@ def parabolic_max(x):
     # for 2D arrays, operate along the last dimension
     ns = x.shape[-1]
     axis = -1
-    imax = np.argmax(x, axis=axis)
+    imax = np.nanargmax(x, axis=axis)
 
     if x.ndim == 1:
         v010 = x[np.maximum(np.minimum(imax + np.array([-1, 0, 1]), ns - 1), 0)]
@@ -268,12 +268,64 @@ def make_channel_index(geom, radius=200.0, pad_val=None):
 
 class WindowGenerator(object):
     """
-    `wg = WindowGenerator(ns, nswin, overlap)`
+    A utility class for generating sliding windows for signal processing applications.
 
-    Provide sliding windows indices generator for signal processing applications.
-    For straightforward spectrogram / periodogram implementation, prefer scipy methods !
+    WindowGenerator provides various methods to iterate through windows of a signal
+    with configurable window size and overlap. It's particularly useful for operations
+    like spectrograms, filtering, or any processing that requires windowed analysis.
 
-    Example of implementations in test_dsp.py.
+    Parameters
+    ----------
+    ns : int
+        Total number of samples in the signal to be windowed.
+    nswin : int
+        Number of samples in each window.
+    overlap : int
+        Number of samples that overlap between consecutive windows.
+
+    Attributes
+    ----------
+    ns : int
+        Total number of samples in the signal.
+    nswin : int
+        Number of samples in each window.
+    overlap : int
+        Number of samples that overlap between consecutive windows.
+    nwin : int
+        Total number of windows.
+    iw : int or None
+        Current window index during iteration.
+
+    Notes
+    -----
+    For straightforward spectrogram or periodogram implementation,
+    scipy methods are recommended over this class.
+
+    Examples
+    --------
+    # straight windowing without overlap
+    >>> wg = WindowGenerator(ns=1000, nwin=111)
+    >>> signal = np.random.randn(1000)
+    >>> for window_slice in wg.slice:
+    ...     window_data = signal[window_slice]
+    ...     # Process window_data
+
+    # windowing with overlap (ie. buffers for apodization)
+    >>> for win_slice, valid_slice, win_valid_slice in wg.slices_valid:
+    ...     window = signal[win_slice]
+    ...     # Process window
+    ...     processed = some_function_with_edge_effect(window)
+    ...     # Only use the valid portion for reconstruction
+    ...     recons[valid_slice] = processed[win_valid_slice]
+
+    # splicing add a fade-in / fade-out in the overlap so that reconstruction has unit amplitude
+    >>> recons = np.zeros_like(signal)
+    >>> for win_slice, amplitude in wg.splice:
+    ...     window = signal[win_slice]
+    ...     # Process window
+    ...     processed = some_function(window)
+    ...     # The processed windows is weighted with the amplitude and added to the reconstructed signal
+    ...     recons[win_slice] = recons[win_slice] + processed * amplitude
     """
 
     def __init__(self, ns, nswin, overlap):
@@ -289,14 +341,35 @@ class WindowGenerator(object):
         self.iw = None
 
     @property
-    def firstlast_splicing(self):
+    def splice(self):
         """
-        Generator that yields the indices as well as an amplitude function that can be used
-        to splice the windows together.
-        In the overlap, the amplitude function gradually transitions the amplitude from one window
-        to the next. The amplitudes always sum to one (ie. windows are symmetrical)
+        Generator that yields slices and amplitude arrays for windowed signal processing with splicing.
+
+        This property provides a convenient way to iterate through all windows with their
+        corresponding amplitude arrays for proper signal reconstruction. The amplitude arrays
+        contain tapering values (from a Hann window) at the overlapping regions to ensure
+        unit amplitude of all samples of the original signal
+
+        Yields
+        ------
+        tuple
+            A tuple containing:
+            - slice: A Python slice object representing the current window
+            - amp: A numpy array containing amplitude values for proper splicing/tapering
+              at overlap regions
+
+        Notes
+        -----
+        This is particularly useful for overlap-add methods where windows need to be
+        properly weighted before being combined in the reconstruction process.
+        """
+        for first, last, amp in self.firstlast_splicing:
+            yield slice(first, last), amp
 
-        :return: tuple of (first_index, last_index, amplitude_vector]
+    @property
+    def firstlast_splicing(self):
+        """
+        cf. self.splice
         """
         w = scipy.signal.windows.hann((self.overlap + 1) * 2 + 1, sym=True)[
             1 : self.overlap + 1
@@ -310,7 +383,7 @@ class WindowGenerator(object):
             yield (first, last, amp)
 
     @property
-    def firstlast_valid(self):
+    def firstlast_valid(self, discard_edges=False):
         """
         Generator that yields a tuple of first, last, first_valid, last_valid index of windows
         The valid indices span up to half of the overlap
@@ -318,12 +391,18 @@ class WindowGenerator(object):
         """
         assert self.overlap % 2 == 0, "Overlap must be even"
         for first, last in self.firstlast:
-            first_valid = 0 if first == 0 else first + self.overlap // 2
-            last_valid = last if last == self.ns else last - self.overlap // 2
+            first_valid = (
+                0 if first == 0 and not discard_edges else first + self.overlap // 2
+            )
+            last_valid = (
+                last
+                if last == self.ns and not discard_edges
+                else last - self.overlap // 2
+            )
             yield (first, last, first_valid, last_valid)
 
     @property
-    def firstlast(self, return_valid=False):
+    def firstlast(self):
         """
         Generator that yields first and last index of windows
 
@@ -343,13 +422,51 @@ class WindowGenerator(object):
     @property
     def slice(self):
         """
-        Generator that yields slices of windows
-
-        :return: a slice of the window
+        Generator that yields slice objects for each window in the signal.
+
+        This property provides a convenient way to iterate through all windows
+        defined by the WindowGenerator parameters. Each yielded slice can be
+        used directly to index into the original signal array.
+
+        Yields
+        ------
+        slice
+            A Python slice object representing the current window, defined by
+            its first and last indices. The slice can be used to extract the
+            corresponding window from the original signal.
         """
         for first, last in self.firstlast:
             yield slice(first, last)
 
+    @property
+    def slices_valid(self):
+        """
+        Generator that yields slices for windowed signal processing with valid regions.
+
+        This method generates tuples of slice objects that can be used to extract windows
+        from a signal and identify the valid (non-overlapping) portions within each window.
+        It's particularly useful for reconstruction operations where overlapping regions
+        need special handling.
+
+        Yields
+        ------
+        tuple
+            A tuple containing three slice objects:
+            - slice(first, last): The full window slice
+            - slice(first_valid, last_valid): The valid portion of the signal in absolute indices
+            - slice_window_valid: The valid portion relative to the window (for use within the window)
+
+        Notes
+        -----
+        This generator relies on the firstlast_valid property which provides the
+        indices for both the full windows and their valid regions.
+        """
+        for first, last, first_valid, last_valid in self.firstlast_valid:
+            slice_window_valid = slice(
+                first_valid - first, None if (lv := -(last - last_valid)) == 0 else lv
+            )
+            yield slice(first, last), slice(first_valid, last_valid), slice_window_valid
+
     def slice_array(self, sig, axis=-1):
         """
         Provided an array or sliceable object, generator that yields