ibl-neuropixel 1.9.0__tar.gz → 1.9.1__tar.gz

{ibl_neuropixel-1.9.0/src/ibl_neuropixel.egg-info → ibl_neuropixel-1.9.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ibl-neuropixel
-Version: 1.9.0
+Version: 1.9.1
 Summary: Collection of tools for Neuropixel 1.0 and 2.0 probes data
 Home-page: https://github.com/int-brain-lab/ibl-neuropixel
 Author: The International Brain Laboratory
@@ -41,6 +41,53 @@ Minimum Python version supported is 3.10
 
 ## Destriping
 ### Getting started
+
+#### Compress a binary file losslessly using `mtscomp`
+
+The mtscomp util implements fast chunked compression for neurophysiology data in a single shard.
+Package repository is [here](https://github.com/int-brain-lab/mtscomp).
+
+
+```python
+from pathlib import Path
+import spikeglx
+file_spikeglx = Path('/datadisk/neuropixel/file.imec0.ap.bin')
+sr = spikeglx.Reader(file_spikeglx)
+sr.compress_file()
+# note: you can use sr.compress_file(keep_original=False) to also remove the orginal bin file
+```
+
+#### Reading raw spikeglx file and manipulating arrays
+
+The mtscomp util implements fast chunked compression for neurophysiology data in a single shard.
+Package repository is [here](https://github.com/int-brain-lab/mtscomp).
+
+```python
+from pathlib import Path
+import spikeglx
+
+import ibldsp.voltage
+
+file_spikeglx = Path('/datadisk/Data/neuropixel/human/Pt01.imec0.ap.bin')
+sr = spikeglx.Reader(file_spikeglx)
+
+# reads in 300ms of data
+raw = sr[10_300_000:10_310_000, :sr.nc - sr.nsync].T
+destripe = ibldsp.voltage.destripe(raw, fs=sr.fs, neuropixel_version=1)
+
+# display with matplotlib backend
+import ibldsp.plots
+ibldsp.plots.voltageshow(raw, fs=sr.fs, title='raw')
+ibldsp.plots.voltageshow(destripe, fs=sr.fs, title='destripe')
+
+# display with QT backend
+from viewephys.gui import viewephys
+eqc = {}
+eqc['raw'] = viewephys(raw, fs=sr.fs, title='raw')
+eqc['destripe'] = viewephys(destripe, fs=sr.fs, title='destripe')
+```
+
+#### Destripe a binary file
 This relies on a fast fourier transform external library: `pip install pyfftw`.
 
 Minimal working example to destripe a neuropixel binary file. 

ibl_neuropixel-1.9.1/README.md

@@ -0,0 +1,89 @@
+# ibl-neuropixel
+Collection of tools to handle Neuropixel 1.0 and 2.0 data
+(documentation coming soon...)
+
+## Installation
+Minimum Python version supported is 3.10
+`pip install ibl-neuropixel`
+
+
+## Destriping
+### Getting started
+
+#### Compress a binary file losslessly using `mtscomp`
+
+The mtscomp util implements fast chunked compression for neurophysiology data in a single shard.
+Package repository is [here](https://github.com/int-brain-lab/mtscomp).
+
+
+```python
+from pathlib import Path
+import spikeglx
+file_spikeglx = Path('/datadisk/neuropixel/file.imec0.ap.bin')
+sr = spikeglx.Reader(file_spikeglx)
+sr.compress_file()
+# note: you can use sr.compress_file(keep_original=False) to also remove the orginal bin file
+```
+
+#### Reading raw spikeglx file and manipulating arrays
+
+The mtscomp util implements fast chunked compression for neurophysiology data in a single shard.
+Package repository is [here](https://github.com/int-brain-lab/mtscomp).
+
+```python
+from pathlib import Path
+import spikeglx
+
+import ibldsp.voltage
+
+file_spikeglx = Path('/datadisk/Data/neuropixel/human/Pt01.imec0.ap.bin')
+sr = spikeglx.Reader(file_spikeglx)
+
+# reads in 300ms of data
+raw = sr[10_300_000:10_310_000, :sr.nc - sr.nsync].T
+destripe = ibldsp.voltage.destripe(raw, fs=sr.fs, neuropixel_version=1)
+
+# display with matplotlib backend
+import ibldsp.plots
+ibldsp.plots.voltageshow(raw, fs=sr.fs, title='raw')
+ibldsp.plots.voltageshow(destripe, fs=sr.fs, title='destripe')
+
+# display with QT backend
+from viewephys.gui import viewephys
+eqc = {}
+eqc['raw'] = viewephys(raw, fs=sr.fs, title='raw')
+eqc['destripe'] = viewephys(destripe, fs=sr.fs, title='destripe')
+```
+
+#### Destripe a binary file
+This relies on a fast fourier transform external library: `pip install pyfftw`.
+
+Minimal working example to destripe a neuropixel binary file. 
+```python
+from pathlib import Path
+from ibldsp.voltage import decompress_destripe_cbin
+sr_file = Path('/datadisk/Data/spike_sorting/pykilosort_tests/imec_385_100s.ap.bin')
+out_file = Path('/datadisk/scratch/imec_385_100s.ap.bin')
+
+decompress_destripe_cbin(sr_file=sr_file, output_file=out_file, nprocesses=8)
+```
+
+### Viewer
+
+The best way to look at the results is to use [viewephys](https://github.com/oliche/viewephys),
+open an ephys viewer on the raw data.
+
+- tick the destripe box.
+- move to a desired location in the file
+- ctr+P will make the gain and axis the same on both windows
+
+![alt text](./docs/raw_bin_viewer_destripe.png "Ephys viewer")
+
+You can then move within the raw data file.
+
+### White Paper
+The following describes the methods implemented in this repository.
+https://doi.org/10.6084/m9.figshare.19705522
+
+## Contribution
+Please see our [contribution guidelines](CONTRIBUTING.md) for details on how to contribute to this project.

{ibl_neuropixel-1.9.0 → ibl_neuropixel-1.9.1}/setup.py

@@ -8,7 +8,7 @@ with open("requirements.txt") as f:
 
 setuptools.setup(
     name="ibl-neuropixel",
-    version="1.9.0",
+    version="1.9.1",
     author="The International Brain Laboratory",
     description="Collection of tools for Neuropixel 1.0 and 2.0 probes data",
     long_description=long_description,

{ibl_neuropixel-1.9.0 → ibl_neuropixel-1.9.1/src/ibl_neuropixel.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ibl-neuropixel
-Version: 1.9.0
+Version: 1.9.1
 Summary: Collection of tools for Neuropixel 1.0 and 2.0 probes data
 Home-page: https://github.com/int-brain-lab/ibl-neuropixel
 Author: The International Brain Laboratory
@@ -41,6 +41,53 @@ Minimum Python version supported is 3.10
 
 ## Destriping
 ### Getting started
+
+#### Compress a binary file losslessly using `mtscomp`
+
+The mtscomp util implements fast chunked compression for neurophysiology data in a single shard.
+Package repository is [here](https://github.com/int-brain-lab/mtscomp).
+
+
+```python
+from pathlib import Path
+import spikeglx
+file_spikeglx = Path('/datadisk/neuropixel/file.imec0.ap.bin')
+sr = spikeglx.Reader(file_spikeglx)
+sr.compress_file()
+# note: you can use sr.compress_file(keep_original=False) to also remove the orginal bin file
+```
+
+#### Reading raw spikeglx file and manipulating arrays
+
+The mtscomp util implements fast chunked compression for neurophysiology data in a single shard.
+Package repository is [here](https://github.com/int-brain-lab/mtscomp).
+
+```python
+from pathlib import Path
+import spikeglx
+
+import ibldsp.voltage
+
+file_spikeglx = Path('/datadisk/Data/neuropixel/human/Pt01.imec0.ap.bin')
+sr = spikeglx.Reader(file_spikeglx)
+
+# reads in 300ms of data
+raw = sr[10_300_000:10_310_000, :sr.nc - sr.nsync].T
+destripe = ibldsp.voltage.destripe(raw, fs=sr.fs, neuropixel_version=1)
+
+# display with matplotlib backend
+import ibldsp.plots
+ibldsp.plots.voltageshow(raw, fs=sr.fs, title='raw')
+ibldsp.plots.voltageshow(destripe, fs=sr.fs, title='destripe')
+
+# display with QT backend
+from viewephys.gui import viewephys
+eqc = {}
+eqc['raw'] = viewephys(raw, fs=sr.fs, title='raw')
+eqc['destripe'] = viewephys(destripe, fs=sr.fs, title='destripe')
+```
+
+#### Destripe a binary file
 This relies on a fast fourier transform external library: `pip install pyfftw`.
 
 Minimal working example to destripe a neuropixel binary file. 

{ibl_neuropixel-1.9.0 → ibl_neuropixel-1.9.1}/src/ibldsp/utils.py

@@ -89,7 +89,7 @@ def parabolic_max(x):
     # for 2D arrays, operate along the last dimension
     ns = x.shape[-1]
     axis = -1
-    imax = np.argmax(x, axis=axis)
+    imax = np.nanargmax(x, axis=axis)
 
     if x.ndim == 1:
         v010 = x[np.maximum(np.minimum(imax + np.array([-1, 0, 1]), ns - 1), 0)]
@@ -383,7 +383,7 @@ class WindowGenerator(object):
             yield (first, last, amp)
 
     @property
-    def firstlast_valid(self):
+    def firstlast_valid(self, discard_edges=False):
         """
         Generator that yields a tuple of first, last, first_valid, last_valid index of windows
         The valid indices span up to half of the overlap
@@ -391,8 +391,14 @@ class WindowGenerator(object):
         """
         assert self.overlap % 2 == 0, "Overlap must be even"
         for first, last in self.firstlast:
-            first_valid = 0 if first == 0 else first + self.overlap // 2
-            last_valid = last if last == self.ns else last - self.overlap // 2
+            first_valid = (
+                0 if first == 0 and not discard_edges else first + self.overlap // 2
+            )
+            last_valid = (
+                last
+                if last == self.ns and not discard_edges
+                else last - self.overlap // 2
+            )
             yield (first, last, first_valid, last_valid)
 
     @property

{ibl_neuropixel-1.9.0 → ibl_neuropixel-1.9.1}/src/ibldsp/voltage.py

@@ -782,10 +782,11 @@ def decompress_destripe_cbin(
         np.save(
             output_qc_path.joinpath("_iblqc_ephysTimeRmsAP.timestamps.npy"), time_data
         )
-        np.save(
-            output_qc_path.joinpath("_iblqc_ephysSaturation.samples.npy"),
+        saturation_samples_to_intervals(
             saturation_data,
+            output_file=output_qc_path.joinpath("_iblqc_ephysSaturation.samples.pqt"),
         )
+        file_saturation.unlink()
 
 
 def detect_bad_channels(

{ibl_neuropixel-1.9.0 → ibl_neuropixel-1.9.1}/src/spikeglx.py

@@ -144,11 +144,7 @@ class Reader:
         sglx_file = str(self.file_bin)
         if self.is_mtscomp:
             self._raw = mtscomp.Reader()
-            self.ch_file = (
-                _get_companion_file(sglx_file, ".ch")
-                if self.ch_file is None
-                else self.ch_file
-            )
+            self.ch_file = self._parse_ch_file()
             self._raw.open(self.file_bin, self.ch_file)
             if self._raw.shape != (self.ns, self.nc):
                 ftsec = self._raw.shape[0] / self.fs
@@ -396,10 +392,8 @@ class Reader:
         if "out" not in kwargs:
             kwargs["out"] = self.file_bin.with_suffix(".bin")
         assert self.is_mtscomp
-        if file_ch is None:
-            file_ch = self.file_bin.with_suffix(".ch")
-
-        r = mtscomp.decompress(self.file_bin, file_ch, **kwargs)
+        ch_file = self._parse_ch_file(file_ch)
+        r = mtscomp.decompress(self.file_bin, ch_file, **kwargs)
         r.close()
         if not keep_original:
             self.close()
@@ -420,9 +414,10 @@ class Reader:
             bin_file = Path(self.file_bin).with_suffix(".bin")
         else:
             scratch_dir.mkdir(exist_ok=True, parents=True)
-            bin_file = scratch_dir / Path(self.file_bin).with_suffix(".bin").name
-            file_meta_scratch = scratch_dir / file_meta.name
-            shutil.copy(self.file_meta_data, file_meta_scratch)
+            bin_file = (
+                Path(scratch_dir).joinpath(self.file_bin.name).with_suffix(".bin")
+            )
+            shutil.copy(self.file_meta_data, bin_file.parent / self.file_meta_data.name)
         if not bin_file.exists():
             t0 = time.time()
             _logger.info("File is compressed, decompressing to a temporary file...")
@@ -464,6 +459,12 @@ class Reader:
         log_func(f"SHA1 computed: {sc}")
         return sm == sc
 
+    def _parse_ch_file(self, ch_file=None):
+        ch_file = (
+            _get_companion_file(self.file_bin, ".ch") if ch_file is None else ch_file
+        )
+        return ch_file
+
 
 class OnlineReader(Reader):
     @property
@@ -999,7 +1000,7 @@ def _mock_spikeglx_file(
     meta_file,
     ns,
     nc,
-    sync_depth,
+    sync_depth=16,
     random=False,
     int2volts=0.6 / 32768,
     corrupt=False,

{ibl_neuropixel-1.9.0 → ibl_neuropixel-1.9.1}/src/tests/integration/test_destripe.py

@@ -4,6 +4,7 @@ import logging
 import shutil
 import unittest
 from pathlib import Path
+import pandas as pd
 
 import neuropixel
 import spikeglx
@@ -84,11 +85,11 @@ class TestEphysSpikeSortingMultiProcess(unittest.TestCase):
             shutil.rmtree(self.file_path.parent)
 
     def _assert_qc(self):
-        sr = spikeglx.Reader(self.file_path)
-        saturated = np.load(
-            self.file_path.parent.joinpath("_iblqc_ephysSaturation.samples.npy")
+        df_saturated = pd.read_parquet(
+            self.file_path.parent.joinpath("_iblqc_ephysSaturation.samples.pqt")
         )
-        self.assertEqual(sr.ns, saturated.size)
+        self.assertTrue(df_saturated.shape[1] == 2)
+
         self.assertTrue(
             self.file_path.parent.joinpath("_iblqc_ephysTimeRmsAP.rms.npy").exists()
         )

{ibl_neuropixel-1.9.0 → ibl_neuropixel-1.9.1}/src/tests/unit/test_spikeglx.py

@@ -2,6 +2,7 @@ from pathlib import Path
 import shutil
 import tempfile
 import unittest
+import uuid
 
 import numpy as np
 from iblutil.io import hashfile
@@ -671,9 +672,29 @@ class TestsBasicReader(unittest.TestCase):
     Tests the basic usage where there is a flat binary and no metadata associated
     """
 
-    def test_get_companion_file(self):
-        import uuid
+    def test_integration_companion_files_cbin(self):
+        with tempfile.TemporaryDirectory() as td:
+            bin_file_orig = Path(td) / "toto.ap.bin"
+            meta_file = Path(td) / f"toto.ap.{str(uuid.uuid4())}.meta"
+            ch_file = Path(td) / f"toto.ap.{str(uuid.uuid4())}.ch"
+            spikeglx._mock_spikeglx_file(
+                bin_file_orig,
+                meta_file=Path(TEST_PATH).joinpath("sample3B_g0_t0.imec1.ap.meta"),
+                ns=90_000,
+                nc=385,
+            )
+            sr = spikeglx.Reader(bin_file_orig)
+            sr.compress_file(keep_original=False)
+            cbin_file = Path(td) / f"toto.ap.{str(uuid.uuid4())}.cbin"
+            shutil.move(bin_file_orig.with_suffix(".cbin"), cbin_file)
+            shutil.move(bin_file_orig.with_suffix(".ch"), ch_file)
+            shutil.move(bin_file_orig.with_suffix(".meta"), meta_file)
+            sr = spikeglx.Reader(cbin_file)
+            self.assertEqual(sr.file_bin, cbin_file)
+            self.assertEqual(sr.file_meta_data, meta_file)
+            self.assertEqual(sr.ch_file, ch_file)
 
+    def test_get_companion_file(self):
         with tempfile.TemporaryDirectory() as td:
             sglx_file = Path(td) / f"sample3A_g0_t0.imec.ap.{str(uuid.uuid4())}.bin"
             meta_file = Path(td) / f"sample3A_g0_t0.imec.ap.{str(uuid.uuid4())}.meta"

{ibl_neuropixel-1.9.0 → ibl_neuropixel-1.9.1}/src/tests/unit/test_utils.py

@@ -69,8 +69,8 @@ class TestSyncTimestamps(unittest.TestCase):
 
 class TestParabolicMax(unittest.TestCase):
     # expected values
-    maxi = np.array([np.nan, 0, 3.04166667, 3.04166667, 5, 5])
-    ipeak = np.array([np.nan, 0, 5.166667, 2.166667, 0, 7])
+    maxi = np.array([0.0, 0.0, 3.04166667, 3.04166667, 5, 5])
+    ipeak = np.array([0.0, 0.0, 5.166667, 2.166667, 0, 7])
     # input
     x = np.array(
         [

{ibl_neuropixel-1.9.0 → ibl_neuropixel-1.9.1}/src/tests/unit/test_voltage.py

@@ -131,6 +131,24 @@ class TestSaturation(unittest.TestCase):
         self.assertGreater(np.sum(saturated), 5)
         self.assertGreater(np.sum(mute == 0), np.sum(saturated))
 
+    def test_saturation_intervals_output(self):
+        saturation = np.zeros(50_000, dtype=bool)
+        # we test empty files, make sure we can read/write from empty parquet
+        with tempfile.TemporaryDirectory() as temp_dir:
+            # Create a file path within the temporary directory
+            temp_file = Path(temp_dir).joinpath("saturation.pqt")
+            df_nothing = ibldsp.voltage.saturation_samples_to_intervals(
+                saturation, output_file=Path(temp_dir).joinpath("saturation.pqt")
+            )
+            df_nothing2 = pd.read_parquet(temp_file)
+        self.assertEqual(df_nothing.shape[0], 0)
+        self.assertEqual(df_nothing2.shape[0], 0)
+        # for the case with saturation intervals, we simply test the number of rows correspond to the events
+        saturation[3441:3509] = True
+        saturation[45852:45865] = True
+        df_sat = ibldsp.voltage.saturation_samples_to_intervals(saturation)
+        self.assertEqual(81, np.sum(df_sat["stop_sample"] - df_sat["start_sample"]))
+
 
 class TestCadzow(unittest.TestCase):
     def test_trajectory_matrixes(self):

ibl_neuropixel-1.9.0/README.md

@@ -1,42 +0,0 @@
-# ibl-neuropixel
-Collection of tools to handle Neuropixel 1.0 and 2.0 data
-(documentation coming soon...)
-
-## Installation
-Minimum Python version supported is 3.10
-`pip install ibl-neuropixel`
-
-
-## Destriping
-### Getting started
-This relies on a fast fourier transform external library: `pip install pyfftw`.
-
-Minimal working example to destripe a neuropixel binary file. 
-```python
-from pathlib import Path
-from ibldsp.voltage import decompress_destripe_cbin
-sr_file = Path('/datadisk/Data/spike_sorting/pykilosort_tests/imec_385_100s.ap.bin')
-out_file = Path('/datadisk/scratch/imec_385_100s.ap.bin')
-
-decompress_destripe_cbin(sr_file=sr_file, output_file=out_file, nprocesses=8)
-```
-
-### Viewer
-
-The best way to look at the results is to use [viewephys](https://github.com/oliche/viewephys),
-open an ephys viewer on the raw data.
-
-- tick the destripe box.
-- move to a desired location in the file
-- ctr+P will make the gain and axis the same on both windows
-
-![alt text](./docs/raw_bin_viewer_destripe.png "Ephys viewer")
-
-You can then move within the raw data file.
-
-### White Paper
-The following describes the methods implemented in this repository.
-https://doi.org/10.6084/m9.figshare.19705522
-
-## Contribution
-Please see our [contribution guidelines](CONTRIBUTING.md) for details on how to contribute to this project.