@synsci/cli-darwin-x64-baseline 1.1.77 → 1.1.78

package/bin/skills/neuropixels-analysis/references/AUTOMATED_CURATION.md

@@ -0,0 +1,358 @@
+# Automated Curation Reference
+
+Guide to automated spike sorting curation using Bombcell, UnitRefine, and other tools.
+
+## Why Automated Curation?
+
+Manual curation is:
+- **Slow**: Hours per recording session
+- **Subjective**: Inter-rater variability
+- **Non-reproducible**: Hard to standardize
+
+Automated tools provide consistent, reproducible quality classification.
+
+## Available Tools
+
+| Tool | Classification | Language | Integration |
+|------|---------------|----------|-------------|
+| **Bombcell** | 4-class (single/multi/noise/non-somatic) | Python/MATLAB | SpikeInterface, Phy |
+| **UnitRefine** | Machine learning-based | Python | SpikeInterface |
+| **SpikeInterface QM** | Threshold-based | Python | Native |
+| **UnitMatch** | Cross-session tracking | Python/MATLAB | Kilosort, Bombcell |
+
+## Bombcell
+
+### Overview
+
+Bombcell classifies units into 4 categories:
+1. **Single somatic units** - Well-isolated single neurons
+2. **Multi-unit activity (MUA)** - Mixed neuronal signals
+3. **Noise** - Non-neural artifacts
+4. **Non-somatic** - Axonal or dendritic signals
+
+### Installation
+
+```bash
+# Python
+pip install bombcell
+
+# Or development version
+git clone https://github.com/Julie-Fabre/bombcell.git
+cd bombcell/py_bombcell
+pip install -e .
+```
+
+### Basic Usage (Python)
+
+```python
+import bombcell as bc
+
+# Load sorted data (Kilosort output)
+kilosort_folder = '/path/to/kilosort/output'
+raw_data_path = '/path/to/recording.ap.bin'
+
+# Run Bombcell
+results = bc.run_bombcell(
+    kilosort_folder,
+    raw_data_path,
+    sample_rate=30000,
+    n_channels=384
+)
+
+# Get classifications
+unit_labels = results['unit_labels']
+# 'good' = single unit, 'mua' = multi-unit, 'noise' = noise
+```
+
+### Integration with SpikeInterface
+
+```python
+import spikeinterface.full as si
+
+# After spike sorting
+sorting = si.run_sorter('kilosort4', recording, output_folder='ks4/')
+
+# Create analyzer and compute required extensions
+analyzer = si.create_sorting_analyzer(sorting, recording, sparse=True)
+analyzer.compute('waveforms')
+analyzer.compute('templates')
+analyzer.compute('spike_amplitudes')
+
+# Export to Phy format (Bombcell can read this)
+si.export_to_phy(analyzer, output_folder='phy_export/')
+
+# Run Bombcell on Phy export
+import bombcell as bc
+results = bc.run_bombcell_phy('phy_export/')
+```
+
+### Bombcell Metrics
+
+Bombcell computes specific metrics for classification:
+
+| Metric | Description | Used For |
+|--------|-------------|----------|
+| `peak_trough_ratio` | Waveform shape | Somatic vs non-somatic |
+| `spatial_decay` | Amplitude across channels | Noise detection |
+| `refractory_period_violations` | ISI violations | Single vs multi |
+| `presence_ratio` | Temporal stability | Unit quality |
+| `waveform_duration` | Peak-to-trough time | Cell type |
+
+### Custom Thresholds
+
+```python
+# Customize classification thresholds
+custom_params = {
+    'isi_threshold': 0.01,          # ISI violation threshold
+    'presence_threshold': 0.9,       # Minimum presence ratio
+    'amplitude_threshold': 20,       # Minimum amplitude (μV)
+    'spatial_decay_threshold': 40,   # Spatial decay (μm)
+}
+
+results = bc.run_bombcell(
+    kilosort_folder,
+    raw_data_path,
+    **custom_params
+)
+```
+
+## SpikeInterface Auto-Curation
+
+### Threshold-Based Curation
+
+```python
+# Compute quality metrics
+analyzer.compute('quality_metrics')
+qm = analyzer.get_extension('quality_metrics').get_data()
+
+# Define curation function
+def auto_curate(qm):
+    labels = {}
+    for unit_id in qm.index:
+        row = qm.loc[unit_id]
+
+        # Classification logic
+        if row['snr'] < 2 or row['presence_ratio'] < 0.5:
+            labels[unit_id] = 'noise'
+        elif row['isi_violations_ratio'] > 0.1:
+            labels[unit_id] = 'mua'
+        elif (row['snr'] > 5 and
+              row['isi_violations_ratio'] < 0.01 and
+              row['presence_ratio'] > 0.9):
+            labels[unit_id] = 'good'
+        else:
+            labels[unit_id] = 'unsorted'
+
+    return labels
+
+unit_labels = auto_curate(qm)
+
+# Filter by label
+good_unit_ids = [u for u, l in unit_labels.items() if l == 'good']
+sorting_curated = sorting.select_units(good_unit_ids)
+```
+
+### Using SpikeInterface Curation Module
+
+```python
+from spikeinterface.curation import (
+    CurationSorting,
+    MergeUnitsSorting,
+    SplitUnitSorting
+)
+
+# Wrap sorting for curation
+curation = CurationSorting(sorting)
+
+# Remove noise units
+noise_units = qm[qm['snr'] < 2].index.tolist()
+curation.remove_units(noise_units)
+
+# Merge similar units (based on template similarity)
+analyzer.compute('template_similarity')
+similarity = analyzer.get_extension('template_similarity').get_data()
+
+# Find highly similar pairs
+import numpy as np
+threshold = 0.9
+similar_pairs = np.argwhere(similarity > threshold)
+# Merge pairs (careful - requires manual review)
+
+# Get curated sorting
+sorting_curated = curation.to_sorting()
+```
+
+## UnitMatch: Cross-Session Tracking
+
+Track the same neurons across recording days.
+
+### Installation
+
+```bash
+pip install unitmatch
+# Or from source
+git clone https://github.com/EnnyvanBeest/UnitMatch.git
+```
+
+### Usage
+
+```python
+# After running Bombcell on multiple sessions
+session_folders = [
+    '/path/to/session1/kilosort/',
+    '/path/to/session2/kilosort/',
+    '/path/to/session3/kilosort/',
+]
+
+from unitmatch import UnitMatch
+
+# Run UnitMatch
+um = UnitMatch(session_folders)
+um.run()
+
+# Get matching results
+matches = um.get_matches()
+# Returns DataFrame with unit IDs matched across sessions
+
+# Assign unique IDs
+unique_ids = um.get_unique_ids()
+```
+
+### Integration with Workflow
+
+```python
+# Typical workflow:
+# 1. Spike sort each session
+# 2. Run Bombcell for quality control
+# 3. Run UnitMatch for cross-session tracking
+
+# Session 1
+sorting1 = si.run_sorter('kilosort4', rec1, output_folder='session1/ks4/')
+# Run Bombcell
+labels1 = bc.run_bombcell('session1/ks4/', raw1_path)
+
+# Session 2
+sorting2 = si.run_sorter('kilosort4', rec2, output_folder='session2/ks4/')
+labels2 = bc.run_bombcell('session2/ks4/', raw2_path)
+
+# Track units across sessions
+um = UnitMatch(['session1/ks4/', 'session2/ks4/'])
+matches = um.get_matches()
+```
+
+## Semi-Automated Workflow
+
+Combine automated and manual curation:
+
+```python
+# Step 1: Automated classification
+analyzer.compute('quality_metrics')
+qm = analyzer.get_extension('quality_metrics').get_data()
+
+# Auto-label obvious cases
+auto_labels = {}
+for unit_id in qm.index:
+    row = qm.loc[unit_id]
+    if row['snr'] < 1.5:
+        auto_labels[unit_id] = 'noise'
+    elif row['snr'] > 8 and row['isi_violations_ratio'] < 0.005:
+        auto_labels[unit_id] = 'good'
+    else:
+        auto_labels[unit_id] = 'needs_review'
+
+# Step 2: Export uncertain units for manual review
+needs_review = [u for u, l in auto_labels.items() if l == 'needs_review']
+
+# Export only uncertain units to Phy
+sorting_review = sorting.select_units(needs_review)
+analyzer_review = si.create_sorting_analyzer(sorting_review, recording)
+analyzer_review.compute('waveforms')
+analyzer_review.compute('templates')
+si.export_to_phy(analyzer_review, output_folder='phy_review/')
+
+# Manual review in Phy: phy template-gui phy_review/params.py
+
+# Step 3: Load manual labels and merge
+manual_labels = si.read_phy('phy_review/').get_property('quality')
+# Combine auto + manual labels for final result
+```
+
+## Comparison of Methods
+
+| Method | Pros | Cons |
+|--------|------|------|
+| **Manual (Phy)** | Gold standard, flexible | Slow, subjective |
+| **SpikeInterface QM** | Fast, reproducible | Simple thresholds only |
+| **Bombcell** | Multi-class, validated | Requires waveform extraction |
+| **UnitRefine** | ML-based, learns from data | Needs training data |
+
+## Best Practices
+
+1. **Always visualize** - Don't blindly trust automated results
+2. **Document thresholds** - Record exact parameters used
+3. **Validate** - Compare automated vs manual on subset
+4. **Be conservative** - When in doubt, exclude the unit
+5. **Report methods** - Include curation criteria in publications
+
+## Pipeline Example
+
+```python
+def curate_sorting(sorting, recording, output_dir):
+    """Complete curation pipeline."""
+
+    # Create analyzer
+    analyzer = si.create_sorting_analyzer(sorting, recording, sparse=True,
+                                          folder=f'{output_dir}/analyzer')
+
+    # Compute required extensions
+    analyzer.compute('random_spikes', max_spikes_per_unit=500)
+    analyzer.compute('waveforms')
+    analyzer.compute('templates')
+    analyzer.compute('noise_levels')
+    analyzer.compute('spike_amplitudes')
+    analyzer.compute('quality_metrics')
+
+    qm = analyzer.get_extension('quality_metrics').get_data()
+
+    # Auto-classify
+    labels = {}
+    for unit_id in qm.index:
+        row = qm.loc[unit_id]
+
+        if row['snr'] < 2:
+            labels[unit_id] = 'noise'
+        elif row['isi_violations_ratio'] > 0.1 or row['presence_ratio'] < 0.8:
+            labels[unit_id] = 'mua'
+        elif (row['snr'] > 5 and
+              row['isi_violations_ratio'] < 0.01 and
+              row['presence_ratio'] > 0.9 and
+              row['amplitude_cutoff'] < 0.1):
+            labels[unit_id] = 'good'
+        else:
+            labels[unit_id] = 'unsorted'
+
+    # Summary
+    from collections import Counter
+    print("Classification summary:")
+    print(Counter(labels.values()))
+
+    # Save labels
+    import json
+    with open(f'{output_dir}/unit_labels.json', 'w') as f:
+        json.dump(labels, f)
+
+    # Return good units
+    good_ids = [u for u, l in labels.items() if l == 'good']
+    return sorting.select_units(good_ids), labels
+
+# Usage
+sorting_curated, labels = curate_sorting(sorting, recording, 'output/')
+```
+
+## References
+
+- [Bombcell GitHub](https://github.com/Julie-Fabre/bombcell)
+- [UnitMatch GitHub](https://github.com/EnnyvanBeest/UnitMatch)
+- [SpikeInterface Curation](https://spikeinterface.readthedocs.io/en/stable/modules/curation.html)
+- Fabre et al. (2023) "Bombcell: automated curation and cell classification"
+- van Beest et al. (2024) "UnitMatch: tracking neurons across days with high-density probes"

package/bin/skills/neuropixels-analysis/references/MOTION_CORRECTION.md

@@ -0,0 +1,323 @@
+# Motion/Drift Correction Reference
+
+Mechanical drift during acute probe insertion is a major challenge for Neuropixels recordings. This guide covers detection, estimation, and correction of motion artifacts.
+
+## Why Motion Correction Matters
+
+- Neuropixels probes can drift 10-100+ μm during recording
+- Uncorrected drift leads to:
+  - Units appearing/disappearing mid-recording
+  - Waveform amplitude changes
+  - Incorrect spike-unit assignments
+  - Reduced unit yield
+
+## Detection: Check Before Sorting
+
+**Always visualize drift before running spike sorting!**
+
+```python
+import spikeinterface.full as si
+from spikeinterface.sortingcomponents.peak_detection import detect_peaks
+from spikeinterface.sortingcomponents.peak_localization import localize_peaks
+
+# Preprocess first (don't whiten - affects peak localization)
+rec = si.highpass_filter(recording, freq_min=400.)
+rec = si.common_reference(rec, operator='median', reference='global')
+
+# Detect peaks
+noise_levels = si.get_noise_levels(rec, return_in_uV=False)
+peaks = detect_peaks(
+    rec,
+    method='locally_exclusive',
+    noise_levels=noise_levels,
+    detect_threshold=5,
+    radius_um=50.,
+    n_jobs=8,
+    chunk_duration='1s',
+    progress_bar=True
+)
+
+# Localize peaks
+peak_locations = localize_peaks(
+    rec, peaks,
+    method='center_of_mass',
+    n_jobs=8,
+    chunk_duration='1s'
+)
+
+# Visualize drift
+si.plot_drift_raster_map(
+    peaks=peaks,
+    peak_locations=peak_locations,
+    recording=rec,
+    clim=(-200, 0)  # Adjust color limits
+)
+```
+
+### Interpreting Drift Plots
+
+| Pattern | Interpretation | Action |
+|---------|---------------|--------|
+| Horizontal bands, stable | No significant drift | Skip correction |
+| Diagonal bands (slow) | Gradual settling drift | Use motion correction |
+| Rapid jumps | Brain pulsation or movement | Use non-rigid correction |
+| Chaotic patterns | Severe instability | Consider discarding segment |
+
+## Motion Correction Methods
+
+### Quick Correction (Recommended Start)
+
+```python
+# Simple one-liner with preset
+rec_corrected = si.correct_motion(
+    recording=rec,
+    preset='nonrigid_fast_and_accurate'
+)
+```
+
+### Available Presets
+
+| Preset | Speed | Accuracy | Best For |
+|--------|-------|----------|----------|
+| `rigid_fast` | Fast | Low | Quick check, small drift |
+| `kilosort_like` | Medium | Good | Kilosort-compatible results |
+| `nonrigid_accurate` | Slow | High | Publication-quality |
+| `nonrigid_fast_and_accurate` | Medium | High | **Recommended default** |
+| `dredge` | Slow | Highest | Best results, complex drift |
+| `dredge_fast` | Medium | High | DREDge with less compute |
+
+### Full Control Pipeline
+
+```python
+from spikeinterface.sortingcomponents.motion import (
+    estimate_motion,
+    interpolate_motion
+)
+
+# Step 1: Estimate motion
+motion, temporal_bins, spatial_bins = estimate_motion(
+    rec,
+    peaks,
+    peak_locations,
+    method='decentralized',
+    direction='y',
+    rigid=False,          # Non-rigid for Neuropixels
+    win_step_um=50,       # Spatial window step
+    win_sigma_um=150,     # Spatial smoothing
+    bin_s=2.0,            # Temporal bin size
+    progress_bar=True
+)
+
+# Step 2: Visualize motion estimate
+si.plot_motion(
+    motion,
+    temporal_bins,
+    spatial_bins,
+    recording=rec
+)
+
+# Step 3: Apply correction via interpolation
+rec_corrected = interpolate_motion(
+    recording=rec,
+    motion=motion,
+    temporal_bins=temporal_bins,
+    spatial_bins=spatial_bins,
+    border_mode='force_extrapolate'
+)
+```
+
+### Save Motion Estimate
+
+```python
+# Save for later use
+import numpy as np
+np.savez('motion_estimate.npz',
+         motion=motion,
+         temporal_bins=temporal_bins,
+         spatial_bins=spatial_bins)
+
+# Load later
+data = np.load('motion_estimate.npz')
+motion = data['motion']
+temporal_bins = data['temporal_bins']
+spatial_bins = data['spatial_bins']
+```
+
+## DREDge: State-of-the-Art Method
+
+DREDge (Decentralized Registration of Electrophysiology Data) is currently the best-performing motion correction method.
+
+### Using DREDge Preset
+
+```python
+# AP-band motion estimation
+rec_corrected = si.correct_motion(rec, preset='dredge')
+
+# Or compute explicitly
+motion, motion_info = si.compute_motion(
+    rec,
+    preset='dredge',
+    output_motion_info=True,
+    folder='motion_output/',
+    **job_kwargs
+)
+```
+
+### LFP-Based Motion Estimation
+
+For very fast drift or when AP-band estimation fails:
+
+```python
+# Load LFP stream
+lfp = si.read_spikeglx('/path/to/data', stream_name='imec0.lf')
+
+# Estimate motion from LFP (faster, handles rapid drift)
+motion_lfp, motion_info = si.compute_motion(
+    lfp,
+    preset='dredge_lfp',
+    output_motion_info=True
+)
+
+# Apply to AP recording
+rec_corrected = interpolate_motion(
+    recording=rec,  # AP recording
+    motion=motion_lfp,
+    temporal_bins=motion_info['temporal_bins'],
+    spatial_bins=motion_info['spatial_bins']
+)
+```
+
+## Integration with Spike Sorting
+
+### Option 1: Pre-correction (Recommended)
+
+```python
+# Correct before sorting
+rec_corrected = si.correct_motion(rec, preset='nonrigid_fast_and_accurate')
+
+# Save corrected recording
+rec_corrected = rec_corrected.save(folder='preprocessed_motion_corrected/',
+                                    format='binary', n_jobs=8)
+
+# Run spike sorting on corrected data
+sorting = si.run_sorter('kilosort4', rec_corrected, output_folder='ks4/')
+```
+
+### Option 2: Let Kilosort Handle It
+
+Kilosort 2.5+ has built-in drift correction:
+
+```python
+sorting = si.run_sorter(
+    'kilosort4',
+    rec,  # Not motion corrected
+    output_folder='ks4/',
+    nblocks=5,  # Non-rigid blocks for drift correction
+    do_correction=True  # Enable Kilosort's drift correction
+)
+```
+
+### Option 3: Post-hoc Correction
+
+```python
+# Sort first
+sorting = si.run_sorter('kilosort4', rec, output_folder='ks4/')
+
+# Then estimate motion from sorted spikes
+# (More accurate as it uses actual spike times)
+from spikeinterface.sortingcomponents.motion import estimate_motion_from_sorting
+
+motion = estimate_motion_from_sorting(sorting, rec)
+```
+
+## Parameters Deep Dive
+
+### Peak Detection
+
+```python
+peaks = detect_peaks(
+    rec,
+    method='locally_exclusive',  # Best for dense probes
+    noise_levels=noise_levels,
+    detect_threshold=5,          # Lower = more peaks (noisier estimate)
+    radius_um=50.,               # Exclusion radius
+    exclude_sweep_ms=0.1,        # Temporal exclusion
+)
+```
+
+### Motion Estimation
+
+```python
+motion = estimate_motion(
+    rec, peaks, peak_locations,
+    method='decentralized',      # 'decentralized' or 'iterative_template'
+    direction='y',               # Along probe axis
+    rigid=False,                 # False for non-rigid
+    bin_s=2.0,                   # Temporal resolution (seconds)
+    win_step_um=50,              # Spatial window step
+    win_sigma_um=150,            # Spatial smoothing sigma
+    margin_um=0,                 # Margin at probe edges
+    win_scale_um=150,            # Window scale for weights
+)
+```
+
+## Troubleshooting
+
+### Over-correction (Wavy Patterns)
+
+```python
+# Increase temporal smoothing
+motion = estimate_motion(..., bin_s=5.0)  # Larger bins
+
+# Or use rigid correction for small drift
+motion = estimate_motion(..., rigid=True)
+```
+
+### Under-correction (Drift Remains)
+
+```python
+# Decrease spatial window for finer non-rigid estimate
+motion = estimate_motion(..., win_step_um=25, win_sigma_um=75)
+
+# Use more peaks
+peaks = detect_peaks(..., detect_threshold=4)  # Lower threshold
+```
+
+### Edge Artifacts
+
+```python
+rec_corrected = interpolate_motion(
+    rec, motion, temporal_bins, spatial_bins,
+    border_mode='force_extrapolate',  # or 'remove_channels'
+    spatial_interpolation_method='kriging'
+)
+```
+
+## Validation
+
+After correction, re-visualize to confirm:
+
+```python
+# Re-detect peaks on corrected recording
+peaks_corrected = detect_peaks(rec_corrected, ...)
+peak_locations_corrected = localize_peaks(rec_corrected, peaks_corrected, ...)
+
+# Plot before/after comparison
+fig, axes = plt.subplots(1, 2, figsize=(14, 6))
+
+# Before
+si.plot_drift_raster_map(peaks, peak_locations, rec, ax=axes[0])
+axes[0].set_title('Before Correction')
+
+# After
+si.plot_drift_raster_map(peaks_corrected, peak_locations_corrected,
+                         rec_corrected, ax=axes[1])
+axes[1].set_title('After Correction')
+```
+
+## References
+
+- [SpikeInterface Motion Correction Docs](https://spikeinterface.readthedocs.io/en/stable/modules/motion_correction.html)
+- [Handle Drift Tutorial](https://spikeinterface.readthedocs.io/en/stable/how_to/handle_drift.html)
+- [DREDge GitHub](https://github.com/evarol/DREDge)
+- Windolf et al. (2023) "DREDge: robust motion correction for high-density extracellular recordings"