aind-ecephys-s3-to-phy 0.4.0__tar.gz

aind_ecephys_s3_to_phy-0.4.0/.copier-answers.yml

@@ -0,0 +1,23 @@
+# Changes here will be overwritten by Copier; NEVER EDIT MANUALLY
+_commit: 0.1.6
+_src_path: /home/galen.lynch/Documents/Code/aind-python-copier-template
+author_email: galen@galenlynch.com
+author_fullname: Galen Lynch
+current_year: '2026'
+github_org: AllenNeuralDynamics
+github_url: https://github.com/AllenNeuralDynamics/aind-ecephys-s3-to-phy
+include_cli: false
+include_docs: false
+include_notebooks: false
+include_type_hints: true
+institute_name: Allen Institute for Neural Dynamics
+license: MIT
+project_description: Tools to get phy-ready data from AIND ecephys data on S3
+project_name: aind-ecephys-s3-to-phy
+project_slug: aind-ecephys-s3-to-phy
+publish_to_pypi: false
+python_package_import_name: aind_ecephys_s3_to_phy
+python_version: '3.10'
+target_python_version: '3.13'
+workflow_source: AllenNeuralDynamics/galen-uv-workflows
+workflow_version: v2

aind_ecephys_s3_to_phy-0.4.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,38 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+**Smartphone (please complete the following information):**
+ - Device: [e.g. iPhone6]
+ - OS: [e.g. iOS8.1]
+ - Browser [e.g. stock browser, safari]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.

aind_ecephys_s3_to_phy-0.4.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

aind_ecephys_s3_to_phy-0.4.0/.github/ISSUE_TEMPLATE/user-story.md

@@ -0,0 +1,27 @@
+---
+name: User story
+about: This template provides a basic structure for user story issues.
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+# User story
+As a ..., I want to ..., so I can ...
+
+*Ideally, this is in the issue title, but if not, you can put it here. If so, delete this section.*
+
+# Acceptance criteria
+- [ ] This is something that can be verified to show that this user story is satisfied.
+
+# Sprint Ready Checklist 
+- [ ] 1. Acceptance criteria defined 
+- [ ] 2. Team understands acceptance criteria 
+- [ ] 3. Team has defined solution / steps to satisfy acceptance criteria 
+- [ ] 4. Acceptance criteria is verifiable / testable 
+- [ ] 5. External / 3rd Party dependencies identified
+- [ ] 6. Ticket is prioritized and sized
+
+# Notes
+*Add any helpful notes here.*

aind_ecephys_s3_to_phy-0.4.0/.github/dependabot.yml

@@ -0,0 +1,19 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
+
+version: 2
+updates:
+  # Enable version updates for GitHub Actions
+  - package-ecosystem: "github-actions"
+    # Workflow files stored in the default location of `.github/workflows`
+    # You don't need to specify `/.github/workflows` for `directory`. You can use `directory: "/"`.
+    directory: "/"
+    schedule:
+      interval: "weekly"
+  # Enable version updates for pip
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "weekly"

aind_ecephys_s3_to_phy-0.4.0/.github/workflows/bump-call.yml

@@ -0,0 +1,33 @@
+name: Bump Version
+
+on:
+  workflow_run:
+    workflows: ["CI"]
+    branches: [ main ]
+    types: [ completed ]
+  workflow_dispatch:
+
+permissions:
+  contents: write    # needed to push commits/tags
+  id-token: write    # needed if your reusable workflow exchanges OIDC tokens
+
+concurrency:
+  group: bump-${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  bump:
+    # Only run when:
+    #  • a CI run just succeeded (and the head commit message doesn't start with 'bump:')
+    #  • OR this is a manual dispatch
+    if: >
+      (github.event_name == 'workflow_run' &&
+       github.event.workflow_run.conclusion == 'success' &&
+       !startsWith(github.event.workflow_run.head_commit.message, 'bump:')) ||
+      (github.event_name == 'workflow_dispatch' &&
+      github.ref == 'refs/heads/main')
+    uses: AllenNeuralDynamics/galen-uv-workflows/.github/workflows/bump-version.yml@v2
+    with:
+      default-branch: main
+    secrets:
+      repo-token: ${{ secrets.SERVICE_TOKEN }}

aind_ecephys_s3_to_phy-0.4.0/.github/workflows/ci-call.yml

@@ -0,0 +1,38 @@
+name: CI
+
+on:
+  workflow_dispatch:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+    types: [opened, synchronize, reopened, ready_for_review]
+
+permissions:
+  contents: read
+  pull-requests: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  run-ci:
+    strategy:
+      matrix:
+        os: [ubuntu-latest]
+        python-version: ["3.11", "3.13"]
+
+    if: >
+      (github.event_name != 'pull_request' || github.event.pull_request.draft == false) &&
+      github.actor != 'github-actions[bot]' &&
+      !(github.event_name == 'push' && startsWith(github.event.head_commit.message, 'bump:'))
+
+    uses: AllenNeuralDynamics/galen-uv-workflows/.github/workflows/ci.yml@v2
+    with:
+      os: ${{ matrix.os }}
+      python-version: ${{ matrix.python-version }}
+      package-name: aind_ecephys_s3_to_phy
+      run-mypy: true
+    secrets:
+      repo-token: ${{ secrets.GITHUB_TOKEN }}

aind_ecephys_s3_to_phy-0.4.0/.github/workflows/publish-call.yml

@@ -0,0 +1,25 @@
+name: "Release"
+
+on:
+  push:
+    tags:
+      - 'v[0-9]*'          # semver tags
+  workflow_dispatch:
+
+permissions:
+  contents: write    # create release + upload assets
+
+concurrency:
+  group: release
+  cancel-in-progress: false   # let releases finish
+
+jobs:
+  run-release:
+    uses: AllenNeuralDynamics/galen-uv-workflows/.github/workflows/publish.yml@v2
+    with:
+      tag-name: ${{ github.ref_name }}
+      python-version: "3.13"
+      publish-to-pypi: true
+    secrets:
+      repo-token: ${{ secrets.GITHUB_TOKEN }}
+      pypi-token: ${{ secrets.AIND_PYPI_TOKEN }}

aind_ecephys_s3_to_phy-0.4.0/.gitignore

@@ -0,0 +1,148 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+.idea/
+
+# MacOs
+**/.DS_Store
+
+# VSCode
+.vscode/
+
+# Claude
+.claude/
+
+# MWE working directory
+mwe_workdir/

aind_ecephys_s3_to_phy-0.4.0/.python-version

@@ -0,0 +1 @@
+3.13

aind_ecephys_s3_to_phy-0.4.0/.readthedocs.yaml

@@ -0,0 +1,26 @@
+# .readthedocs.yaml
+version: 2
+
+# RTD will run `sphinx-build` for you using this conf.py
+sphinx:
+  configuration: docs/source/conf.py
+
+build:
+  os: ubuntu-24.04
+  tools:
+    python: "3.13"
+
+  jobs:
+    # Install uv once at the OS level, before creating the venv
+    pre_create_environment:
+      - asdf plugin add uv
+      - asdf install uv latest
+      - asdf global uv latest
+
+    # Create the RTD virtualenv using uv so RTD will use it for the build
+    create_environment:
+      - uv venv "$READTHEDOCS_VIRTUALENV_PATH"
+
+    # Install your docs deps from pyproject (respects uv.lock if present)
+    install:
+      - UV_PROJECT_ENVIRONMENT="$READTHEDOCS_VIRTUALENV_PATH" uv sync --frozen --group docs

aind_ecephys_s3_to_phy-0.4.0/CHANGELOG.md

@@ -0,0 +1,53 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## v0.4.0 (2026-10-24)
+
+### Feat
+
+- add lazy-trace export to convert
+
+### Fix
+
+- correct traces with KS4's saved motion via aind-ephys-assets
+
+## v0.3.0 (2026-04-12)
+
+### Feat
+
+- apply KS4 motion correction when exporting to phy
+- populate cluster_group.tsv from KSLabel + decoder_label
+
+### Fix
+
+- remove unsupported overwrite kwarg from Motion.save() in tests (#4)
+- stash binary files in parent dir during re-export
+
+## v0.2.0 (2026-03-30)
+
+### Feat
+
+- write binary before spike export, compute PCs from local disk
+- make export_phy idempotent for binary write and compression
+- add --check-compression flag to skip mtscomp verification
+- enable kebab-case CLI flags and subcommand names
+- support local paths in discovery functions
+- add --compress flag to extract_recording subcommand
+- add extract_recording subcommand and rewrite README
+- implement S3-to-phy conversion pipeline with CLI
+
+### Fix
+
+- guarantee binary file restoration with try/finally
+- stash binary files in parent dir during re-export
+- preserve binary files when re-exporting spike data
+- enable amplitudes and PC features in phy export
+- use thread pool for binary writing instead of fork
+
+### Refactor
+
+- switch CLI from pydantic-settings to tyro

aind_ecephys_s3_to_phy-0.4.0/CLAUDE.md

@@ -0,0 +1,135 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+**aind-ecephys-s3-to-phy** — CLI tools to convert AIND ecephys sorted recordings and raw recordings (two separate S3 assets) into [phy](https://github.com/cortex-lab/phy) directories for manual curation, including raw trace access. Uses [SpikeInterface](https://github.com/SpikeInterface/spikeinterface) for the conversion and [pydantic-settings](https://docs.pydantic.dev/latest/concepts/pydantic_settings/) for CLI argument parsing.
+
+`src/` layout, `hatchling` build backend, `uv` for dependency management.
+
+## Commands
+
+```bash
+# Setup
+uv sync
+
+# Full linting + tests (use before committing)
+./scripts/run_linters_and_checks.sh -c
+
+# Format only (no checks)
+./scripts/run_linters_and_checks.sh
+
+# Individual tools (when venv is active, drop "uv run --frozen")
+uv run --frozen ruff format
+uv run --frozen ruff check
+uv run --frozen mypy
+uv run --frozen interrogate -v
+uv run --frozen codespell --check-filenames
+uv run --frozen pytest --cov aind_ecephys_s3_to_phy
+
+# Run a single test
+uv run --frozen pytest tests/test_example.py -k "test_name"
+
+# Pass extra pytest args via the script
+./scripts/run_linters_and_checks.sh -c -- -k "test_name"
+```
+
+## Code Style & Tooling
+
+- **Line length**: 120
+- **Target Python**: 3.11+ (minimum 3.10)
+- **Ruff rules**: Q, RUF100, C90, I, F, E, W, D, UP, ANN, PYI (type annotations not required in tests)
+- **Docstrings**: NumPy convention, 30% coverage minimum (interrogate)
+- **Type checking**: mypy strict mode on `src/` (relaxed in tests)
+- **Commits**: Angular convention via commitizen (semantic versioning)
+
+## Architecture
+
+### Domain: AIND Ecephys Data on S3
+
+Two separate S3 assets are involved per session:
+
+1. **Raw asset** (e.g. `s3://aind-open-data/ecephys_786867_2025-09-25_12-43-56/`)
+   - Contains compressed Zarr recordings under `ecephys/ecephys_compressed/`
+   - One Zarr per probe per experiment (e.g. `experiment1_Record Node 109#Neuropix-PXI-100.46101.zarr`)
+
+2. **Sorted asset** (e.g. `s3://aind-open-data/ecephys_786867_2025-09-25_12-43-56_sorted_2025-12-24_08-46-14/`)
+   - `postprocessed/` — SortingAnalyzer as Zarr (one per recording), with precomputed extensions (templates, waveforms, PCs, quality metrics, etc.)
+   - `preprocessed/` — JSON files describing SpikeInterface preprocessing chains (not actual trace data)
+   - `spikesorted/` — NumpySorting format
+   - `curated/` — Subset with auto-curation labels
+
+Each sorted asset can have many recordings (e.g. 10 probes x 3 experiments = 30). Users select specific probes/streams/experiments to export.
+
+### Conversion Pipeline
+
+```
+SortingAnalyzer (S3)
+  ├─ Load from S3 (stream, don't download)
+  ├─ Extract recording dict (full preprocessing chain with ChannelSlice)
+  └─ Load templates extension (lazy, on demand)
+         │
+         ▼
+Recording dict  ──→  Substitute raw Zarr S3 URI at leaf
+                     ──→  _load_extractor_from_dict()
+                          ──→  Preprocessed recording (373 ch, filtered)
+                               │
+                               ├─ Attach to analyzer via _recording = rec
+                               │
+                               ▼
+                     si.export_to_phy(analyzer, copy_binary=False)
+                          ──→  spike data (.npy, .tsv, params.py)
+                               │
+                               ▼
+                     write_binary_recording(rec, "recording.dat")
+                          ──→  Streams from S3 → flat .dat
+                               │
+                               ▼
+                     mtscomp.compress("recording.dat")
+                          ──→  recording.cbin + recording.ch
+                               ──→  Update params.py, delete .dat
+```
+
+Key details:
+- **Preprocessing reconstruction**: `get_preprocessing_dict_from_analyzer()` extracts the chain from the SortingAnalyzer Zarr. `apply_preprocessing_pipeline()` applies it to the raw recording, so phy shows filtered/referenced traces (not raw).
+- **mtscomp (.cbin)**: Phy natively supports `.cbin` files via phylib's `MtscompEphysReader`. ~3x lossless compression with random-access decompression. Avoids storing massive flat `.dat` files.
+- **Phy trace format support**: `.dat`/`.bin`/`.raw`/`.mda` (flat binary), `.cbin` (mtscomp), `.npy` (numpy). No Zarr/HDF5.
+- **`export_to_phy` requires**: SortingAnalyzer with at least `templates` extension computed. Single-segment recordings only.
+- **S3 access**: SpikeInterface uses fsspec/s3fs. Anonymous access tried first (works for public `aind-open-data`). Standard AWS credential chain for private buckets.
+
+### MWE Results & Design Decisions (from `scripts/test_data_access.py`)
+
+**Data access strategy** (benchmarked on a single recording, 60s binary write):
+- **SortingAnalyzer**: Stream from S3 (3s load + 2.4s templates). Do NOT download first (55s download + 9s load is much slower). Extensions are lazy-loaded on demand for remote paths.
+- **Raw recording → binary**: Stream directly from S3 during `write_binary_recording` (87s for 60s of data). Only 1.35x slower than local, and avoids the ~290s download of 30 GB raw zarr. **Streaming is the right default.**
+- **mtscomp**: 2.52x compression ratio, 10s for 1.3 GB. Fast and worth doing by default.
+
+**Critical gotchas discovered**:
+- **`get_preprocessing_dict_from_analyzer()` skips `ChannelSlice` steps** (by design). The AIND pipeline applies bad-channel removal via `ChannelSliceRecording`, so the simplified pipeline produces a recording with 384 channels while the analyzer expects ~373. **Must reconstruct the full chain** from the analyzer's stored recording dict (`zarr_root["recording"][0]`) and substitute the raw recording path at the leaf `ZarrRecordingExtractor`.
+- **`export_to_phy` ignores `_temporary_recording`**: The function checks `has_recording()` which only looks at `_recording`, not `_temporary_recording`. Workaround: set `analyzer._recording = preprocessed_rec` directly after validation via `set_temporary_recording()`.
+- **Raw recordings use WavPack codec**: Requires `wavpack-numcodecs` package.
+- **Analyzer naming**: Postprocessed zarr names have `_recording1` suffix (e.g. `...50213-1_recording1.zarr`) while raw zarr names don't (e.g. `...50213-1.zarr`). Strip `_recording\d+` to match.
+- **Only experiment1 has postprocessed data** in the test dataset, and raw recordings are ~30 GB each (95 GB as flat .dat).
+
+### CLI Design (pydantic-settings)
+
+Uses `pydantic_settings.BaseSettings` with `CliApp.run` pattern and subcommands:
+
+- **Happy-path command**: One command that downloads + preprocesses + exports + compresses
+- **Subcommands**: Separate steps for download, export, compress — for flexibility/debugging
+- **Entry point**: `[project.scripts]` in pyproject.toml
+
+Output structure: working directory based on S3 asset name, with phy folders per probe/experiment.
+
+### Key Dependencies
+
+- **spikeinterface** (PyPI) — SortingAnalyzer loading, preprocessing pipeline, `export_to_phy`
+- **pydantic-settings** — CLI argument parsing via BaseSettings
+- **mtscomp** — Compress recording.dat → .cbin for phy trace view
+- **s3fs/fsspec** — S3 access (transitive via spikeinterface)
+
+### Local Reference Repos
+
+- SpikeInterface: `~/Documents/Code/spikeinterface`
+- phy: `~/Documents/Code/phy`

aind_ecephys_s3_to_phy-0.4.0/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2026 Allen Institute for Neural Dynamics
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+