ethoscopy 2.0.3__tar.gz → 2.0.5__tar.gz

ethoscopy-2.0.5/.claude/settings.local.json

@@ -0,0 +1,40 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(wc:*)",
+      "Bash(paplay:*)",
+      "Bash(ETHOSCOPE_DATA_PATH=/mnt/ethoscope_data docker compose:*)",
+      "Bash(docker ps:*)",
+      "Bash(docker logs:*)",
+      "Bash(docker compose:*)",
+      "Bash(curl:*)",
+      "Bash(docker exec:*)",
+      "Bash(mkdir -p /tmp/ethoscopy_wheel_inspect)",
+      "Read(//tmp/**)",
+      "Bash(pip download *)",
+      "Bash(unzip -l ethoscopy-2.0.4-py3-none-any.whl)",
+      "Bash(git check-ignore *)",
+      "Bash(/home/gg/Code/ethoscope_project/ethoscopy/.venv/bin/pip install *)",
+      "Bash(.venv/bin/python -m pytest tests/test_get_tutorials.py -v)",
+      "Bash(.venv/bin/python *)",
+      "Bash(.venv/bin/pip install *)",
+      "Bash(gh auth *)",
+      "Bash(gh repo *)",
+      "Bash(docker info *)",
+      "mcp__bookstack__bookstack_search",
+      "Bash(.venv/bin/ruff check *)",
+      "Bash(.venv/bin/black --check src/ tests/)",
+      "Bash(.venv/bin/black src/ethoscopy/misc/get_tutorials.py tests/test_get_tutorials.py)",
+      "mcp__bookstack__bookstack_pages_read",
+      "mcp__bookstack__bookstack_pages_update",
+      "Bash(gh issue create --repo gilestrolab/ethoscopy --title 'Tutorial data pickles missing from PyPI wheel \\(2.0.0 – 2.0.4\\)' --body ' *)",
+      "Bash(git add *)",
+      "Bash(git commit *)",
+      "Bash(git push *)",
+      "Bash(git tag *)",
+      "Bash(gh release create v2.0.5 --repo gilestrolab/ethoscopy --title 'v2.0.5 — package tutorial data separately' --notes ' *)",
+      "Bash(gh run *)",
+      "Bash(gh issue *)"
+    ]
+  }
+}

{ethoscopy-2.0.3 → ethoscopy-2.0.5}/.pre-commit-config.yaml

@@ -51,7 +51,7 @@ repos:
         language: system
         pass_filenames: false
         always_run: true
-        stages: [commit]
+        stages: [pre-commit]
 
       - id: pytest-fast
         name: pytest-fast
@@ -59,4 +59,4 @@ repos:
         language: system
         pass_filenames: false
         always_run: true
-        stages: [push]
+        stages: [pre-push]

{ethoscopy-2.0.3 → ethoscopy-2.0.5}/CLAUDE.md

@@ -45,6 +45,41 @@ pip install -e ".[dev]"
 - Build: `JUPYTER_HUB_TAG=5.3.0 ETHOSCOPE_LAB_TAG=1.0 docker compose build`
 - Run: `docker compose up -d` (from Docker/ directory)
 
+### Troubleshooting: Database "Malformed" Errors
+
+**Problem**: Intermittent "database disk image is malformed" errors when loading ethoscope data in Docker
+
+**Root Cause**: SQLite databases in WAL (Write-Ahead Logging) mode on read-only Docker mounts
+
+**Solution Options**:
+1. **Convert databases to DELETE mode** (recommended for immediate fix)
+   ```bash
+   # Using the conversion script (from project root)
+   python3 scripts/convert_wal_to_delete.py /mnt/ethoscope_data/results --verbose
+
+   # Or using bash wrapper
+   ./scripts/convert_databases.sh /mnt/ethoscope_data/results
+   ```
+
+2. **Improved connection handling** (v2.0.4+)
+   - The `_connect_db()` function in `load.py` now detects WAL mode automatically
+   - Uses `mode=ro&nolock=1` URI parameters for WAL databases on read-only mounts
+   - Includes retry logic in `read_single_roi_optimized()` for resilience
+
+**Key Implementation Details** (load.py):
+- Lines 18-70: `_connect_db()` - Smart connection with WAL detection and appropriate parameters
+- Lines 943-979: Retry logic in `read_single_roi_optimized()` - Handles transient errors with fresh connections
+
+**Testing After Changes**:
+```python
+import ethoscopy as etho
+metadata = etho.link_meta_index('metadata.csv', '/mnt/ethoscope_results')
+data = etho.load_ethoscope(metadata, reference_hour=9.0, FUN=etho.sleep_annotation)
+# Should load all ROIs without "malformed" errors
+```
+
+**See Also**: `Docker/README.md` for detailed database preparation instructions
+
 ## Architecture and Code Structure
 
 ### Core Architecture

ethoscopy-2.0.5/Docker/.env.dummy.example

@@ -0,0 +1,25 @@
+# ============================================================================
+# DUMMY AUTHENTICATOR (Simple Shared Password)
+# ============================================================================
+# Default authentication method - all users share the same password
+# Best for: Lab environments, small teams, trusted networks
+
+# Authenticator type (must be 'dummy' for this configuration)
+AUTHENTICATOR_CLASS=dummy
+
+# Shared password for all users
+DUMMY_PASSWORD=ethoscope
+
+# Comma-separated list of allowed users
+ALLOWED_USERS=amadabhushi,ggilestro,mjoyce,lguo,labguest1,labguest2,labguest3,labguest4,labguest5,labguest6,labguest7,labguest8,ethoscopelab
+
+# Comma-separated list of admin users
+ADMIN_USERS=ggilestro
+
+# Docker user/group (optional, defaults to 1000:1000)
+UID=1000
+GID=1000
+
+# JupyterHub and Ethoscope Lab versions
+JUPYTER_HUB_TAG=5.3.0
+ETHOSCOPE_LAB_TAG=1.1

ethoscopy-2.0.5/Docker/.env.github.example

@@ -0,0 +1,39 @@
+# ============================================================================
+# GITHUB OAUTH AUTHENTICATOR
+# ============================================================================
+# Authenticate users via their GitHub accounts
+# Best for: Open source projects, teams already using GitHub
+#
+# Setup Instructions:
+# 1. Go to GitHub Settings > Developer Settings > OAuth Apps
+# 2. Create a new OAuth App
+# 3. Set Homepage URL: http://your-server:8082
+# 4. Set Authorization callback URL: http://your-server:8082/hub/oauth_callback
+# 5. Copy the Client ID and generate a Client Secret
+
+# Authenticator type
+AUTHENTICATOR_CLASS=github
+
+# OAuth credentials from GitHub OAuth App
+OAUTH_CLIENT_ID=your_github_client_id_here
+OAUTH_CLIENT_SECRET=your_github_client_secret_here
+
+# Callback URL (must match what you set in GitHub OAuth App)
+OAUTH_CALLBACK_URL=http://your-server:8082/hub/oauth_callback
+
+# (Optional) Restrict to GitHub organization members
+# GITHUB_ORG=your-organization-name
+
+# Comma-separated list of allowed users (GitHub usernames)
+ALLOWED_USERS=username1,username2,username3
+
+# Comma-separated list of admin users (GitHub usernames)
+ADMIN_USERS=your-github-username
+
+# Docker user/group (optional, defaults to 1000:1000)
+UID=1000
+GID=1000
+
+# JupyterHub and Ethoscope Lab versions
+JUPYTER_HUB_TAG=5.3.0
+ETHOSCOPE_LAB_TAG=1.1

ethoscopy-2.0.5/Docker/.env.google.example

@@ -0,0 +1,41 @@
+# ============================================================================
+# GOOGLE OAUTH AUTHENTICATOR
+# ============================================================================
+# Authenticate users via their Google accounts
+# Best for: Organizations using Google Workspace, universities with Google
+#
+# Setup Instructions:
+# 1. Go to Google Cloud Console (console.cloud.google.com)
+# 2. Create a new project or select existing
+# 3. Enable Google+ API
+# 4. Go to Credentials > Create Credentials > OAuth 2.0 Client ID
+# 5. Set Application type: Web application
+# 6. Add Authorized redirect URI: http://your-server:8082/hub/oauth_callback
+# 7. Copy the Client ID and Client Secret
+
+# Authenticator type
+AUTHENTICATOR_CLASS=google
+
+# OAuth credentials from Google Cloud Console
+OAUTH_CLIENT_ID=your_google_client_id.apps.googleusercontent.com
+OAUTH_CLIENT_SECRET=your_google_client_secret_here
+
+# Callback URL (must match what you set in Google Cloud Console)
+OAUTH_CALLBACK_URL=http://your-server:8082/hub/oauth_callback
+
+# (Optional) Restrict to Google Workspace domain
+# GOOGLE_HOSTED_DOMAIN=yourdomain.com
+
+# Comma-separated list of allowed users (Gmail addresses without @gmail.com or @yourdomain.com)
+ALLOWED_USERS=user1,user2,user3
+
+# Comma-separated list of admin users
+ADMIN_USERS=your-email-username
+
+# Docker user/group (optional, defaults to 1000:1000)
+UID=1000
+GID=1000
+
+# JupyterHub and Ethoscope Lab versions
+JUPYTER_HUB_TAG=5.3.0
+ETHOSCOPE_LAB_TAG=1.1

ethoscopy-2.0.5/Docker/.env.keycloak.example

@@ -0,0 +1,53 @@
+# ============================================================================
+# KEYCLOAK / GENERIC OIDC AUTHENTICATOR
+# ============================================================================
+# Authenticate users via Keycloak or any OIDC-compliant provider
+# Best for: Organizations with existing SSO, enterprise environments
+#
+# Setup Instructions for Keycloak:
+# 1. Log into your Keycloak Admin Console
+# 2. Select your realm (e.g., 'gilestro-lab')
+# 3. Go to Clients > Create Client
+# 4. Set Client ID: jupyterhub
+# 5. Set Client Protocol: openid-connect
+# 6. Set Access Type: confidential
+# 7. Set Valid Redirect URIs: http://your-server:8082/hub/oauth_callback
+# 8. Save and go to Credentials tab to get the Client Secret
+# 9. Create users in Keycloak matching your existing home directory names
+
+# Authenticator type
+AUTHENTICATOR_CLASS=generic
+
+# OAuth credentials from Keycloak client
+OAUTH_CLIENT_ID=jupyterhub
+OAUTH_CLIENT_SECRET=your_keycloak_client_secret_here
+
+# Callback URL
+OAUTH_CALLBACK_URL=http://your-server:8082/hub/oauth_callback
+
+# Keycloak OIDC endpoints (replace YOUR_KEYCLOAK_SERVER and YOUR_REALM)
+# Example: http://keycloak.example.com/realms/gilestro-lab
+OAUTH_AUTHORIZE_URL=http://YOUR_KEYCLOAK_SERVER/realms/YOUR_REALM/protocol/openid-connect/auth
+OAUTH_TOKEN_URL=http://YOUR_KEYCLOAK_SERVER/realms/YOUR_REALM/protocol/openid-connect/token
+OAUTH_USERDATA_URL=http://YOUR_KEYCLOAK_SERVER/realms/YOUR_REALM/protocol/openid-connect/userinfo
+
+# Username claim (Keycloak uses 'preferred_username' by default)
+OAUTH_USERNAME_CLAIM=preferred_username
+
+# OAuth scopes (comma-separated, optional)
+OAUTH_SCOPE=openid,profile,email
+
+# CRITICAL: Keycloak usernames MUST match existing home directory names exactly
+# Comma-separated list of allowed users
+ALLOWED_USERS=amadabhushi,ggilestro,mjoyce,lguo,labguest1,labguest2,labguest3,labguest4,labguest5,labguest6,labguest7,labguest8,ethoscopelab
+
+# Comma-separated list of admin users
+ADMIN_USERS=ggilestro
+
+# Docker user/group (optional, defaults to 1000:1000)
+UID=1000
+GID=1000
+
+# JupyterHub and Ethoscope Lab versions
+JUPYTER_HUB_TAG=5.3.0
+ETHOSCOPE_LAB_TAG=1.1

{ethoscopy-2.0.3 → ethoscopy-2.0.5}/Docker/Dockerfile

@@ -70,16 +70,23 @@ RUN pip3 install jupyterlab-spreadsheet-editor
 # ============================================================================
 
 # Install Python data science packages
+# Including oauthenticator for flexible authentication support
 RUN pip3 install \
+        oauthenticator \
         pandas \
         seaborn \
         bokeh \
         lifelines \
         opencv-python \
         mysql-connector \
-        ethoscopy 
+        ethoscopy
        # pycatch22
 
+# Pre-populate tutorial datasets inside the installed package so non-root
+# JupyterHub users don't have to download them on first notebook run. The
+# pickles (~36 MB total) live in the ethoscopy repo, not the PyPI wheel.
+RUN python3 -c "from ethoscopy.misc.get_tutorials import download_tutorial_data, package_tutorial_data_dir; download_tutorial_data(dest_dir=package_tutorial_data_dir(), verbose=True)"
+
 # Install Ethoscope from source
 RUN git clone https://github.com/gilestrolab/ethoscope.git /opt/ethoscope && \
     cd /opt/ethoscope/ && git checkout dev && \

ethoscopy-2.0.5/Docker/README.md

@@ -0,0 +1,220 @@
+# Ethoscopelab docker instance
+
+Note: Most users will **not** need to recreate this image. These instructions are just provided as reference.
+The ethoscopelab docker instance lives on dockerhub at the following address: [https://hub.docker.com/r/ggilestro/ethoscope-lab](https://hub.docker.com/r/ggilestro/ethoscope-lab) and this is what regular users should download and run. Follow instructions there and on the [ethoscopy manual](https://bookstack.lab.gilest.ro/books/ethoscopy/page/getting-started).
+
+
+## Docker files that were used to create the ethoscope-lab docker instance
+
+The files in this folder can be used to recreate the image as uploaded on dockerhub.
+
+The command to use to recreate that image is `JUPYTER_HUB_TAG=5.3.0 ETHOSCOPE_LAB_TAG=1.0 docker compose build`. This creates the image with the specified tag. For Docker Hub deployment, push the image: `docker push ggilestro/ethoscope-lab:1.0`. To also create a latest tag: `docker tag ggilestro/ethoscope-lab:1.0 ggilestro/ethoscope-lab:latest && docker push ggilestro/ethoscope-lab:latest`. You can verify your local images with `docker images | grep ethoscope-lab`.
+
+After creation, the image can be run using the enclosed `docker-compose.yml` file, replacing values as fit.
+
+## Authentication and User Management
+
+This JupyterHub instance supports multiple authentication methods including:
+- **Keycloak OAuth** (current production setup at auth.gilest.ro)
+- GitHub OAuth
+- Google OAuth
+- GitLab OAuth
+- Dummy authenticator (shared password)
+
+### Adding New Users
+
+**⚠️ Important**: Adding a new user requires steps on BOTH the authentication provider AND the JupyterHub host.
+
+For detailed instructions on adding users, making users admins, and switching authentication methods, see:
+
+**📖 [README_AUTH.md](./README_AUTH.md) - Complete Authentication and User Management Guide**
+
+### Quick Reference
+
+**For Keycloak (current setup)**:
+1. Add user to Keycloak at `auth.gilest.ro` in the `gilestro-lab` realm
+2. Create home directory on host: `sudo mkdir -p /mnt/homes/username && sudo chown 1000:1000 /mnt/homes/username`
+3. (Optional) Add to `config/users.py` if using whitelist enforcement
+
+**For Dummy Auth**:
+1. Add username to `config/users.py` or `.env` file
+2. Create home directory on host
+3. Restart container
+
+See [README_AUTH.md](./README_AUTH.md) for complete step-by-step instructions.
+
+## Mounting Home Directories as Volumes
+
+To persist user data and notebooks across container restarts, you should mount user home directories as Docker volumes. This is done by modifying the `docker-compose.yml` file.
+
+### Benefits of mounting home directories:
+
+1. **Data Persistence**: User notebooks, data files, and configurations survive container restarts and updates
+2. **Backup and Recovery**: Easy to backup user data by copying the mounted directories
+3. **Performance**: Direct access to host filesystem, avoiding container storage overhead
+4. **Sharing**: Users can access their files from the host system if needed
+
+### Example volume configuration:
+
+Add volumes to your `docker-compose.yml`:
+
+```yaml
+services:
+  jupyterhub:
+    volumes:
+      - ./user_data:/home  # Maps host ./user_data to container /home
+      - ./jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.py
+```
+
+Or for individual user directories:
+
+```yaml
+volumes:
+  - ./users/ggilestro:/home/ggilestro
+  - ./users/amadabhushi:/home/amadabhushi
+  - ./users/shared:/home/shared  # Shared directory for all users
+```
+
+This ensures all user work is preserved even when containers are recreated or updated.
+
+## Database Preparation for Docker
+
+When mounting ethoscope database files into Docker containers with read-only permissions (`:ro`), you may encounter "database disk image is malformed" errors. This occurs when SQLite databases are in WAL (Write-Ahead Logging) mode but mounted on read-only filesystems where WAL companion files (`.db-wal`, `.db-shm`) cannot be accessed.
+
+### Understanding the Issue
+
+**Why this happens:**
+- Ethoscope databases may be created in **WAL mode** for better write performance
+- WAL mode requires companion files (`.db-wal` and `.db-shm`) alongside the main `.db` file
+- Docker read-only mounts (`:ro`) prevent SQLite from accessing these files properly
+- This causes intermittent "database disk image is malformed" errors during data loading
+
+**Symptoms:**
+- Intermittent loading failures for specific ROIs
+- Error message: "database disk image is malformed"
+- Same ROI may succeed on some loads and fail on others
+- More common with large databases (>1 GB)
+
+### Solution: Convert Databases to DELETE Mode
+
+Before mounting databases in Docker, convert them from WAL to DELETE mode:
+
+#### Using the Python Script (Recommended)
+
+```bash
+# From the ethoscopy project root
+python3 scripts/convert_wal_to_delete.py /mnt/ethoscope_data/results --dry-run
+
+# Convert all databases with detailed output
+python3 scripts/convert_wal_to_delete.py /mnt/ethoscope_data/results --verbose
+
+# Force conversion even for recently modified files (use with caution)
+python3 scripts/convert_wal_to_delete.py /mnt/ethoscope_data/results --force
+```
+
+#### Using the Bash Wrapper
+
+```bash
+# From the ethoscopy project root
+./scripts/convert_databases.sh /mnt/ethoscope_data/results
+```
+
+#### Manual Conversion Using sqlite3
+
+For individual databases:
+
+```bash
+sqlite3 /path/to/database.db "PRAGMA wal_checkpoint(TRUNCATE); PRAGMA journal_mode=DELETE;"
+```
+
+For bulk conversion:
+
+```bash
+find /mnt/ethoscope_data/results -name "*.db" -type f -exec sqlite3 {} "PRAGMA wal_checkpoint(TRUNCATE); PRAGMA journal_mode=DELETE;" \;
+```
+
+### Verification
+
+After conversion, verify the database is in DELETE mode:
+
+```bash
+sqlite3 /path/to/database.db "PRAGMA journal_mode;"
+# Should output: delete
+
+sqlite3 /path/to/database.db "PRAGMA integrity_check;"
+# Should output: ok
+```
+
+### Best Practices
+
+1. **Run conversion BEFORE starting Docker containers**
+   ```bash
+   # From the ethoscopy project root, convert databases first
+   ./scripts/convert_databases.sh /mnt/ethoscope_data/results
+
+   # Then start containers
+   cd Docker && docker compose up -d
+   ```
+
+2. **Do NOT convert databases while ethoscopes are actively writing**
+   - The conversion script skips files modified in the last 24 hours by default
+   - Use `--force` to override this safety check if needed
+
+3. **Test on a backup first**
+   ```bash
+   # Create a test copy
+   cp /path/to/database.db /tmp/test.db
+
+   # Test conversion
+   sqlite3 /tmp/test.db "PRAGMA wal_checkpoint(TRUNCATE); PRAGMA journal_mode=DELETE;"
+
+   # Verify it works
+   sqlite3 /tmp/test.db "PRAGMA integrity_check;"
+   ```
+
+4. **Consider converting at the ethoscope source** (upstream fix)
+   - Configure ethoscopes to use DELETE mode by default
+   - Prevents the need for post-processing
+
+### Troubleshooting
+
+**Q: I still get "malformed" errors after conversion**
+
+A: The ethoscopy library now includes improved connection handling that should work with both WAL and DELETE mode databases. If you still encounter issues:
+1. Verify the database was actually converted: `sqlite3 database.db "PRAGMA journal_mode;"`
+2. Check database integrity: `sqlite3 database.db "PRAGMA integrity_check;"`
+3. Ensure you're using ethoscopy version 2.0.4 or later
+
+**Q: Can I convert databases while Docker is running?**
+
+A: Yes, but you should restart the container after conversion:
+```bash
+# From the ethoscopy project root
+./scripts/convert_databases.sh /mnt/ethoscope_data/results
+cd Docker && docker compose restart ethoscope-lab
+```
+
+**Q: Will this affect data quality or analysis?**
+
+A: No. The conversion only changes how SQLite manages the database internally. All data remains identical and analysis results are unaffected.
+
+**Q: How do I check if a database is in WAL mode?**
+
+A:
+```bash
+sqlite3 database.db "PRAGMA journal_mode;"
+```
+Output will be either `wal` or `delete` (or `truncate`, `persist`, `memory`).
+
+### Emergency Quick Fix
+
+If you need to work immediately and can't run the full conversion:
+
+```bash
+# SSH to host machine (not inside container)
+# Convert just the failing database
+sqlite3 /path/to/failing/database.db "PRAGMA wal_checkpoint(TRUNCATE); PRAGMA journal_mode=DELETE;"
+
+# Restart container
+docker compose restart ethoscope-lab
+```