ethoscopy 2.0.3__tar.gz → 2.0.4__tar.gz

{ethoscopy-2.0.3 → ethoscopy-2.0.4}/CLAUDE.md

@@ -45,6 +45,41 @@ pip install -e ".[dev]"
 - Build: `JUPYTER_HUB_TAG=5.3.0 ETHOSCOPE_LAB_TAG=1.0 docker compose build`
 - Run: `docker compose up -d` (from Docker/ directory)
 
+### Troubleshooting: Database "Malformed" Errors
+
+**Problem**: Intermittent "database disk image is malformed" errors when loading ethoscope data in Docker
+
+**Root Cause**: SQLite databases in WAL (Write-Ahead Logging) mode on read-only Docker mounts
+
+**Solution Options**:
+1. **Convert databases to DELETE mode** (recommended for immediate fix)
+   ```bash
+   # Using the conversion script (from project root)
+   python3 scripts/convert_wal_to_delete.py /mnt/ethoscope_data/results --verbose
+
+   # Or using bash wrapper
+   ./scripts/convert_databases.sh /mnt/ethoscope_data/results
+   ```
+
+2. **Improved connection handling** (v2.0.4+)
+   - The `_connect_db()` function in `load.py` now detects WAL mode automatically
+   - Uses `mode=ro&nolock=1` URI parameters for WAL databases on read-only mounts
+   - Includes retry logic in `read_single_roi_optimized()` for resilience
+
+**Key Implementation Details** (load.py):
+- Lines 18-70: `_connect_db()` - Smart connection with WAL detection and appropriate parameters
+- Lines 943-979: Retry logic in `read_single_roi_optimized()` - Handles transient errors with fresh connections
+
+**Testing After Changes**:
+```python
+import ethoscopy as etho
+metadata = etho.link_meta_index('metadata.csv', '/mnt/ethoscope_results')
+data = etho.load_ethoscope(metadata, reference_hour=9.0, FUN=etho.sleep_annotation)
+# Should load all ROIs without "malformed" errors
+```
+
+**See Also**: `Docker/README.md` for detailed database preparation instructions
+
 ## Architecture and Code Structure
 
 ### Core Architecture

ethoscopy-2.0.4/Docker/.claude/settings.local.json

@@ -0,0 +1,41 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(ls:*)",
+      "Bash(git checkout:*)",
+      "Bash(docker images:*)",
+      "Read(//tmp/**)",
+      "Bash(JUPYTER_HUB_TAG=5.3.0 ETHOSCOPE_LAB_TAG=1.1 docker compose build:*)",
+      "Bash(tee:*)",
+      "Bash(docker compose:*)",
+      "Bash(find:*)",
+      "Bash(sqlite3:*)",
+      "Bash(python3:*)",
+      "Bash(docker exec:*)",
+      "Bash(docker cp:*)",
+      "Bash(ip addr:*)",
+      "Bash(systemctl status:*)",
+      "Bash(docker ps:*)",
+      "Bash(wg show:*)",
+      "Bash(ip route:*)",
+      "Bash(ip rule:*)",
+      "Bash(docker inspect:*)",
+      "Bash(docker logs:*)",
+      "Bash(nc:*)",
+      "Bash(timeout 3 bash -c 'cat < /dev/tcp/192.168.254.10/8082')",
+      "Bash(docker network inspect:*)",
+      "Bash(ss:*)",
+      "Bash(host:*)",
+      "Bash(chmod:*)",
+      "Bash(tree:*)",
+      "Bash(source:*)",
+      "Bash(pip install:*)",
+      "Bash(git add:*)",
+      "Bash(git commit -m \"$\\(cat <<''EOF''\nfix: resolve \"database disk image is malformed\" errors for WAL-mode databases\n\nAdd comprehensive solution for SQLite WAL-mode database errors on read-only Docker mounts:\n\n- Improve connection handling in load.py:\n  - Detect WAL mode databases and use appropriate connection parameters\n  - Add retry logic with fresh connections for transient errors\n  - Use mode=ro&nolock=1 for WAL databases on read-only filesystems\n\n- Add database conversion scripts:\n  - convert_wal_to_delete.py: Python script to convert databases from WAL to DELETE mode\n  - convert_databases.sh: Bash wrapper for easy bulk conversion\n  - scripts/README.md: Comprehensive documentation for database utilities\n\n- Update documentation:\n  - Docker/README.md: Add \"Database Preparation for Docker\" section\n  - CLAUDE.md: Add troubleshooting section for database errors\n\n- Bump version to 2.0.4\n\nThis resolves intermittent loading failures for ROIs when ethoscope databases\nin WAL mode are mounted read-only in Docker containers.\n\nCo-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>\nEOF\n\\)\")",
+      "Bash(git push:*)",
+      "Bash(git tag:*)",
+      "Bash(python -m build:*)"
+    ],
+    "deny": []
+  }
+}

ethoscopy-2.0.4/Docker/README.md

@@ -0,0 +1,227 @@
+# Ethoscopelab docker instance
+
+Note: Most users will **not** need to recreate this image. These instructions are just provided as reference.
+The ethoscopelab docker instance lives on dockerhub at the following address: [https://hub.docker.com/r/ggilestro/ethoscope-lab](https://hub.docker.com/r/ggilestro/ethoscope-lab) and this is what regular users should download and run. Follow instructions there and on the [ethoscopy manual](https://bookstack.lab.gilest.ro/books/ethoscopy/page/getting-started).
+
+
+## Docker files that were used to create the ethoscope-lab docker instance
+
+The files in this folder can be used to recreate the image as uploaded on dockerhub.
+
+The command to use to recreate that image is `JUPYTER_HUB_TAG=5.3.0 ETHOSCOPE_LAB_TAG=1.0 docker compose build`. This creates the image with the specified tag. For Docker Hub deployment, push the image: `docker push ggilestro/ethoscope-lab:1.0`. To also create a latest tag: `docker tag ggilestro/ethoscope-lab:1.0 ggilestro/ethoscope-lab:latest && docker push ggilestro/ethoscope-lab:latest`. You can verify your local images with `docker images | grep ethoscope-lab`.
+
+After creation, the image can be run using the enclosed `docker-compose.yml` file, replacing values as fit.
+
+## Add new users to the JupyterHub
+
+To add new users to the JupyterHub instance:
+
+### 1. Modify the jupyterhub_config.py file
+
+Edit the `allowed_users` set on lines 14-19 to include your new usernames:
+
+```python
+c.Authenticator.allowed_users = {
+    'amadabhushi', 'ggilestro', 'mjoyce', 'lguo',
+    'labguest1', 'labguest2', 'labguest3', 'labguest4',
+    'labguest5', 'labguest6', 'labguest7', 'labguest8',
+    'ethoscopelab', 'newuser1', 'newuser2'  # Add your new users here
+}
+```
+
+To make a user an admin, add them to the `admin_users` set on line 22:
+
+```python
+c.Authenticator.admin_users = {'ggilestro', 'newadmin'}
+```
+
+### 2. Restart the Docker container
+
+After modifying the config file:
+
+```bash
+docker compose down
+docker compose up -d
+```
+
+**Notes:**
+- All users share the same password: `ethoscope` (line 11)
+- The system uses DummyAuthenticator for simple shared-password authentication
+- Each user gets their own home directory at `/home/{username}`
+- Home directories are created automatically when users first log in
+
+## Mounting Home Directories as Volumes
+
+To persist user data and notebooks across container restarts, you should mount user home directories as Docker volumes. This is done by modifying the `docker-compose.yml` file.
+
+### Benefits of mounting home directories:
+
+1. **Data Persistence**: User notebooks, data files, and configurations survive container restarts and updates
+2. **Backup and Recovery**: Easy to backup user data by copying the mounted directories
+3. **Performance**: Direct access to host filesystem, avoiding container storage overhead
+4. **Sharing**: Users can access their files from the host system if needed
+
+### Example volume configuration:
+
+Add volumes to your `docker-compose.yml`:
+
+```yaml
+services:
+  jupyterhub:
+    volumes:
+      - ./user_data:/home  # Maps host ./user_data to container /home
+      - ./jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.py
+```
+
+Or for individual user directories:
+
+```yaml
+volumes:
+  - ./users/ggilestro:/home/ggilestro
+  - ./users/amadabhushi:/home/amadabhushi
+  - ./users/shared:/home/shared  # Shared directory for all users
+```
+
+This ensures all user work is preserved even when containers are recreated or updated.
+
+## Database Preparation for Docker
+
+When mounting ethoscope database files into Docker containers with read-only permissions (`:ro`), you may encounter "database disk image is malformed" errors. This occurs when SQLite databases are in WAL (Write-Ahead Logging) mode but mounted on read-only filesystems where WAL companion files (`.db-wal`, `.db-shm`) cannot be accessed.
+
+### Understanding the Issue
+
+**Why this happens:**
+- Ethoscope databases may be created in **WAL mode** for better write performance
+- WAL mode requires companion files (`.db-wal` and `.db-shm`) alongside the main `.db` file
+- Docker read-only mounts (`:ro`) prevent SQLite from accessing these files properly
+- This causes intermittent "database disk image is malformed" errors during data loading
+
+**Symptoms:**
+- Intermittent loading failures for specific ROIs
+- Error message: "database disk image is malformed"
+- Same ROI may succeed on some loads and fail on others
+- More common with large databases (>1 GB)
+
+### Solution: Convert Databases to DELETE Mode
+
+Before mounting databases in Docker, convert them from WAL to DELETE mode:
+
+#### Using the Python Script (Recommended)
+
+```bash
+# From the ethoscopy project root
+python3 scripts/convert_wal_to_delete.py /mnt/ethoscope_data/results --dry-run
+
+# Convert all databases with detailed output
+python3 scripts/convert_wal_to_delete.py /mnt/ethoscope_data/results --verbose
+
+# Force conversion even for recently modified files (use with caution)
+python3 scripts/convert_wal_to_delete.py /mnt/ethoscope_data/results --force
+```
+
+#### Using the Bash Wrapper
+
+```bash
+# From the ethoscopy project root
+./scripts/convert_databases.sh /mnt/ethoscope_data/results
+```
+
+#### Manual Conversion Using sqlite3
+
+For individual databases:
+
+```bash
+sqlite3 /path/to/database.db "PRAGMA wal_checkpoint(TRUNCATE); PRAGMA journal_mode=DELETE;"
+```
+
+For bulk conversion:
+
+```bash
+find /mnt/ethoscope_data/results -name "*.db" -type f -exec sqlite3 {} "PRAGMA wal_checkpoint(TRUNCATE); PRAGMA journal_mode=DELETE;" \;
+```
+
+### Verification
+
+After conversion, verify the database is in DELETE mode:
+
+```bash
+sqlite3 /path/to/database.db "PRAGMA journal_mode;"
+# Should output: delete
+
+sqlite3 /path/to/database.db "PRAGMA integrity_check;"
+# Should output: ok
+```
+
+### Best Practices
+
+1. **Run conversion BEFORE starting Docker containers**
+   ```bash
+   # From the ethoscopy project root, convert databases first
+   ./scripts/convert_databases.sh /mnt/ethoscope_data/results
+
+   # Then start containers
+   cd Docker && docker compose up -d
+   ```
+
+2. **Do NOT convert databases while ethoscopes are actively writing**
+   - The conversion script skips files modified in the last 24 hours by default
+   - Use `--force` to override this safety check if needed
+
+3. **Test on a backup first**
+   ```bash
+   # Create a test copy
+   cp /path/to/database.db /tmp/test.db
+
+   # Test conversion
+   sqlite3 /tmp/test.db "PRAGMA wal_checkpoint(TRUNCATE); PRAGMA journal_mode=DELETE;"
+
+   # Verify it works
+   sqlite3 /tmp/test.db "PRAGMA integrity_check;"
+   ```
+
+4. **Consider converting at the ethoscope source** (upstream fix)
+   - Configure ethoscopes to use DELETE mode by default
+   - Prevents the need for post-processing
+
+### Troubleshooting
+
+**Q: I still get "malformed" errors after conversion**
+
+A: The ethoscopy library now includes improved connection handling that should work with both WAL and DELETE mode databases. If you still encounter issues:
+1. Verify the database was actually converted: `sqlite3 database.db "PRAGMA journal_mode;"`
+2. Check database integrity: `sqlite3 database.db "PRAGMA integrity_check;"`
+3. Ensure you're using ethoscopy version 2.0.4 or later
+
+**Q: Can I convert databases while Docker is running?**
+
+A: Yes, but you should restart the container after conversion:
+```bash
+# From the ethoscopy project root
+./scripts/convert_databases.sh /mnt/ethoscope_data/results
+cd Docker && docker compose restart ethoscope-lab
+```
+
+**Q: Will this affect data quality or analysis?**
+
+A: No. The conversion only changes how SQLite manages the database internally. All data remains identical and analysis results are unaffected.
+
+**Q: How do I check if a database is in WAL mode?**
+
+A:
+```bash
+sqlite3 database.db "PRAGMA journal_mode;"
+```
+Output will be either `wal` or `delete` (or `truncate`, `persist`, `memory`).
+
+### Emergency Quick Fix
+
+If you need to work immediately and can't run the full conversion:
+
+```bash
+# SSH to host machine (not inside container)
+# Convert just the failing database
+sqlite3 /path/to/failing/database.db "PRAGMA wal_checkpoint(TRUNCATE); PRAGMA journal_mode=DELETE;"
+
+# Restart container
+docker compose restart ethoscope-lab
+```

{ethoscopy-2.0.3 → ethoscopy-2.0.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ethoscopy
-Version: 2.0.3
+Version: 2.0.4
 Summary: "A python based toolkit to download and anlyse data from the Ethoscope hardware system."
 Author-email: Lblackhurst29 <lblackhurst29@gmail.com>
 License-File: LICENSE

{ethoscopy-2.0.3 → ethoscopy-2.0.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ethoscopy"
-version = "2.0.3"
+version = "2.0.4"
 description = "\"A python based toolkit to download and anlyse data from the Ethoscope hardware system.\""
 authors = [{name = "Lblackhurst29",email = "lblackhurst29@gmail.com"}]
 readme = "README.md"

ethoscopy-2.0.4/scripts/README.md

@@ -0,0 +1,121 @@
+# Ethoscopy Database Utility Scripts
+
+This directory contains utility scripts for managing and maintaining ethoscope SQLite databases.
+
+## Database Journal Mode Conversion
+
+### convert_wal_to_delete.py
+
+Converts SQLite databases from WAL (Write-Ahead Logging) mode to DELETE mode. This is essential when mounting databases read-only in Docker containers, as WAL mode requires write access to create companion `.db-wal` and `.db-shm` files.
+
+**Usage:**
+
+```bash
+# Dry run to preview what would be converted
+python3 scripts/convert_wal_to_delete.py /path/to/ethoscope_results --dry-run
+
+# Convert all databases with detailed output
+python3 scripts/convert_wal_to_delete.py /path/to/ethoscope_results --verbose
+
+# Force conversion even for recently modified databases (use with caution)
+python3 scripts/convert_wal_to_delete.py /path/to/ethoscope_results --force
+
+# Specify custom file pattern
+python3 scripts/convert_wal_to_delete.py /path/to/ethoscope_results --pattern "*.sqlite"
+```
+
+**Features:**
+- Recursively searches for all `.db` files
+- Checks current journal mode before conversion
+- Safety check: skips files modified within last 24 hours (unless `--force` used)
+- Performs WAL checkpoint to merge uncommitted data
+- Verifies database integrity after conversion
+- Provides detailed progress and summary statistics
+
+### convert_databases.sh
+
+Bash wrapper script that provides a simple interface to the Python conversion script.
+
+**Usage:**
+
+```bash
+# Simple one-line conversion
+./scripts/convert_databases.sh /path/to/ethoscope_results
+
+# Pass additional flags to the Python script
+./scripts/convert_databases.sh /path/to/ethoscope_results --verbose
+./scripts/convert_databases.sh /path/to/ethoscope_results --dry-run
+```
+
+## When to Use These Scripts
+
+### Problem Symptoms
+- Intermittent "database disk image is malformed" errors
+- Errors occur when loading ethoscope data in Docker with read-only mounts
+- Same ROI sometimes succeeds and sometimes fails to load
+- More common with large databases (>1 GB)
+
+### Solution Workflow
+
+1. **Before starting Docker containers:**
+   ```bash
+   # Convert databases
+   ./scripts/convert_databases.sh /mnt/ethoscope_data/results
+
+   # Start containers
+   cd Docker && docker compose up -d
+   ```
+
+2. **If databases are already mounted:**
+   ```bash
+   # Convert databases
+   ./scripts/convert_databases.sh /mnt/ethoscope_data/results
+
+   # Restart containers to use new connections
+   cd Docker && docker compose restart ethoscope-lab
+   ```
+
+### Safety Notes
+
+- **Do NOT convert databases while ethoscopes are actively writing to them**
+- The script automatically skips files modified within the last 24 hours
+- Use `--force` flag to override this safety check if needed
+- Always test on a backup or single database first
+- Conversion does not modify data, only the internal journal mode
+
+### Verification
+
+Check if a database is in WAL mode:
+```bash
+sqlite3 database.db "PRAGMA journal_mode;"
+```
+
+Verify conversion was successful:
+```bash
+sqlite3 database.db "PRAGMA journal_mode;"  # Should return: delete
+sqlite3 database.db "PRAGMA integrity_check;"  # Should return: ok
+```
+
+## Technical Background
+
+**Why this is needed:**
+
+- Ethoscope databases may be created in WAL mode for better write performance
+- WAL mode requires companion files (`.db-wal`, `.db-shm`) for proper operation
+- Docker read-only mounts (`:ro`) prevent SQLite from accessing these files
+- Without proper WAL file access, SQLite can report "database disk image is malformed"
+- Converting to DELETE mode eliminates the need for companion files
+
+**What the conversion does:**
+
+1. Executes `PRAGMA wal_checkpoint(TRUNCATE)` - Merges WAL data into main database
+2. Executes `PRAGMA journal_mode=DELETE` - Switches to DELETE mode
+3. Verifies database integrity with `PRAGMA integrity_check`
+
+The conversion is safe and does not modify any data - it only changes how SQLite manages internal transactions.
+
+## See Also
+
+- `Docker/README.md` - Complete documentation on database preparation for Docker
+- `CLAUDE.md` - Troubleshooting section for developer reference
+- `src/ethoscopy/load.py` - Implementation of improved connection handling

ethoscopy-2.0.4/scripts/convert_databases.sh

@@ -0,0 +1,51 @@
+#!/bin/bash
+# Wrapper script to convert all ethoscope databases from WAL to DELETE mode
+# Run this BEFORE docker compose up to prepare databases for read-only mounting
+#
+# Usage:
+#   ./convert_databases.sh /mnt/ethoscope_data/results
+#   ./convert_databases.sh /mnt/ethoscope_data/results --dry-run
+
+set -e
+
+# Default path if not provided
+DATA_PATH="${1:-/mnt/ethoscope_data/results}"
+
+# Check if path exists
+if [ ! -d "$DATA_PATH" ]; then
+    echo "ERROR: Directory does not exist: $DATA_PATH"
+    echo "Usage: $0 <path_to_ethoscope_results> [--dry-run]"
+    exit 1
+fi
+
+echo "Converting SQLite databases in: $DATA_PATH"
+echo "This may take a while for large datasets..."
+echo ""
+
+# Check if Python script exists
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PYTHON_SCRIPT="$SCRIPT_DIR/convert_wal_to_delete.py"
+
+if [ ! -f "$PYTHON_SCRIPT" ]; then
+    echo "ERROR: Python script not found: $PYTHON_SCRIPT"
+    exit 1
+fi
+
+# Pass all arguments to the Python script
+python3 "$PYTHON_SCRIPT" "$@"
+
+exit_code=$?
+
+if [ $exit_code -eq 0 ]; then
+    echo ""
+    echo "✓ Database conversion completed successfully"
+    echo ""
+    echo "You can now start Docker containers with:"
+    echo "  cd $(dirname "$SCRIPT_DIR") && docker compose up -d"
+else
+    echo ""
+    echo "✗ Database conversion failed with exit code $exit_code"
+    echo "Please check the errors above and try again"
+fi
+
+exit $exit_code

ethoscopy-2.0.4/scripts/convert_wal_to_delete.py

@@ -0,0 +1,240 @@
+#!/usr/bin/env python3
+"""
+Convert SQLite databases from WAL mode to DELETE mode for read-only Docker mounts.
+
+This script walks through a directory tree, finds all SQLite database files,
+and converts them from WAL (Write-Ahead Logging) mode to DELETE mode.
+This is necessary for ethoscope databases that will be mounted read-only in Docker.
+
+Usage:
+    python convert_wal_to_delete.py /path/to/ethoscope_results
+    python convert_wal_to_delete.py /path/to/ethoscope_results --dry-run
+    python convert_wal_to_delete.py /path/to/ethoscope_results --verbose
+    python convert_wal_to_delete.py /path/to/ethoscope_results --force
+"""
+
+import argparse
+import sqlite3
+import sys
+from pathlib import Path
+from datetime import datetime, timedelta
+from typing import Tuple, List
+
+
+def check_journal_mode(db_path: Path) -> str:
+    """Check the current journal mode of a SQLite database."""
+    try:
+        conn = sqlite3.connect(f'file:{db_path}?mode=ro', uri=True)
+        cursor = conn.cursor()
+        cursor.execute("PRAGMA journal_mode;")
+        mode = cursor.fetchone()[0]
+        conn.close()
+        return mode
+    except Exception as e:
+        return f"ERROR: {str(e)}"
+
+
+def convert_database(db_path: Path, force: bool = False, verbose: bool = False) -> Tuple[bool, str]:
+    """
+    Convert a database from WAL to DELETE mode.
+
+    Args:
+        db_path: Path to the database file
+        force: If True, skip safety checks for recently modified files
+        verbose: If True, print detailed progress
+
+    Returns:
+        Tuple of (success, message)
+    """
+    try:
+        # Check if file was recently modified (within last 24 hours)
+        if not force:
+            mtime = datetime.fromtimestamp(db_path.stat().st_mtime)
+            if datetime.now() - mtime < timedelta(hours=24):
+                return False, "SKIPPED: Database modified within last 24 hours (use --force to override)"
+
+        # Check current journal mode
+        current_mode = check_journal_mode(db_path)
+        if "ERROR" in current_mode:
+            return False, f"FAILED: Could not read journal mode: {current_mode}"
+
+        if current_mode.lower() == 'delete':
+            return True, "SKIPPED: Already in DELETE mode"
+
+        if verbose:
+            print(f"  Current mode: {current_mode}")
+
+        # Open database in read-write mode for conversion
+        conn = sqlite3.connect(str(db_path))
+        cursor = conn.cursor()
+
+        # Checkpoint the WAL file to merge changes back to main database
+        if verbose:
+            print(f"  Checkpointing WAL...")
+        cursor.execute("PRAGMA wal_checkpoint(TRUNCATE);")
+
+        # Convert to DELETE mode
+        if verbose:
+            print(f"  Converting to DELETE mode...")
+        cursor.execute("PRAGMA journal_mode=DELETE;")
+        new_mode = cursor.fetchone()[0]
+
+        # Verify integrity
+        if verbose:
+            print(f"  Verifying integrity...")
+        cursor.execute("PRAGMA integrity_check;")
+        integrity = cursor.fetchone()[0]
+
+        conn.close()
+
+        if integrity.lower() != 'ok':
+            return False, f"FAILED: Integrity check failed: {integrity}"
+
+        if new_mode.lower() != 'delete':
+            return False, f"FAILED: Conversion unsuccessful, mode is {new_mode}"
+
+        return True, f"SUCCESS: Converted from {current_mode} to {new_mode}"
+
+    except sqlite3.OperationalError as e:
+        return False, f"FAILED: Database locked or in use: {str(e)}"
+    except Exception as e:
+        return False, f"FAILED: {str(e)}"
+
+
+def find_databases(root_path: Path, pattern: str = "*.db") -> List[Path]:
+    """Find all database files in the directory tree."""
+    return list(root_path.rglob(pattern))
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Convert SQLite databases from WAL mode to DELETE mode",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog="""
+Examples:
+  # Dry run to see what would be converted
+  python convert_wal_to_delete.py /mnt/ethoscope_data/results --dry-run
+
+  # Convert all databases with verbose output
+  python convert_wal_to_delete.py /mnt/ethoscope_data/results --verbose
+
+  # Force conversion even for recently modified databases
+  python convert_wal_to_delete.py /mnt/ethoscope_data/results --force
+        """
+    )
+
+    parser.add_argument(
+        'root_path',
+        type=Path,
+        help='Root directory to search for database files'
+    )
+    parser.add_argument(
+        '--dry-run',
+        action='store_true',
+        help='Show what would be done without making changes'
+    )
+    parser.add_argument(
+        '--verbose',
+        action='store_true',
+        help='Print detailed progress information'
+    )
+    parser.add_argument(
+        '--force',
+        action='store_true',
+        help='Force conversion even for recently modified databases'
+    )
+    parser.add_argument(
+        '--pattern',
+        type=str,
+        default='*.db',
+        help='File pattern to match (default: *.db)'
+    )
+
+    args = parser.parse_args()
+
+    # Validate root path
+    if not args.root_path.exists():
+        print(f"ERROR: Path does not exist: {args.root_path}", file=sys.stderr)
+        sys.exit(1)
+
+    if not args.root_path.is_dir():
+        print(f"ERROR: Path is not a directory: {args.root_path}", file=sys.stderr)
+        sys.exit(1)
+
+    print(f"Searching for database files in: {args.root_path}")
+    print(f"Pattern: {args.pattern}")
+
+    if args.dry_run:
+        print("DRY RUN MODE - No changes will be made")
+
+    print()
+
+    # Find all database files
+    databases = find_databases(args.root_path, args.pattern)
+
+    if not databases:
+        print("No database files found.")
+        sys.exit(0)
+
+    print(f"Found {len(databases)} database file(s)")
+    print()
+
+    # Process each database
+    stats = {
+        'total': len(databases),
+        'converted': 0,
+        'skipped': 0,
+        'failed': 0,
+        'already_delete': 0
+    }
+
+    for i, db_path in enumerate(databases, 1):
+        rel_path = db_path.relative_to(args.root_path)
+        size_mb = db_path.stat().st_size / (1024 * 1024)
+
+        print(f"[{i}/{len(databases)}] {rel_path} ({size_mb:.1f} MB)")
+
+        if args.dry_run:
+            mode = check_journal_mode(db_path)
+            print(f"  Current mode: {mode}")
+            if mode.lower() == 'wal':
+                print(f"  Would convert from WAL to DELETE")
+            elif mode.lower() == 'delete':
+                print(f"  Already in DELETE mode")
+            stats['skipped'] += 1
+        else:
+            success, message = convert_database(db_path, args.force, args.verbose)
+            print(f"  {message}")
+
+            if success:
+                if "Already in DELETE mode" in message:
+                    stats['already_delete'] += 1
+                else:
+                    stats['converted'] += 1
+            elif "SKIPPED" in message:
+                stats['skipped'] += 1
+            else:
+                stats['failed'] += 1
+
+        print()
+
+    # Print summary
+    print("=" * 60)
+    print("SUMMARY")
+    print("=" * 60)
+    print(f"Total databases found:     {stats['total']}")
+
+    if args.dry_run:
+        print(f"Would be processed:        {stats['skipped']}")
+    else:
+        print(f"Successfully converted:    {stats['converted']}")
+        print(f"Already in DELETE mode:    {stats['already_delete']}")
+        print(f"Skipped:                   {stats['skipped']}")
+        print(f"Failed:                    {stats['failed']}")
+
+    if stats['failed'] > 0:
+        sys.exit(1)
+
+
+if __name__ == '__main__':
+    main()

{ethoscopy-2.0.3 → ethoscopy-2.0.4}/src/ethoscopy/load.py

@@ -21,8 +21,8 @@ def _connect_db(path):
 
     When the database directory is read-only (e.g., mounted with :ro in Docker),
     SQLite cannot create journal/WAL files and will fail to open the database.
-    This function detects read-only filesystems and uses immutable mode to bypass
-    this limitation.
+    This function detects read-only filesystems and uses appropriate connection
+    parameters to handle WAL-mode databases safely.
 
     Args:
         path (str): Path to the SQLite database file
@@ -31,8 +31,9 @@ def _connect_db(path):
         sqlite3.Connection: Database connection object
 
     Note:
-        When using immutable mode on an active database with WAL journaling,
-        any uncommitted WAL data will not be visible. This is acceptable for
+        For WAL-mode databases on read-only mounts, this function uses mode=ro
+        with nolock=1 to prevent "database disk image is malformed" errors.
+        Any uncommitted WAL data will not be visible, which is acceptable for
         read-only mounts where the data cannot change anyway.
     """
     path_str = str(path)
@@ -40,8 +41,30 @@ def _connect_db(path):
 
     # Check if we can write to the directory
     if not os.access(dir_path, os.W_OK):
-        # Read-only filesystem - use immutable mode to avoid journal/lock file creation
-        return sqlite3.connect(f"file:{path_str}?immutable=1", uri=True)
+        # Read-only filesystem - check if database is in WAL mode
+        try:
+            # Try to detect WAL mode by opening in read-only mode first
+            temp_conn = sqlite3.connect(f"file:{path_str}?mode=ro", uri=True)
+            cursor = temp_conn.cursor()
+            cursor.execute("PRAGMA journal_mode;")
+            journal_mode = cursor.fetchone()[0].lower()
+            temp_conn.close()
+
+            if journal_mode == 'wal':
+                # WAL mode on read-only mount: use mode=ro with nolock
+                # This prevents "database disk image is malformed" errors
+                # by avoiding operations that require WAL/SHM files
+                return sqlite3.connect(
+                    f"file:{path_str}?mode=ro&nolock=1",
+                    uri=True,
+                    timeout=10.0
+                )
+            else:
+                # Non-WAL mode: use immutable mode for better performance
+                return sqlite3.connect(f"file:{path_str}?immutable=1", uri=True)
+        except Exception:
+            # If detection fails, fall back to immutable mode
+            return sqlite3.connect(f"file:{path_str}?immutable=1", uri=True)
     else:
         # Normal read-write access
         return sqlite3.connect(path_str)
@@ -916,7 +939,41 @@ def read_single_roi_optimized(
         sql_query = "SELECT * FROM ROI_{} WHERE t >= {} {}".format(
             file["region_id"], min_time, max_time_condtion
         )
-        data = pd.read_sql_query(sql_query, conn)
+
+        # Execute query with retry logic for WAL-related errors
+        try:
+            data = pd.read_sql_query(sql_query, conn)
+        except sqlite3.DatabaseError as e:
+            # Handle "database disk image is malformed" errors
+            # This can occur with WAL-mode databases on read-only mounts
+            if "malformed" in str(e).lower() or "disk image" in str(e).lower():
+                print(f"Warning: Database error for ROI {file['region_id']}, attempting retry with fresh connection...")
+
+                # Get database path from file metadata
+                db_path = file.get("path")
+                if not db_path:
+                    print(f"Error: Cannot retry - database path not found in file metadata")
+                    raise
+
+                # Create a fresh connection just for the retry
+                retry_conn = None
+                try:
+                    retry_conn = _connect_db(db_path)
+                    data = pd.read_sql_query(sql_query, retry_conn)
+                    print(f"Success: ROI {file['region_id']} loaded on retry")
+                except Exception as retry_error:
+                    print(f"Error: Retry failed for ROI {file['region_id']}: {retry_error}")
+                    raise
+                finally:
+                    # Clean up retry connection
+                    if retry_conn:
+                        try:
+                            retry_conn.close()
+                        except Exception:
+                            pass
+            else:
+                # Re-raise other database errors
+                raise
 
         if "id" in data.columns:
             # Check if 'id' is a primary key (reuse cursor)

ethoscopy-2.0.3/Docker/.claude/settings.local.json

@@ -1,19 +0,0 @@
-{
-  "permissions": {
-    "allow": [
-      "Bash(ls:*)",
-      "Bash(git checkout:*)",
-      "Bash(docker images:*)",
-      "Read(//tmp/**)",
-      "Bash(JUPYTER_HUB_TAG=5.3.0 ETHOSCOPE_LAB_TAG=1.1 docker compose build:*)",
-      "Bash(tee:*)",
-      "Bash(docker compose:*)",
-      "Bash(find:*)",
-      "Bash(sqlite3:*)",
-      "Bash(python3:*)",
-      "Bash(docker exec:*)",
-      "Bash(docker cp:*)"
-    ],
-    "deny": []
-  }
-}

ethoscopy-2.0.3/Docker/README.md

@@ -1,85 +0,0 @@
-# Ethoscopelab docker instance
-
-Note: Most users will **not** need to recreate this image. These instructions are just provided as reference.
-The ethoscopelab docker instance lives on dockerhub at the following address: [https://hub.docker.com/r/ggilestro/ethoscope-lab](https://hub.docker.com/r/ggilestro/ethoscope-lab) and this is what regular users should download and run. Follow instructions there and on the [ethoscopy manual](https://bookstack.lab.gilest.ro/books/ethoscopy/page/getting-started).
-
-
-## Docker files that were used to create the ethoscope-lab docker instance
-
-The files in this folder can be used to recreate the image as uploaded on dockerhub.
-
-The command to use to recreate that image is `JUPYTER_HUB_TAG=5.3.0 ETHOSCOPE_LAB_TAG=1.0 docker compose build`. This creates the image with the specified tag. For Docker Hub deployment, push the image: `docker push ggilestro/ethoscope-lab:1.0`. To also create a latest tag: `docker tag ggilestro/ethoscope-lab:1.0 ggilestro/ethoscope-lab:latest && docker push ggilestro/ethoscope-lab:latest`. You can verify your local images with `docker images | grep ethoscope-lab`.
-
-After creation, the image can be run using the enclosed `docker-compose.yml` file, replacing values as fit.
-
-## Add new users to the JupyterHub
-
-To add new users to the JupyterHub instance:
-
-### 1. Modify the jupyterhub_config.py file
-
-Edit the `allowed_users` set on lines 14-19 to include your new usernames:
-
-```python
-c.Authenticator.allowed_users = {
-    'amadabhushi', 'ggilestro', 'mjoyce', 'lguo',
-    'labguest1', 'labguest2', 'labguest3', 'labguest4',
-    'labguest5', 'labguest6', 'labguest7', 'labguest8',
-    'ethoscopelab', 'newuser1', 'newuser2'  # Add your new users here
-}
-```
-
-To make a user an admin, add them to the `admin_users` set on line 22:
-
-```python
-c.Authenticator.admin_users = {'ggilestro', 'newadmin'}
-```
-
-### 2. Restart the Docker container
-
-After modifying the config file:
-
-```bash
-docker compose down
-docker compose up -d
-```
-
-**Notes:**
-- All users share the same password: `ethoscope` (line 11)
-- The system uses DummyAuthenticator for simple shared-password authentication
-- Each user gets their own home directory at `/home/{username}`
-- Home directories are created automatically when users first log in
-
-## Mounting Home Directories as Volumes
-
-To persist user data and notebooks across container restarts, you should mount user home directories as Docker volumes. This is done by modifying the `docker-compose.yml` file.
-
-### Benefits of mounting home directories:
-
-1. **Data Persistence**: User notebooks, data files, and configurations survive container restarts and updates
-2. **Backup and Recovery**: Easy to backup user data by copying the mounted directories
-3. **Performance**: Direct access to host filesystem, avoiding container storage overhead
-4. **Sharing**: Users can access their files from the host system if needed
-
-### Example volume configuration:
-
-Add volumes to your `docker-compose.yml`:
-
-```yaml
-services:
-  jupyterhub:
-    volumes:
-      - ./user_data:/home  # Maps host ./user_data to container /home
-      - ./jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.py
-```
-
-Or for individual user directories:
-
-```yaml
-volumes:
-  - ./users/ggilestro:/home/ggilestro
-  - ./users/amadabhushi:/home/amadabhushi
-  - ./users/shared:/home/shared  # Shared directory for all users
-```
-
-This ensures all user work is preserved even when containers are recreated or updated.