ethoscopy 2.2.0__tar.gz → 2.2.1__tar.gz

{ethoscopy-2.2.0 → ethoscopy-2.2.1}/.github/workflows/release.yml

@@ -3,9 +3,6 @@ name: Release to PyPI
 on:
   release:
     types: [published]
-  push:
-    tags:
-      - 'v*.*.*'
 
 jobs:
   test:
@@ -58,16 +55,7 @@ jobs:
       run: |
         twine check dist/*
 
-    - name: Publish to Test PyPI
-      if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
-      env:
-        TWINE_USERNAME: __token__
-        TWINE_PASSWORD: ${{ secrets.TEST_PYPI_API_TOKEN }}
-      run: |
-        twine upload --repository testpypi dist/*
-
     - name: Publish to PyPI
-      if: github.event_name == 'release'
       env:
         TWINE_USERNAME: __token__
         TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}

{ethoscopy-2.2.0 → ethoscopy-2.2.1}/CLAUDE.md

@@ -36,48 +36,62 @@ pip install -e ".[dev]"
 ```
 
 ### Testing
-- **No formal test suite found** - project relies on Jupyter notebooks for validation
-- Use the tutorial notebooks in `tutorial_notebook/` for testing functionality
-- Run notebooks: `jupyter notebook tutorial_notebook/`
+- **Pytest suite in `tests/`**: `python -m pytest tests/ -q` (~230 tests, seconds to run)
+- **Caveat**: `pytest.ini` declares its section as `[tool:pytest]`, which is the
+  *setup.cfg* spelling — pytest silently ignores the file, so `addopts` (coverage,
+  `--strict-markers`) and the `markers` registrations do **not** apply. Renaming the
+  section to `[pytest]` would switch on `--cov-fail-under=70` and `--strict-markers`
+  for the whole suite; check it passes before doing so.
+- Tutorial notebooks in `tutorial_notebook/` complement the suite for end-to-end checks
 
 ### Docker Environment
 - **Docker available** for JupyterHub deployment
 - Build: `JUPYTER_HUB_TAG=5.3.0 ETHOSCOPE_LAB_TAG=1.0 docker compose build`
 - Run: `docker compose up -d` (from Docker/ directory)
 
-### Troubleshooting: Database "Malformed" Errors
+### Opening databases: WAL mode and read-only mounts
 
-**Problem**: Intermittent "database disk image is malformed" errors when loading ethoscope data in Docker
+Ethoscopes record in **WAL** (Write-Ahead Logging) mode and the results tree is
+normally mounted read-only (`:ro` in Docker). That combination has broken loading
+repeatedly — as `database disk image is malformed`, and as
+`unable to open database file` on `SELECT * FROM ROI_MAP`.
 
-**Root Cause**: SQLite databases in WAL (Write-Ahead Logging) mode on read-only Docker mounts
+**How `_connect_db()` handles it** (load.py)
 
-**Solution Options**:
-1. **Convert databases to DELETE mode** (recommended for immediate fix)
-   ```bash
-   # Using the conversion script (from project root)
-   python3 scripts/convert_wal_to_delete.py /mnt/ethoscope_data/results --verbose
+ethoscopy never writes to these files, so the connection is *always* read-only —
+this also stops a load from checkpointing or truncating raw data as a side effect.
+It walks an ordered ladder of open modes, `_READ_STRATEGIES`, and **probes each
+one with a real statement** before returning it:
 
-   # Or using bash wrapper
-   ./scripts/convert_databases.sh /mnt/ethoscope_data/results
-   ```
+| rung | reads | notes |
+|---|---|---|
+| `mode=ro` | every recoverable state, incl. WAL with `-shm`, corrupt `-shm`, hot rollback journal | the only mode that sees data still in an uncheckpointed `-wal` |
+| `immutable=1` | WAL database whose sidecars are absent on a read-only mount | ignores the `-wal`; warns via `_warn_if_wal_ignored()` if it would hide committed rows |
 
-2. **Improved connection handling** (v2.0.4+)
-   - The `_connect_db()` function in `load.py` now detects WAL mode automatically
-   - Uses `mode=ro&nolock=1` URI parameters for WAL databases on read-only mounts
-   - Includes retry logic in `read_single_roi_optimized()` for resilience
+Two traps worth not rediscovering:
 
-**Key Implementation Details** (load.py):
-- Lines 18-70: `_connect_db()` - Smart connection with WAL detection and appropriate parameters
-- Lines 943-979: Retry logic in `read_single_roi_optimized()` - Handles transient errors with fresh connections
+- **Never use `mode=ro&nolock=1`.** SQLite rejects it for *every* WAL database.
+- **`sqlite3.connect()` never touches the file**, so a bad URI opens "successfully"
+  and only fails on the first query — past any `try/except` around the connect
+  call. That is why every rung is probed, and it is how the `nolock=1` bug survived
+  a fallback that looked like it covered the case.
+- `immutable=1` will also "open" a **missing** path, *creating* an empty database
+  and reporting `no such table: ROI_MAP`. `_connect_db()` checks the file exists first.
 
-**Testing After Changes**:
-```python
-import ethoscopy as etho
-metadata = etho.link_meta_index('metadata.csv', '/mnt/ethoscope_results')
-data = etho.load_ethoscope(metadata, reference_hour=9.0, FUN=etho.sleep_annotation)
-# Should load all ROIs without "malformed" errors
+`read_single_roi_optimized()` retries a failed read with `_connect_db(path,
+degraded=True)`, which skips to the last rung — a retry with the default ladder
+would just pick the same failing mode again.
+
+**Optional**: convert databases to DELETE mode to avoid the question entirely
+```bash
+python3 scripts/convert_wal_to_delete.py /mnt/ethoscope_data/results --verbose
+./scripts/convert_databases.sh /mnt/ethoscope_data/results   # bash wrapper
 ```
 
+**Tests**: `tests/test_load_wal.py` builds real SQLite files in each on-disk state
+(read-only and writable directories) rather than mocking sqlite3 — the failures
+being guarded against come from SQLite itself and only appear when a statement runs.
+
 **See Also**: `Docker/README.md` for detailed database preparation instructions
 
 ## Architecture and Code Structure

{ethoscopy-2.2.0 → ethoscopy-2.2.1}/Docker/.env.dummy.example

@@ -16,6 +16,10 @@ ALLOWED_USERS=amadabhushi,ggilestro,mjoyce,lguo,labguest1,labguest2,labguest3,la
 # Comma-separated list of admin users
 ADMIN_USERS=ggilestro
 
+# Superusers see /home as their JupyterLab root and can browse/open/run
+# notebooks across every user. Defaults to ADMIN_USERS if unset.
+SUPERUSERS=ggilestro
+
 # Docker user/group (optional, defaults to 1000:1000)
 UID=1000
 GID=1000

{ethoscopy-2.2.0 → ethoscopy-2.2.1}/Docker/.env.github.example

@@ -30,6 +30,10 @@ ALLOWED_USERS=username1,username2,username3
 # Comma-separated list of admin users (GitHub usernames)
 ADMIN_USERS=your-github-username
 
+# Superusers see /home as their JupyterLab root and can browse/open/run
+# notebooks across every user. Defaults to ADMIN_USERS if unset.
+SUPERUSERS=your-github-username
+
 # Docker user/group (optional, defaults to 1000:1000)
 UID=1000
 GID=1000

{ethoscopy-2.2.0 → ethoscopy-2.2.1}/Docker/.env.google.example

@@ -32,6 +32,10 @@ ALLOWED_USERS=user1,user2,user3
 # Comma-separated list of admin users
 ADMIN_USERS=your-email-username
 
+# Superusers see /home as their JupyterLab root and can browse/open/run
+# notebooks across every user. Defaults to ADMIN_USERS if unset.
+SUPERUSERS=your-email-username
+
 # Docker user/group (optional, defaults to 1000:1000)
 UID=1000
 GID=1000

{ethoscopy-2.2.0 → ethoscopy-2.2.1}/Docker/.env.keycloak.example

@@ -44,6 +44,10 @@ ALLOWED_USERS=amadabhushi,ggilestro,mjoyce,lguo,labguest1,labguest2,labguest3,la
 # Comma-separated list of admin users
 ADMIN_USERS=ggilestro
 
+# Superusers see /home as their JupyterLab root and can browse/open/run
+# notebooks across every user. Defaults to ADMIN_USERS if unset.
+SUPERUSERS=ggilestro
+
 # Docker user/group (optional, defaults to 1000:1000)
 UID=1000
 GID=1000

{ethoscopy-2.2.0 → ethoscopy-2.2.1}/Docker/Dockerfile

@@ -66,7 +66,7 @@ RUN ln -s /bin/tar /bin/gtar && \
 # JUPYTER INSTALLATION AND CONFIGURATION
 # ============================================================================
 
-# Install JupyterLab (version 3.6.4)
+# Install JupyterLab (latest 4.x; pip resolves the newest compatible release)
 RUN pip3 install jupyterlab --ignore-installed
 
 # To upgrade to version 4.0.2 (commented out)
@@ -75,6 +75,14 @@ RUN pip3 install jupyterlab --ignore-installed
 # Install Jupyter extensions (compatible with 3.x only)
 RUN pip3 install jupyterlab-spreadsheet-editor
 
+# AI assistant: Copilot-style inline completions + chat panel.
+# Reason: lightweight (declares only jupyter-server as its Jupyter dep), so it
+# does NOT pull in jupyter-collaboration / server-documents like jupyter-ai does
+# — that RTC stack is what caused "File ID could not be retrieved" on file open.
+# Pinned for reproducible builds. Supports OpenRouter via its OpenAI-compatible
+# provider (base URL https://openrouter.ai/api/v1).
+RUN pip3 install --no-cache-dir "notebook-intelligence==5.1.0"
+
 # ============================================================================
 # PYTHON PACKAGES AND ETHOSCOPE SETUP
 # ============================================================================
@@ -98,7 +106,7 @@ RUN pip3 install \
         pyarrow \
         tqdm \
         jupyterlab-git \
-        ethoscopy==2.1.0
+        ethoscopy==2.2.0
        # pycatch22
 
 # Pre-populate tutorial datasets inside the installed package so non-root

{ethoscopy-2.2.0 → ethoscopy-2.2.1}/Docker/README_AUTH.md

@@ -15,6 +15,7 @@ This JupyterHub deployment supports multiple authentication methods, configurabl
   - [Adding New Users (OAuth)](#adding-new-users-oauth)
   - [Adding New Users (Dummy Auth)](#adding-new-users-dummy-auth)
   - [Making Users Admins](#making-users-admins)
+  - [Granting Cross-User File Access (Superusers)](#granting-cross-user-file-access-superusers)
 - [Switching Authentication Methods](#switching-authentication-methods)
 - [Dockerfile Optimization](#dockerfile-optimization)
 - [Troubleshooting](#troubleshooting)
@@ -388,6 +389,33 @@ docker compose restart
 - Navigate to: `https://jupyter.lab.gilest.ro/hub/admin`
 - You should see the admin panel with all users listed
 
+### Granting Cross-User File Access (Superusers)
+
+By default JupyterLab roots each user's file browser at their own home (`/home/<username>`). A **superuser** instead gets `/home` as their root, so they can browse, open, and run notebooks across every user's directory directly from their own JupyterLab session.
+
+This works because all single-user servers in this deployment run under the same container UID (see `ConfigUserSpawner` in `config/jupyterhub_config.py`), so filesystem-level access to all homes already exists — `SUPERUSERS` simply surfaces it in the UI.
+
+**Configuration**:
+
+```bash
+# .env
+SUPERUSERS=ggilestro,jdoe
+```
+
+If `SUPERUSERS` is unset, it defaults to `ADMIN_USERS`.
+
+**Apply changes**:
+```bash
+docker compose restart
+```
+
+**Verify**:
+- Log in as a superuser
+- The JupyterLab file browser should list every user's home folder
+- Opening another user's `.ipynb` runs it in place; edits and kernel output write back to that user's home
+
+**Security note**: Regular users are not isolated from each other at the filesystem level in this setup — anyone with a JupyterLab terminal can already `cd` into another user's home. `SUPERUSERS` only changes what is shown by default in the file browser. True per-user isolation would require a different spawner (e.g. `DockerSpawner` or `KubeSpawner` with distinct UIDs).
+
 ---
 
 ## Dockerfile Optimization

{ethoscopy-2.2.0 → ethoscopy-2.2.1}/Docker/config/jupyterhub_config.py

@@ -120,6 +120,17 @@ else:
     # Use imported list from users.py
     c.Authenticator.admin_users = ADMIN_USERS
 
+# Superusers: users whose JupyterLab file browser is rooted at /home instead
+# of their own home, so they can browse/open/run notebooks across every user.
+# Reason: all single-user servers run as the same container UID (see
+# ConfigUserSpawner below), so filesystem access already exists — this just
+# exposes it in the UI. Defaults to admins; override with SUPERUSERS env var.
+superusers_env = os.getenv('SUPERUSERS')
+if superusers_env is not None:
+    SUPERUSERS = {u.strip() for u in superusers_env.split(',') if u.strip()}
+else:
+    SUPERUSERS = set(c.Authenticator.admin_users)
+
 # ============================================================================
 # CUSTOM SPAWNER (Preserved - critical for current setup)
 # ============================================================================
@@ -129,25 +140,48 @@ class ConfigUserSpawner(LocalProcessSpawner):
     def make_preexec_fn(self, name):
         """Don't try to switch users - run everything as current user"""
         return None
-    
+
     def user_env(self, env):
         """Set user environment without system user lookup"""
         env = env.copy()
         home_dir = f'/home/{self.user.name}'
-        
+
         # Ensure home directory exists with proper permissions
         os.makedirs(home_dir, mode=0o755, exist_ok=True)
-        
+
         # Set environment variables
         env['USER'] = self.user.name
         env['HOME'] = home_dir
         env['SHELL'] = '/bin/bash'
         env['LOGNAME'] = self.user.name
-        
+
+        # Notebook Intelligence (AI assistant): forward central LLM config from
+        # the container environment so students never enter API keys themselves
+        # (avoids keys leaking into dotfiles, cf. the pxin OpenAI-key incident).
+        # LiteLLM reads OPENROUTER_API_KEY directly and routes the
+        # `openrouter/...` model id; values come from .env via docker-compose.
+        for _key in (
+            'OPENROUTER_API_KEY',
+            'NBI_CHAT_MODEL_PROVIDER', 'NBI_CHAT_MODEL_ID',
+            'NBI_INLINE_COMPLETION_MODEL_PROVIDER', 'NBI_INLINE_COMPLETION_MODEL_ID',
+        ):
+            _val = os.getenv(_key)
+            if _val:
+                env[_key] = _val
+
         return env
 
 c.JupyterHub.spawner_class = ConfigUserSpawner
-c.Spawner.notebook_dir = '/home/{username}'
+
+
+def _set_notebook_dir(spawner):
+    if spawner.user.name in SUPERUSERS:
+        spawner.notebook_dir = '/home'
+    else:
+        spawner.notebook_dir = f'/home/{spawner.user.name}'
+
+
+c.Spawner.pre_spawn_hook = _set_notebook_dir
 
 # Timeouts
 c.Spawner.http_timeout = 60

{ethoscopy-2.2.0 → ethoscopy-2.2.1}/Docker/docker-compose.yml

@@ -6,13 +6,14 @@ services:
       dockerfile: Dockerfile
       args:
         JUPYTER_HUB_TAG: ${JUPYTER_HUB_TAG:-5.4.0}
-    image: ggilestro/ethoscope-lab:${ETHOSCOPE_LAB_TAG:-1.2}
+    image: ggilestro/ethoscope-lab:${ETHOSCOPE_LAB_TAG:-1.3}
     container_name: ethoscope-lab
     ports:
       - 8082:8000
     volumes:
       - /mnt/ethoscope_data/results:/mnt/ethoscope_results:ro
       - /mnt/ethoscope_data/ethoscope_metadata:/opt/ethoscope_metadata
+      - /mnt/data/projects:/mnt/data/projects:ro
       - /mnt/homes:/home
       - /mnt/cache:/home/cache
       - ./jupyterhub_data:/srv/jupyterhub/
@@ -34,6 +35,7 @@ services:
       - DUMMY_PASSWORD=${DUMMY_PASSWORD:-ethoscope}
       - ALLOWED_USERS=${ALLOWED_USERS}
       - ADMIN_USERS=${ADMIN_USERS}
+      - SUPERUSERS=${SUPERUSERS}
       # OAuth settings (only needed when using OAuth authenticators)
       - OAUTH_CLIENT_ID=${OAUTH_CLIENT_ID}
       - OAUTH_CLIENT_SECRET=${OAUTH_CLIENT_SECRET}
@@ -47,6 +49,14 @@ services:
       - GITHUB_ORG=${GITHUB_ORG}
       - GOOGLE_HOSTED_DOMAIN=${GOOGLE_HOSTED_DOMAIN}
       - GITLAB_URL=${GITLAB_URL}
+      # Notebook Intelligence (AI assistant) — central OpenRouter config.
+      # Put OPENROUTER_API_KEY in .env (gitignored). Models are overridable;
+      # verify/adjust slugs at https://openrouter.ai/models
+      - OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
+      - NBI_CHAT_MODEL_PROVIDER=${NBI_CHAT_MODEL_PROVIDER:-litellm-compatible}
+      - NBI_CHAT_MODEL_ID=${NBI_CHAT_MODEL_ID:-openrouter/anthropic/claude-sonnet-4.5}
+      - NBI_INLINE_COMPLETION_MODEL_PROVIDER=${NBI_INLINE_COMPLETION_MODEL_PROVIDER:-litellm-compatible}
+      - NBI_INLINE_COMPLETION_MODEL_ID=${NBI_INLINE_COMPLETION_MODEL_ID:-openrouter/openai/gpt-4o-mini}
 
     user: "${UID:-1000}:${GID:-1000}" # Adjust to your own user. Make sure this matches the permissions of your /mnt/{homes,cache} folder
     restart: always

{ethoscopy-2.2.0 → ethoscopy-2.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: ethoscopy
-Version: 2.2.0
+Version: 2.2.1
 Summary: "A python based toolkit to download and anlyse data from the Ethoscope hardware system."
 Author-email: Lblackhurst29 <lblackhurst29@gmail.com>
 License-File: LICENSE

{ethoscopy-2.2.0 → ethoscopy-2.2.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ethoscopy"
-version = "2.2.0"
+version = "2.2.1"
 description = "\"A python based toolkit to download and anlyse data from the Ethoscope hardware system.\""
 authors = [{name = "Lblackhurst29",email = "lblackhurst29@gmail.com"}]
 readme = "README.md"

{ethoscopy-2.2.0 → ethoscopy-2.2.1}/src/ethoscopy/load.py

@@ -3,6 +3,7 @@ import ftplib
 import os
 import sqlite3
 import time
+import warnings
 
 # Reason: newer ethoscope firmwares serialize the "selected_options" METADATA
 # field with an ``OrderedDict([...])`` wrapper. ``get_meta`` below round-trips
@@ -22,57 +23,119 @@ from ethoscopy.misc.validate_datetime import validate_datetime
 pd.options.mode.chained_assignment = None
 
 
-def _connect_db(path):
+# Ordered ladder of read-only SQLite open modes, most faithful first.
+#
+# Reason: whether an ethoscope database can be opened is not predictable from
+# its journal mode alone. Journal mode, the presence and state of the -wal/-shm
+# sidecars and the writability of the containing directory interact, and the
+# combinations are not rare in practice -- ethoscopes write in WAL mode and the
+# results tree is usually mounted read-only. Empirically (sqlite 3.45 and 3.53,
+# read-only and writable directories):
+#
+#   * mode=ro reads every recoverable state, including a WAL database whose
+#     -shm exists, a stale or corrupt -shm, and a hot rollback journal, and it
+#     is the only mode that sees data still sitting in an uncheckpointed -wal.
+#   * immutable=1 is the sole survivor of one real case: a WAL-mode database
+#     whose sidecars are absent on a read-only mount (SQLite would have to
+#     create the -shm to read it). It ignores the -wal entirely, so it is the
+#     fallback, never the first choice -- see _warn_if_wal_ignored.
+#
+# Note nolock=1 is deliberately absent: SQLite rejects it for *every* WAL
+# database, and because sqlite3.connect() never touches the file the rejection
+# only surfaces on the first query. That is why each rung below is probed.
+_READ_STRATEGIES = ("mode=ro", "immutable=1")
+
+# Errors that mean "this open mode cannot read this file" rather than "this
+# query is wrong" -- worth dropping to the next rung of the ladder for.
+_UNREADABLE_ERRORS = ("unable to open database file", "readonly database", "malformed")
+
+
+def _warn_if_wal_ignored(path_str):
     """
-    Connect to a SQLite database with smart read-only filesystem detection.
+    Warn when falling back to immutable=1 would hide committed data.
 
-    When the database directory is read-only (e.g., mounted with :ro in Docker),
-    SQLite cannot create journal/WAL files and will fail to open the database.
-    This function detects read-only filesystems and uses appropriate connection
-    parameters to handle WAL-mode databases safely.
+    immutable=1 reads the main database file only. If an uncheckpointed -wal
+    sidecar still holds committed transactions, those rows silently disappear
+    from the loaded data -- the worst possible failure for an analysis library,
+    so make it loud.
+
+    Args:
+        path_str (str): Path to the SQLite database file
+    """
+    try:
+        wal_size = os.path.getsize(f"{path_str}-wal")
+    except OSError:
+        return
+
+    if wal_size > 0:
+        warnings.warn(
+            f"{path_str} could only be opened in SQLite's immutable mode, which "
+            f"ignores its {wal_size} byte -wal sidecar: data committed to the WAL "
+            "but not yet checkpointed will be missing. Checkpoint the database "
+            "(scripts/convert_wal_to_delete.py) or make its directory writable.",
+            RuntimeWarning,
+            stacklevel=3,
+        )
+
+
+def _connect_db(path, degraded=False):
+    """
+    Open an ethoscope database read-only, tolerating WAL state and read-only mounts.
+
+    ethoscopy never writes to these files, so the connection is always read-only:
+    that also stops a load from checkpointing or truncating the raw data as a side
+    effect. Each candidate open mode is probed with a real statement before being
+    handed back, because sqlite3.connect() does not touch the file and an
+    unusable mode would otherwise only fail deep inside the first data query.
 
     Args:
         path (str): Path to the SQLite database file
+        degraded (bool, optional): Skip the faithful open modes and go straight to
+            the last-resort one. Used to escalate after a connection that opened
+            cleanly fails part-way through a read. Default is False.
 
     Returns:
-        sqlite3.Connection: Database connection object
+        sqlite3.Connection: A connection that has answered at least one statement
 
-    Note:
-        For WAL-mode databases on read-only mounts, this function uses mode=ro
-        with nolock=1 to prevent "database disk image is malformed" errors.
-        Any uncommitted WAL data will not be visible, which is acceptable for
-        read-only mounts where the data cannot change anyway.
+    Raises:
+        FileNotFoundError: If path does not exist
+        sqlite3.OperationalError: If no open mode can read the file
     """
     path_str = str(path)
-    dir_path = os.path.dirname(path_str)
 
-    # Check if we can write to the directory
-    if not os.access(dir_path, os.W_OK):
-        # Read-only filesystem - check if database is in WAL mode
+    # Reason: immutable=1 happily "opens" a path that is not there, creating an
+    # empty database file and leaving the caller with a baffling
+    # "no such table: ROI_MAP". Fail on the real problem instead.
+    if not os.path.isfile(path_str):
+        raise FileNotFoundError(
+            errno.ENOENT, "No such ethoscope database file", path_str
+        )
+
+    strategies = _READ_STRATEGIES[-1:] if degraded else _READ_STRATEGIES
+
+    last_error = None
+    for strategy in strategies:
+        conn = None
         try:
-            # Try to detect WAL mode by opening in read-only mode first
-            temp_conn = sqlite3.connect(f"file:{path_str}?mode=ro", uri=True)
-            cursor = temp_conn.cursor()
-            cursor.execute("PRAGMA journal_mode;")
-            journal_mode = cursor.fetchone()[0].lower()
-            temp_conn.close()
-
-            if journal_mode == "wal":
-                # WAL mode on read-only mount: use mode=ro with nolock
-                # This prevents "database disk image is malformed" errors
-                # by avoiding operations that require WAL/SHM files
-                return sqlite3.connect(
-                    f"file:{path_str}?mode=ro&nolock=1", uri=True, timeout=10.0
-                )
-            else:
-                # Non-WAL mode: use immutable mode for better performance
-                return sqlite3.connect(f"file:{path_str}?immutable=1", uri=True)
-        except Exception:
-            # If detection fails, fall back to immutable mode
-            return sqlite3.connect(f"file:{path_str}?immutable=1", uri=True)
-    else:
-        # Normal read-write access
-        return sqlite3.connect(path_str)
+            conn = sqlite3.connect(
+                f"file:{path_str}?{strategy}", uri=True, timeout=10.0
+            )
+            # Probe: forces SQLite to actually reach the file and its sidecars
+            conn.execute("PRAGMA journal_mode;").fetchone()
+        except sqlite3.Error as e:
+            last_error = e
+            if conn is not None:
+                conn.close()
+            continue
+
+        if strategy == "immutable=1":
+            _warn_if_wal_ignored(path_str)
+        return conn
+
+    raise sqlite3.OperationalError(
+        f"Could not open {path_str} for reading with any of {list(strategies)}. "
+        f"Last error: {last_error}"
+    )
 
 
 def download_from_remote_dir(meta, remote_dir, local_dir, progress=True):
@@ -512,6 +575,7 @@ def load_ethoscope(
     try:
         for db_path, group in grouped_metadata:
             conn = None
+            pbar_at_db_start = pbar.n
 
             try:
                 # Open connection once per database file
@@ -603,6 +667,19 @@ def load_ethoscope(
                     finally:
                         pbar.update(1)
 
+            except Exception as e:
+                # Reason: opening the database or reading its shared tables sits
+                # outside the per-ROI handler below, so without this one
+                # unreadable file aborts the whole load and discards every ROI
+                # already read from the other databases. Report it and move on.
+                if verbose is True:
+                    tqdm.write(
+                        "Skipping {} - none of its {} ROIs could be loaded: {}".format(
+                            db_path, len(group), e
+                        )
+                    )
+                pbar.update(len(group) - (pbar.n - pbar_at_db_start))
+
             finally:
                 # Close connection when done with this database
                 if conn:
@@ -973,46 +1050,48 @@ def read_single_roi_optimized(
             file["region_id"], min_time, max_time_condtion
         )
 
-        # Execute query with retry logic for WAL-related errors
+        # Execute query, escalating to a degraded open mode if the shared
+        # connection turns out to be unable to read this database after all.
+        # Reason: the connection was probed at open time, but the -wal/-shm
+        # sidecars can change underneath a long read on a live mount, so the
+        # failure can still land here. Retrying with _connect_db() alone would
+        # just pick the same open mode again -- degraded=True is what makes the
+        # retry a different attempt rather than a repeat of the failed one.
         try:
             data = pd.read_sql_query(sql_query, conn)
         except sqlite3.DatabaseError as e:
-            # Handle "database disk image is malformed" errors
-            # This can occur with WAL-mode databases on read-only mounts
-            if "malformed" in str(e).lower() or "disk image" in str(e).lower():
+            if not any(marker in str(e).lower() for marker in _UNREADABLE_ERRORS):
+                # A genuine query error - retrying will not help
+                raise
+
+            tqdm.write(
+                f"Warning: Database error for ROI {file['region_id']} ({e}), "
+                "retrying with a fresh read-only connection..."
+            )
+
+            db_path = file.get("path")
+            if not db_path:
                 tqdm.write(
-                    f"Warning: Database error for ROI {file['region_id']}, attempting retry with fresh connection..."
+                    "Error: Cannot retry - database path not found in file metadata"
                 )
+                raise
 
-                # Get database path from file metadata
-                db_path = file.get("path")
-                if not db_path:
-                    tqdm.write(
-                        "Error: Cannot retry - database path not found in file metadata"
-                    )
-                    raise
-
-                # Create a fresh connection just for the retry
-                retry_conn = None
-                try:
-                    retry_conn = _connect_db(db_path)
-                    data = pd.read_sql_query(sql_query, retry_conn)
-                    tqdm.write(f"Success: ROI {file['region_id']} loaded on retry")
-                except Exception as retry_error:
-                    tqdm.write(
-                        f"Error: Retry failed for ROI {file['region_id']}: {retry_error}"
-                    )
-                    raise
-                finally:
-                    # Clean up retry connection
-                    if retry_conn:
-                        try:
-                            retry_conn.close()
-                        except Exception:
-                            pass
-            else:
-                # Re-raise other database errors
+            retry_conn = None
+            try:
+                retry_conn = _connect_db(db_path, degraded=True)
+                data = pd.read_sql_query(sql_query, retry_conn)
+                tqdm.write(f"Success: ROI {file['region_id']} loaded on retry")
+            except Exception as retry_error:
+                tqdm.write(
+                    f"Error: Retry failed for ROI {file['region_id']}: {retry_error}"
+                )
                 raise
+            finally:
+                if retry_conn:
+                    try:
+                        retry_conn.close()
+                    except Exception:
+                        pass
 
         if "id" in data.columns:
             # Check if 'id' is a primary key (reuse cursor)