ethoscopy 2.2.0__tar.gz → 2.3.0__tar.gz

{ethoscopy-2.2.0 → ethoscopy-2.3.0}/.github/workflows/release.yml

@@ -3,9 +3,6 @@ name: Release to PyPI
 on:
   release:
     types: [published]
-  push:
-    tags:
-      - 'v*.*.*'
 
 jobs:
   test:
@@ -58,16 +55,7 @@ jobs:
       run: |
         twine check dist/*
 
-    - name: Publish to Test PyPI
-      if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
-      env:
-        TWINE_USERNAME: __token__
-        TWINE_PASSWORD: ${{ secrets.TEST_PYPI_API_TOKEN }}
-      run: |
-        twine upload --repository testpypi dist/*
-
     - name: Publish to PyPI
-      if: github.event_name == 'release'
       env:
         TWINE_USERNAME: __token__
         TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}

ethoscopy-2.3.0/.gitignore

@@ -0,0 +1,24 @@
+*.txt
+*.pyc
+*venv
+dist/
+*vscode
+__pycache__/
+ethoscopy.egg-info/
+.gitignore
+*.pkl
+.python-version
+tmp
+
+# JupyterHub runtime state, generated when the Docker stack first runs.
+# jupyterhub_cookie_secret signs session cookies and jupyterhub.sqlite holds
+# user records and API tokens; neither belongs in the repo. Committing the
+# secret would be the same failure as the tracked .env.keycloak removed in
+# 6cbcbbe. jupyterhub_config.py stays tracked - it is configuration, not state.
+Docker/jupyterhub_data/jupyterhub_cookie_secret
+Docker/jupyterhub_data/jupyterhub.sqlite
+
+# Coverage output
+.coverage
+htmlcov/
+coverage.xml

{ethoscopy-2.2.0 → ethoscopy-2.3.0}/CLAUDE.md

@@ -36,48 +36,62 @@ pip install -e ".[dev]"
 ```
 
 ### Testing
-- **No formal test suite found** - project relies on Jupyter notebooks for validation
-- Use the tutorial notebooks in `tutorial_notebook/` for testing functionality
-- Run notebooks: `jupyter notebook tutorial_notebook/`
+- **Pytest suite in `tests/`**: `python -m pytest tests/ -q` (~230 tests, seconds to run)
+- **Caveat**: `pytest.ini` declares its section as `[tool:pytest]`, which is the
+  *setup.cfg* spelling — pytest silently ignores the file, so `addopts` (coverage,
+  `--strict-markers`) and the `markers` registrations do **not** apply. Renaming the
+  section to `[pytest]` would switch on `--cov-fail-under=70` and `--strict-markers`
+  for the whole suite; check it passes before doing so.
+- Tutorial notebooks in `tutorial_notebook/` complement the suite for end-to-end checks
 
 ### Docker Environment
 - **Docker available** for JupyterHub deployment
 - Build: `JUPYTER_HUB_TAG=5.3.0 ETHOSCOPE_LAB_TAG=1.0 docker compose build`
 - Run: `docker compose up -d` (from Docker/ directory)
 
-### Troubleshooting: Database "Malformed" Errors
+### Opening databases: WAL mode and read-only mounts
 
-**Problem**: Intermittent "database disk image is malformed" errors when loading ethoscope data in Docker
+Ethoscopes record in **WAL** (Write-Ahead Logging) mode and the results tree is
+normally mounted read-only (`:ro` in Docker). That combination has broken loading
+repeatedly — as `database disk image is malformed`, and as
+`unable to open database file` on `SELECT * FROM ROI_MAP`.
 
-**Root Cause**: SQLite databases in WAL (Write-Ahead Logging) mode on read-only Docker mounts
+**How `_connect_db()` handles it** (load.py)
 
-**Solution Options**:
-1. **Convert databases to DELETE mode** (recommended for immediate fix)
-   ```bash
-   # Using the conversion script (from project root)
-   python3 scripts/convert_wal_to_delete.py /mnt/ethoscope_data/results --verbose
+ethoscopy never writes to these files, so the connection is *always* read-only —
+this also stops a load from checkpointing or truncating raw data as a side effect.
+It walks an ordered ladder of open modes, `_READ_STRATEGIES`, and **probes each
+one with a real statement** before returning it:
 
-   # Or using bash wrapper
-   ./scripts/convert_databases.sh /mnt/ethoscope_data/results
-   ```
+| rung | reads | notes |
+|---|---|---|
+| `mode=ro` | every recoverable state, incl. WAL with `-shm`, corrupt `-shm`, hot rollback journal | the only mode that sees data still in an uncheckpointed `-wal` |
+| `immutable=1` | WAL database whose sidecars are absent on a read-only mount | ignores the `-wal`; warns via `_warn_if_wal_ignored()` if it would hide committed rows |
 
-2. **Improved connection handling** (v2.0.4+)
-   - The `_connect_db()` function in `load.py` now detects WAL mode automatically
-   - Uses `mode=ro&nolock=1` URI parameters for WAL databases on read-only mounts
-   - Includes retry logic in `read_single_roi_optimized()` for resilience
+Two traps worth not rediscovering:
 
-**Key Implementation Details** (load.py):
-- Lines 18-70: `_connect_db()` - Smart connection with WAL detection and appropriate parameters
-- Lines 943-979: Retry logic in `read_single_roi_optimized()` - Handles transient errors with fresh connections
+- **Never use `mode=ro&nolock=1`.** SQLite rejects it for *every* WAL database.
+- **`sqlite3.connect()` never touches the file**, so a bad URI opens "successfully"
+  and only fails on the first query — past any `try/except` around the connect
+  call. That is why every rung is probed, and it is how the `nolock=1` bug survived
+  a fallback that looked like it covered the case.
+- `immutable=1` will also "open" a **missing** path, *creating* an empty database
+  and reporting `no such table: ROI_MAP`. `_connect_db()` checks the file exists first.
 
-**Testing After Changes**:
-```python
-import ethoscopy as etho
-metadata = etho.link_meta_index('metadata.csv', '/mnt/ethoscope_results')
-data = etho.load_ethoscope(metadata, reference_hour=9.0, FUN=etho.sleep_annotation)
-# Should load all ROIs without "malformed" errors
+`read_single_roi_optimized()` retries a failed read with `_connect_db(path,
+degraded=True)`, which skips to the last rung — a retry with the default ladder
+would just pick the same failing mode again.
+
+**Optional**: convert databases to DELETE mode to avoid the question entirely
+```bash
+python3 scripts/convert_wal_to_delete.py /mnt/ethoscope_data/results --verbose
+./scripts/convert_databases.sh /mnt/ethoscope_data/results   # bash wrapper
 ```
 
+**Tests**: `tests/test_load_wal.py` builds real SQLite files in each on-disk state
+(read-only and writable directories) rather than mocking sqlite3 — the failures
+being guarded against come from SQLite itself and only appear when a statement runs.
+
 **See Also**: `Docker/README.md` for detailed database preparation instructions
 
 ## Architecture and Code Structure

{ethoscopy-2.2.0 → ethoscopy-2.3.0}/Docker/.env.dummy.example

@@ -16,6 +16,10 @@ ALLOWED_USERS=amadabhushi,ggilestro,mjoyce,lguo,labguest1,labguest2,labguest3,la
 # Comma-separated list of admin users
 ADMIN_USERS=ggilestro
 
+# Superusers see /home as their JupyterLab root and can browse/open/run
+# notebooks across every user. Defaults to ADMIN_USERS if unset.
+SUPERUSERS=ggilestro
+
 # Docker user/group (optional, defaults to 1000:1000)
 UID=1000
 GID=1000

{ethoscopy-2.2.0 → ethoscopy-2.3.0}/Docker/.env.github.example

@@ -30,6 +30,10 @@ ALLOWED_USERS=username1,username2,username3
 # Comma-separated list of admin users (GitHub usernames)
 ADMIN_USERS=your-github-username
 
+# Superusers see /home as their JupyterLab root and can browse/open/run
+# notebooks across every user. Defaults to ADMIN_USERS if unset.
+SUPERUSERS=your-github-username
+
 # Docker user/group (optional, defaults to 1000:1000)
 UID=1000
 GID=1000

{ethoscopy-2.2.0 → ethoscopy-2.3.0}/Docker/.env.google.example

@@ -32,6 +32,10 @@ ALLOWED_USERS=user1,user2,user3
 # Comma-separated list of admin users
 ADMIN_USERS=your-email-username
 
+# Superusers see /home as their JupyterLab root and can browse/open/run
+# notebooks across every user. Defaults to ADMIN_USERS if unset.
+SUPERUSERS=your-email-username
+
 # Docker user/group (optional, defaults to 1000:1000)
 UID=1000
 GID=1000

{ethoscopy-2.2.0 → ethoscopy-2.3.0}/Docker/.env.keycloak.example

@@ -44,6 +44,10 @@ ALLOWED_USERS=amadabhushi,ggilestro,mjoyce,lguo,labguest1,labguest2,labguest3,la
 # Comma-separated list of admin users
 ADMIN_USERS=ggilestro
 
+# Superusers see /home as their JupyterLab root and can browse/open/run
+# notebooks across every user. Defaults to ADMIN_USERS if unset.
+SUPERUSERS=ggilestro
+
 # Docker user/group (optional, defaults to 1000:1000)
 UID=1000
 GID=1000

{ethoscopy-2.2.0 → ethoscopy-2.3.0}/Docker/Dockerfile

@@ -66,7 +66,7 @@ RUN ln -s /bin/tar /bin/gtar && \
 # JUPYTER INSTALLATION AND CONFIGURATION
 # ============================================================================
 
-# Install JupyterLab (version 3.6.4)
+# Install JupyterLab (latest 4.x; pip resolves the newest compatible release)
 RUN pip3 install jupyterlab --ignore-installed
 
 # To upgrade to version 4.0.2 (commented out)
@@ -75,6 +75,14 @@ RUN pip3 install jupyterlab --ignore-installed
 # Install Jupyter extensions (compatible with 3.x only)
 RUN pip3 install jupyterlab-spreadsheet-editor
 
+# AI assistant: Copilot-style inline completions + chat panel.
+# Reason: lightweight (declares only jupyter-server as its Jupyter dep), so it
+# does NOT pull in jupyter-collaboration / server-documents like jupyter-ai does
+# — that RTC stack is what caused "File ID could not be retrieved" on file open.
+# Pinned for reproducible builds. Supports OpenRouter via its OpenAI-compatible
+# provider (base URL https://openrouter.ai/api/v1).
+RUN pip3 install --no-cache-dir "notebook-intelligence==5.1.0"
+
 # ============================================================================
 # PYTHON PACKAGES AND ETHOSCOPE SETUP
 # ============================================================================
@@ -98,7 +106,7 @@ RUN pip3 install \
         pyarrow \
         tqdm \
         jupyterlab-git \
-        ethoscopy==2.1.0
+        ethoscopy==2.2.1
        # pycatch22
 
 # Pre-populate tutorial datasets inside the installed package so non-root

{ethoscopy-2.2.0 → ethoscopy-2.3.0}/Docker/README_AUTH.md

@@ -15,6 +15,7 @@ This JupyterHub deployment supports multiple authentication methods, configurabl
   - [Adding New Users (OAuth)](#adding-new-users-oauth)
   - [Adding New Users (Dummy Auth)](#adding-new-users-dummy-auth)
   - [Making Users Admins](#making-users-admins)
+  - [Granting Cross-User File Access (Superusers)](#granting-cross-user-file-access-superusers)
 - [Switching Authentication Methods](#switching-authentication-methods)
 - [Dockerfile Optimization](#dockerfile-optimization)
 - [Troubleshooting](#troubleshooting)
@@ -388,6 +389,33 @@ docker compose restart
 - Navigate to: `https://jupyter.lab.gilest.ro/hub/admin`
 - You should see the admin panel with all users listed
 
+### Granting Cross-User File Access (Superusers)
+
+By default JupyterLab roots each user's file browser at their own home (`/home/<username>`). A **superuser** instead gets `/home` as their root, so they can browse, open, and run notebooks across every user's directory directly from their own JupyterLab session.
+
+This works because all single-user servers in this deployment run under the same container UID (see `ConfigUserSpawner` in `config/jupyterhub_config.py`), so filesystem-level access to all homes already exists — `SUPERUSERS` simply surfaces it in the UI.
+
+**Configuration**:
+
+```bash
+# .env
+SUPERUSERS=ggilestro,jdoe
+```
+
+If `SUPERUSERS` is unset, it defaults to `ADMIN_USERS`.
+
+**Apply changes**:
+```bash
+docker compose restart
+```
+
+**Verify**:
+- Log in as a superuser
+- The JupyterLab file browser should list every user's home folder
+- Opening another user's `.ipynb` runs it in place; edits and kernel output write back to that user's home
+
+**Security note**: Regular users are not isolated from each other at the filesystem level in this setup — anyone with a JupyterLab terminal can already `cd` into another user's home. `SUPERUSERS` only changes what is shown by default in the file browser. True per-user isolation would require a different spawner (e.g. `DockerSpawner` or `KubeSpawner` with distinct UIDs).
+
 ---
 
 ## Dockerfile Optimization

{ethoscopy-2.2.0 → ethoscopy-2.3.0}/Docker/config/jupyterhub_config.py

@@ -120,6 +120,17 @@ else:
     # Use imported list from users.py
     c.Authenticator.admin_users = ADMIN_USERS
 
+# Superusers: users whose JupyterLab file browser is rooted at /home instead
+# of their own home, so they can browse/open/run notebooks across every user.
+# Reason: all single-user servers run as the same container UID (see
+# ConfigUserSpawner below), so filesystem access already exists — this just
+# exposes it in the UI. Defaults to admins; override with SUPERUSERS env var.
+superusers_env = os.getenv('SUPERUSERS')
+if superusers_env is not None:
+    SUPERUSERS = {u.strip() for u in superusers_env.split(',') if u.strip()}
+else:
+    SUPERUSERS = set(c.Authenticator.admin_users)
+
 # ============================================================================
 # CUSTOM SPAWNER (Preserved - critical for current setup)
 # ============================================================================
@@ -129,25 +140,48 @@ class ConfigUserSpawner(LocalProcessSpawner):
     def make_preexec_fn(self, name):
         """Don't try to switch users - run everything as current user"""
         return None
-    
+
     def user_env(self, env):
         """Set user environment without system user lookup"""
         env = env.copy()
         home_dir = f'/home/{self.user.name}'
-        
+
         # Ensure home directory exists with proper permissions
         os.makedirs(home_dir, mode=0o755, exist_ok=True)
-        
+
         # Set environment variables
         env['USER'] = self.user.name
         env['HOME'] = home_dir
         env['SHELL'] = '/bin/bash'
         env['LOGNAME'] = self.user.name
-        
+
+        # Notebook Intelligence (AI assistant): forward central LLM config from
+        # the container environment so students never enter API keys themselves
+        # (avoids keys leaking into dotfiles, cf. the pxin OpenAI-key incident).
+        # LiteLLM reads OPENROUTER_API_KEY directly and routes the
+        # `openrouter/...` model id; values come from .env via docker-compose.
+        for _key in (
+            'OPENROUTER_API_KEY',
+            'NBI_CHAT_MODEL_PROVIDER', 'NBI_CHAT_MODEL_ID',
+            'NBI_INLINE_COMPLETION_MODEL_PROVIDER', 'NBI_INLINE_COMPLETION_MODEL_ID',
+        ):
+            _val = os.getenv(_key)
+            if _val:
+                env[_key] = _val
+
         return env
 
 c.JupyterHub.spawner_class = ConfigUserSpawner
-c.Spawner.notebook_dir = '/home/{username}'
+
+
+def _set_notebook_dir(spawner):
+    if spawner.user.name in SUPERUSERS:
+        spawner.notebook_dir = '/home'
+    else:
+        spawner.notebook_dir = f'/home/{spawner.user.name}'
+
+
+c.Spawner.pre_spawn_hook = _set_notebook_dir
 
 # Timeouts
 c.Spawner.http_timeout = 60

{ethoscopy-2.2.0 → ethoscopy-2.3.0}/Docker/docker-compose.yml

@@ -6,13 +6,14 @@ services:
       dockerfile: Dockerfile
       args:
         JUPYTER_HUB_TAG: ${JUPYTER_HUB_TAG:-5.4.0}
-    image: ggilestro/ethoscope-lab:${ETHOSCOPE_LAB_TAG:-1.2}
+    image: ggilestro/ethoscope-lab:${ETHOSCOPE_LAB_TAG:-1.4}
     container_name: ethoscope-lab
     ports:
       - 8082:8000
     volumes:
       - /mnt/ethoscope_data/results:/mnt/ethoscope_results:ro
       - /mnt/ethoscope_data/ethoscope_metadata:/opt/ethoscope_metadata
+      - /mnt/data/projects:/mnt/data/projects:ro
       - /mnt/homes:/home
       - /mnt/cache:/home/cache
       - ./jupyterhub_data:/srv/jupyterhub/
@@ -34,6 +35,7 @@ services:
       - DUMMY_PASSWORD=${DUMMY_PASSWORD:-ethoscope}
       - ALLOWED_USERS=${ALLOWED_USERS}
       - ADMIN_USERS=${ADMIN_USERS}
+      - SUPERUSERS=${SUPERUSERS}
       # OAuth settings (only needed when using OAuth authenticators)
       - OAUTH_CLIENT_ID=${OAUTH_CLIENT_ID}
       - OAUTH_CLIENT_SECRET=${OAUTH_CLIENT_SECRET}
@@ -47,6 +49,14 @@ services:
       - GITHUB_ORG=${GITHUB_ORG}
       - GOOGLE_HOSTED_DOMAIN=${GOOGLE_HOSTED_DOMAIN}
       - GITLAB_URL=${GITLAB_URL}
+      # Notebook Intelligence (AI assistant) — central OpenRouter config.
+      # Put OPENROUTER_API_KEY in .env (gitignored). Models are overridable;
+      # verify/adjust slugs at https://openrouter.ai/models
+      - OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
+      - NBI_CHAT_MODEL_PROVIDER=${NBI_CHAT_MODEL_PROVIDER:-litellm-compatible}
+      - NBI_CHAT_MODEL_ID=${NBI_CHAT_MODEL_ID:-openrouter/anthropic/claude-sonnet-4.5}
+      - NBI_INLINE_COMPLETION_MODEL_PROVIDER=${NBI_INLINE_COMPLETION_MODEL_PROVIDER:-litellm-compatible}
+      - NBI_INLINE_COMPLETION_MODEL_ID=${NBI_INLINE_COMPLETION_MODEL_ID:-openrouter/openai/gpt-4o-mini}
 
     user: "${UID:-1000}:${GID:-1000}" # Adjust to your own user. Make sure this matches the permissions of your /mnt/{homes,cache} folder
     restart: always

{ethoscopy-2.2.0 → ethoscopy-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: ethoscopy
-Version: 2.2.0
+Version: 2.3.0
 Summary: "A python based toolkit to download and anlyse data from the Ethoscope hardware system."
 Author-email: Lblackhurst29 <lblackhurst29@gmail.com>
 License-File: LICENSE
@@ -19,6 +19,7 @@ Requires-Dist: hmmlearn<0.4.0,>=0.3.2
 Requires-Dist: nbformat<6.0.0,>=5.10.4
 Requires-Dist: numpy<3.0.0,>=2.0.0
 Requires-Dist: pandas<3.0.0,>=2.2.2
+Requires-Dist: pillow<13.0,>=10.0
 Requires-Dist: plotly<6.0.0,>=5.22.0
 Requires-Dist: pywavelets<2.0.0,>=1.6.0
 Requires-Dist: seaborn<0.14.0,>=0.13.2

{ethoscopy-2.2.0 → ethoscopy-2.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ethoscopy"
-version = "2.2.0"
+version = "2.3.0"
 description = "\"A python based toolkit to download and anlyse data from the Ethoscope hardware system.\""
 authors = [{name = "Lblackhurst29",email = "lblackhurst29@gmail.com"}]
 readme = "README.md"
@@ -16,6 +16,10 @@ dependencies = [
     "astropy>=7.0,<8.0",
     "nbformat>=5.10.4,<6.0.0",
     "tqdm>=4.66,<5.0",
+    # estimate_light_cycle() decodes the stored IMG_SNAPSHOTS JPEGs. Pillow
+    # already arrived transitively through seaborn -> matplotlib; this makes the
+    # direct dependency explicit.
+    "pillow>=10.0,<13.0",
 ]
 requires-python = ">=3.12,<4.0"
 classifiers = [

{ethoscopy-2.2.0 → ethoscopy-2.3.0}/src/ethoscopy/__init__.py

@@ -15,8 +15,11 @@ from ethoscopy.analyse import (
 from ethoscopy.behavpy import behavpy
 from ethoscopy.load import (
     download_from_remote_dir,
+    estimate_light_cycle,
     link_meta_index,
     load_ethoscope,
+    load_ethoscope_diagnostics,
+    load_ethoscope_light_schedule,
     load_ethoscope_metadata,
 )
 from ethoscopy.misc.general_functions import concat
@@ -28,9 +31,12 @@ __all__ = [
     "behavpy",
     "download_from_remote_dir",
     "download_tutorial_data",
+    "estimate_light_cycle",
     "get_tutorial",
     "link_meta_index",
     "load_ethoscope",
+    "load_ethoscope_diagnostics",
+    "load_ethoscope_light_schedule",
     "load_ethoscope_metadata",
     "max_velocity_detector",
     "sleep_annotation",