endoreg-db 0.8.6.3__tar.gz → 0.8.9.5__tar.gz

endoreg_db-0.8.9.5/.env.example

@@ -0,0 +1,103 @@
+# EndoReg-DB .env.example
+# Copy to .env for local development: cp .env.example .env
+DJANGO_SETTINGS_MODULE=config.settings.dev
+# -------------------------------------------------------------------
+# Core
+# -------------------------------------------------------------------
+# DJANGO_SETTINGS_MODULE=config.settings.dev
+TIME_ZONE=Europe/Berlin
+
+# -------------------------------------------------------------------
+# Paths (relative to repo root unless absolute)
+# -------------------------------------------------------------------
+STORAGE_DIR=data/storage
+ASSET_DIR=tests/assets
+STATIC_URL=/static/
+MEDIA_URL=/media/
+
+IO_DIR="../"
+
+TAG="default"
+
+# -------------------------------------------------------------------
+# Feature flags
+# -------------------------------------------------------------------
+RUN_VIDEO_TESTS=false
+SKIP_EXPENSIVE_TESTS=true
+
+# -------------------------------------------------------------------
+# Django REST Framework throttles
+# -------------------------------------------------------------------
+DRF_THROTTLE_USER=1000/hour
+DRF_THROTTLE_ANON=1000/hour
+
+# -------------------------------------------------------------------
+# Development database (SQLite by default)
+# For non-SQLite, set ENGINE and provide NAME/USER/PASSWORD/HOST/PORT
+# -------------------------------------------------------------------
+DEV_DB_ENGINE=django.db.backends.sqlite3
+DEV_DB_NAME=dev_db.sqlite3
+# DEV_DB_USER=
+# DEV_DB_PASSWORD=
+# DEV_DB_HOST=
+# DEV_DB_PORT=
+
+# -------------------------------------------------------------------
+# Testing database (persistent SQLite by default)
+# pytest uses config.settings.test; first run: pytest --reuse-db --create-db
+# -------------------------------------------------------------------
+TEST_DB_ENGINE=django.db.backends.sqlite3
+TEST_DB_NAME=data/tests/db/test_db.sqlite3
+TEST_DISABLE_MIGRATIONS=false
+
+# -------------------------------------------------------------------
+# Production (override these in your deployment environment)
+# -------------------------------------------------------------------
+# DJANGO_SECRET_KEY=<generate-strong-random-value>  # REQUIRED in production; generate securely (e.g., python -c 'import secrets; print(secrets.token_urlsafe(64))'); never commit real secrets
+# DJANGO_DEBUG=false  # Use false in production; set true only for local debugging
+# DJANGO_ALLOWED_HOSTS=example.com,api.example.com  # Set to your domains; required in production
+# DB_ENGINE=django.db.backends.postgresql  # Example: use Postgres in production
+# DB_NAME=endoreg_db  # Production database name
+# DB_USER=postgres  # Database user
+# DB_PASSWORD=  # Database password; never commit real secrets
+# DB_HOST=localhost  # Database host
+# DB_PORT=5432  # Database port
+# SECURE_SSL_REDIRECT=true  # Redirect HTTP->HTTPS in production; set false only locally
+# SESSION_COOKIE_SECURE=true  # Secure cookies in production; set false only for local/dev
+# CSRF_COOKIE_SECURE=true  # Secure CSRF cookie in production; set false only for local/dev
+# SECURE_HSTS_SECONDS=31536000  # 1 year HSTS; enable only after confirming HTTPS everywhere
+# SECURE_HSTS_INCLUDE_SUBDOMAINS=true  # Include subdomains for HSTS when appropriate
+# SECURE_HSTS_PRELOAD=false  # Enable only if you intend to submit to preload list
+
+# -------------------------------------------------------------------
+# Optional: Keycloak token validator (used by manage.py runserver)
+# -------------------------------------------------------------------
+# KEYCLOAK_REALM=master
+# KEYCLOAK_SERVER_URL=https://keycloak.endo-reg.net
+# KEYCLOAK_CLIENT_ID=lx-frontend
+# KEYCLOAK_CLIENT_SECRET=
+# TOKEN_VALIDATOR_PORT=3001
+
+# -------------------------------------------------------------------
+# Overrides when embedding this app into another Django project
+# -------------------------------------------------------------------
+# STATIC_ROOT=/abs/path/to/staticfiles
+# STORAGE_DIR=/abs/path/to/storage
+
+
+TEST_RUN=False
+TEST_RUN_FRAME_NUMBER=1000
+RUST_BACKTRACE=1
+DJANGO_FFMPEG_EXTRACT_FRAME_BATCHSIZE=500
+VIDEO_ALLOW_FPS_FALLBACK=True
+VIDEO_DEFAULT_FPS=50
+
+# Caching optimization for huggingface, ollama, etc. 
+# CAUTION: SETUP ASSUMES LUXNIX USER 'admin' WITH HOME /home/admin
+
+HF_HOME=/home/admin/.cache/huggingface
+HF_HUB_CACHE=/home/admin/.cache/huggingface/hub
+HF_HUB_ENABLE_HF_TRANSFER=1
+OLLAMA_KEEP_ALIVE=4h
+OLLAMA_MODELS=/home/admin/.ollama/models
+TRANSFORMERS_CACHE=/home/admin/.cache/huggingface/hub

endoreg_db-0.8.9.5/.github/copilot-instructions.md

@@ -0,0 +1,115 @@
+## Big Picture
+- **Core app**: Django project lives in `endoreg_db/`, `config/settings/*` drive environments via `endoreg_db.config.env`; root URLs mount the entire API under `/api/`.
+- **Domain**: Focus on ingesting and anonymizing endoscopy videos and clinical reports; `services/video_import.py` handles the full pipeline, mirrored by management commands for CLI workflows.
+- **Data layout**: Working assets stay under `storage/` (media root) and `data/` (raw/test fixtures); `endoreg_db/utils/paths.py` centralizes directories—respect existing folder contracts when writing files.
+- **Knowledge base**: Medical vocabularies live in YAML under `endoreg_db/data/`; management commands in `endoreg_db/management/commands/load_*` hydrate models from those files—keep YAML schema changes aligned with the commands.
+- **Async flows**: `celery_app.py` wires Celery but operational imports use Redis Queue (`endoreg_db/tasks/video_ingest.py`); align new background work with that queue unless migrating everything to Celery.
+
+## Django Db Model Type Hinting:
+- **TYPE_CHECKING**: Use `if TYPE_CHECKING:` blocks to declare model attributes for type checkers without runtime overhead.
+
+**Implementation Examples**:
+```python
+from django.db import models
+from typing import TYPE_CHECKING, cast
+
+class ExampleClass1(models.Model):
+    name = models.CharField(max_length=255, unique=True)
+    description = models.TextField(blank=True, null=True)
+
+    # One-to-many relationship
+    example_1 = models.ForeignKey(
+        "ExampleClass2",
+        on_delete=models.CASCADE,
+        related_name="example_class_1_set",
+    )
+
+    # One-to-one relationship
+    example_2 = models.OneToOneField(
+        "ExampleClass2",
+        on_delete=models.CASCADE,
+        related_name="example_class_1_one_to_one",
+    )
+
+    # Many to Many
+    example_3 = models.ManyToManyField(
+        "ExampleClass2",
+        related_name="example_class_1_many_to_many_set",
+    )
+
+    # Nullable ForeignKey
+    example_4 = models.ForeignKey(
+        "ExampleClass2",
+        on_delete=models.SET_NULL,
+        null=True,
+        blank=True,
+        related_name="example_class_1_nullable_foreign_key_set",
+    )
+
+    if TYPE_CHECKING:
+
+        # One-to-manyrelationship
+        example_1: models.ForeignKey["ExampleClass2"]
+
+        # One-to-one relationship
+        example_2: models.OneToOneField["ExampleClass2"]
+
+        # Many to Many
+        example_3 = cast(models.manager.RelatedManager["ExampleClass2"], example_3)
+
+        # Nullable ForeignKey
+        example_4: models.ForeignKey["ExampleClass2|None"]
+
+class ExampleClass2(models.Model):
+    name = models.CharField(max_length=255)
+
+    if TYPE_CHECKING:
+        # Reverse relation for one-to-many relationship
+        @property
+        def example_class_1_set(self) -> models.RelatedManager["ExampleClass1"]: ...
+
+        # Reverse relation for one-to-one relationship
+        example_class_1_one_to_one: models.OneToOneField["ExampleClass1"]
+
+        # Reverse relation for many-to-many relationship
+        @property
+        def example_class_1_many_to_many_set(self) -> models.RelatedManager["ExampleClass1"]: ...
+
+        # Reverse relation for nullable ForeignKey
+        @property
+        def example_class_1_nullable_foreign_key_set(self) -> models.RelatedManager["ExampleClass1"]: ... 
+
+```
+## Environment & Tooling
+- **Platform baseline**: Most development happens on NixOS; run `devenv up` (or allow via `direnv`) before any command so the pinned toolchain and `uv` environment from `devenv.nix` activate.
+- **Dev shell**: Once inside the shell, use `devenv task run env:build` to refresh `.env` via `env_setup.py`; the task relies on `.devenv-vars.json` generated during shell entry.
+- **Python entry**: Always call `uv run ...` (e.g. `uv run python manage.py migrate`) so dependencies resolve inside `.devenv/state/venv`.
+- **Settings switch**: `DJANGO_SETTINGS_MODULE` defaults to `config.settings.dev`; tests use `config.settings.test` which persists an SQLite DB at `data/tests/db/test_db.sqlite3`.
+- **External repos**: LX anonymizer support is expected (`lx-anonymizer>=0.8.2.1`); the Nix enter hook can clone it—check this before debugging anonymization failures.
+
+## Testing
+- **Copilot test instructions**: Use `uv run pytest` to run tests since other test discovery causes issues
+- **Test runner**: Prefer `uv run python runtests.py` to execute curated suites; arguments map to subpackages under `tests/` (e.g. `runtests helpers`).
+- **Pytest direct**: For ad-hoc cases run `uv run pytest tests/path::TestClass -k keyword`; `pytest.ini` already points at the Django settings.
+- **Video guards**: GPU/ffmpeg-heavy specs are behind `RUN_VIDEO_TESTS`; default false—set `RUN_VIDEO_TESTS=true` when required assets in `tests/assets/` are available.
+- **Fixtures**: `tests/conftest.py` configures storage dirs and shared fixtures; extend `endoreg_db/factories/` instead of hand-building ORM objects in tests.
+
+## Data & Management Commands
+- **Database seed**: `load_base_db_data` (and other `load_*` commands) populate clinical taxonomies from CSVs under `data/` and `models_table.*`.
+- **Knowledge YAML**: Commands like `load_disease_data`, `load_finding_data`, etc. expect curated YAML in `endoreg_db/data/`; keep keys stable and update factories/tests when the YAML schema shifts.
+- **Import scripts**: Video/report ingestion commands delegate to `services/video_import.py` and `services/pdf_import.py`; reuse the service layer when adding new commands.
+- **Backups**: `export_db.sh` and `import_db.sh` wrap Django `dumpdata/loaddata`—update them instead of reinventing backup flows.
+- **Paths**: `env_setup.py` writes `STORAGE_DIR` into `.env`; new code should resolve paths via `endoreg_db.utils.data_paths` or `config.env_path` to stay relocatable.
+
+## API & Presentation
+- **Routing**: `endoreg_db/root_urls.py` includes `endoreg_db/urls/`; keep new REST endpoints inside `endoreg_db/api/views` + serializers and register them in `api_urls.py`.
+- **Translations**: `django-modeltranslation` is preloaded (languages `de`/`en`); text fields need `TranslationOptions` under `endoreg_db/models/**/translation*.py`.
+- **Forms/Admin**: Custom admin and form logic lives in `endoreg_db/admin.py` and `endoreg_db/forms/`; follow existing Bootstrap 5 widgets in `endoreg_db/templates/`.
+- **Security**: Sensitive metadata models in `endoreg_db/models/metadata/` use state objects—preserve their transitions (see `VideoImportService._process_frames_and_metadata`).
+
+## Observability & Utilities
+- **Logging**: `config/settings/base.py` defines `TEST_LOGGER_NAMES`; use `logging.getLogger(__name__)` so unit tests capture module-level logs consistently.
+- **Pipelines**: `endoreg_db/utils/pipelines/process_video_dir.py` shows the orchestration pattern—refresh model state after `pipe_1/pipe_2` and reuse its helpers for batch jobs.
+- **Permissions**: Storage-aware helpers live in `endoreg_db/utils/permissions.py` and related services—reuse them instead of manual `os.path` checks.
+- **Path hygiene**: File movement bugs usually stem from bypassing `endoreg_db.utils.paths` or `endoreg_db.config.env`; always derive locations from `data_paths[...]`/`env_path(...)` so raw/anonymized video & report paths stay consistent.
+- **RQ usage**: Enqueue jobs via `endoreg_db/tasks/video_ingest.enqueue_video_import`; ensure `RQ_REDIS_URL` is configured and Django is initialized before background execution.

endoreg_db-0.8.9.5/AGENTS.md

@@ -0,0 +1,11 @@
+# Endoreg-db Agents.md
+
+Case Generator and Requirements Module:
+
+- Prefer yaml config
+- Reference load_base_db_data
+
+API and Integration
+
+- No camelCase whatsoever
+- LX-Annotate has automatic conversion as is expected from other api accessors.

endoreg_db-0.8.9.5/CONFIGURATION_GUIDE.md

@@ -0,0 +1,124 @@
+# EndoReg-DB Configuration and Environment Guide
+
+This repository is a reusable Django app. It ships a small, robust settings package for local development and CI, while encouraging host projects to provide their own settings.
+
+## Settings modules
+
+- config/settings/base.py: shared defaults; driven by environment variables.
+- config/settings/dev.py: local development; SQLite by default.
+- config/settings/test.py: tests; persistent SQLite test DB by default.
+- config/settings/prod.py: production defaults; fully env-driven.
+
+Legacy settings (prod_settings.py, dev/dev_settings.py, tests/test_settings.py) are thin wrappers and can be removed after consumers update.
+
+## Centralized environment handling
+
+- Use helpers in `endoreg_db/config/env.py` (env_str, env_bool, env_int, env_path).
+- .env is not loaded during pytest to prevent test runs from picking up dev settings.
+- Under pytest, `DJANGO_SETTINGS_MODULE` is forced to `config.settings.test`.
+
+## Key environment variables
+
+General
+- DJANGO_SETTINGS_MODULE: choose settings module (defaults used in manage.py/wsgi.py/pytest.ini).
+- STORAGE_DIR: absolute path to media storage (defaults to storage/ in repo).
+- STATIC_URL, STATIC_ROOT, MEDIA_URL: override static/media paths if embedding.
+- IO_DIR: Input Output dir, here data dropoff and export is handled.
+- TIME_ZONE: defaults to Europe/Berlin.
+
+Development (config.settings.dev)
+- DEV_DB_ENGINE: default django.db.backends.sqlite3
+- DEV_DB_NAME: default BASE_DIR/dev_db.sqlite3
+- DEV_DB_USER, DEV_DB_PASSWORD, DEV_DB_HOST, DEV_DB_PORT: used for non-SQLite engines.
+
+Testing (config.settings.test)
+- TEST_DB_ENGINE: default django.db.backends.sqlite3
+- TEST_DB_NAME: default data/tests/db/test_db.sqlite3
+- TEST_DB_FILE: alternative way to set SQLite DB path
+- TEST_DISABLE_MIGRATIONS: true|false (default false)
+
+Production (config.settings.prod)
+- DJANGO_SECRET_KEY: required (must be a strong random value; never commit real secrets)
+- DJANGO_DEBUG: true|false (use false in production)
+- DJANGO_ALLOWED_HOSTS: comma-separated
+- DB_ENGINE, DB_NAME, DB_USER, DB_PASSWORD, DB_HOST, DB_PORT
+- SECURE_SSL_REDIRECT, SESSION_COOKIE_SECURE, CSRF_COOKIE_SECURE
+- SECURE_HSTS_SECONDS, SECURE_HSTS_INCLUDE_SUBDOMAINS, SECURE_HSTS_PRELOAD
+
+## Typical usage patterns
+
+As an embedded app in a host project:
+- Add 'endoreg_db' to INSTALLED_APPS in the host settings.
+- Define STORAGE_DIR in the host environment.
+- Run migrations in the host project (this app contributes its migrations).
+- Run the complete setup command: `python manage.py setup_endoreg_db`
+
+The `setup_endoreg_db` command performs all necessary initialization:
+1. Loads base database data (medical vocabularies, centers, etc.)
+2. Creates Django cache table for API functionality (only when using database-backed caching)
+3. Sets up AI models and labels (unless --skip-ai-setup is used)
+4. Creates AI model metadata with weights
+5. Verifies the setup was successful
+
+The command automatically detects your cache configuration:
+- For LocMemCache (default): Skips cache table creation
+- For database caching: Creates the required cache tables
+
+Use `--skip-ai-setup` if AI video processing features are not needed, or `--force-recreate` to recreate AI metadata.
+
+This repo standalone (local):
+- Development server: DJANGO_SETTINGS_MODULE=config.settings.dev python manage.py runserver
+- Tests (persistent test DB): pytest --reuse-db --create-db
+- Clean test DB: rm -f data/tests/db/test_db.sqlite3
+
+CI tips
+- Use DJANGO_SETTINGS_MODULE=config.settings.test
+- First run use --create-db to run migrations once; subsequent runs can cache the database file.
+- Override TEST_DB_NAME to a workspace cache path if needed.
+
+## Direnv/Devenv
+- Ensure devenv.nix and direnv don’t mutate repo files. Editor should inherit direnv env if used.
+
+## Removing legacy settings
+- Replace imports of prod_settings, dev/dev_settings.py, tests/test_settings.py with config.settings.prod/dev/test.
+- Update scripts: scripts/django_setup.py, check_video_files.py, etc., to default to config.settings.dev/test.
+
+## AI Model Setup (for video processing features)
+
+When using EndoReg DB's AI-powered video processing features, ensure model weights are available:
+
+### Model Weights Location
+The system looks for model weights in these locations (in order of preference):
+1. `STORAGE_DIR/model_weights/` (recommended for production)
+2. `tests/assets/` (for development/testing)
+3. `assets/` (fallback location)
+
+### Required Model Files
+For colonoscopy video processing, the following model file is required:
+- `colo_segmentation_RegNetX800MF_6.safetensors` - Multilabel classification model for colonoscopy
+
+### Automatic Setup
+The `setup_endoreg_db` command automatically:
+- Loads AI model definitions and labels
+- Creates model metadata with weights
+- Sets up the default AI model for video processing
+
+### Manual Setup (if needed)
+If automatic setup fails, run these commands individually:
+```bash
+python manage.py load_ai_model_data
+python manage.py load_ai_model_label_data
+python manage.py createcachetable
+python manage.py create_multilabel_model_meta --model_name image_multilabel_classification_colonoscopy_default --model_meta_version 1 --image_classification_labelset_name multilabel_classification_colonoscopy_default
+```
+
+### Troubleshooting AI Setup
+- **"Model file not found"**: Ensure model weights are in one of the expected locations
+- **"No model metadata found"**: Run the setup commands or use `--force-recreate`
+- **Import errors**: Check that the `EndoscopyProcessor` import fix is applied in `video_import.py`
+
+## Production checklist
+- Set DJANGO_SECRET_KEY to a strong random value (never commit). 
+- Set DJANGO_ALLOWED_HOSTS to your domains.
+- Enforce HTTPS: SECURE_SSL_REDIRECT=true, cookie secure flags true.
+- Consider HSTS: set SECURE_HSTS_SECONDS (e.g., 31536000) only when ready; include subdomains/preload as appropriate.