endoreg-db 0.8.4.1__tar.gz → 0.8.4.2__tar.gz

endoreg_db-0.8.4.2/CONFIGURATION_GUIDE.md

@@ -0,0 +1,123 @@
+# EndoReg-DB Configuration and Environment Guide
+
+This repository is a reusable Django app. It ships a small, robust settings package for local development and CI, while encouraging host projects to provide their own settings.
+
+## Settings modules
+
+- config/settings/base.py: shared defaults; driven by environment variables.
+- config/settings/dev.py: local development; SQLite by default.
+- config/settings/test.py: tests; persistent SQLite test DB by default.
+- config/settings/prod.py: production defaults; fully env-driven.
+
+Legacy settings (prod_settings.py, dev/dev_settings.py, tests/test_settings.py) are thin wrappers and can be removed after consumers update.
+
+## Centralized environment handling
+
+- Use helpers in `endoreg_db/config/env.py` (env_str, env_bool, env_int, env_path).
+- .env is not loaded during pytest to prevent test runs from picking up dev settings.
+- Under pytest, `DJANGO_SETTINGS_MODULE` is forced to `config.settings.test`.
+
+## Key environment variables
+
+General
+- DJANGO_SETTINGS_MODULE: choose settings module (defaults used in manage.py/wsgi.py/pytest.ini).
+- STORAGE_DIR: absolute path to media storage (defaults to storage/ in repo).
+- STATIC_URL, STATIC_ROOT, MEDIA_URL: override static/media paths if embedding.
+- TIME_ZONE: defaults to Europe/Berlin.
+
+Development (config.settings.dev)
+- DEV_DB_ENGINE: default django.db.backends.sqlite3
+- DEV_DB_NAME: default BASE_DIR/dev_db.sqlite3
+- DEV_DB_USER, DEV_DB_PASSWORD, DEV_DB_HOST, DEV_DB_PORT: used for non-SQLite engines.
+
+Testing (config.settings.test)
+- TEST_DB_ENGINE: default django.db.backends.sqlite3
+- TEST_DB_NAME: default data/tests/db/test_db.sqlite3
+- TEST_DB_FILE: alternative way to set SQLite DB path
+- TEST_DISABLE_MIGRATIONS: true|false (default false)
+
+Production (config.settings.prod)
+- DJANGO_SECRET_KEY: required (must be a strong random value; never commit real secrets)
+- DJANGO_DEBUG: true|false (use false in production)
+- DJANGO_ALLOWED_HOSTS: comma-separated
+- DB_ENGINE, DB_NAME, DB_USER, DB_PASSWORD, DB_HOST, DB_PORT
+- SECURE_SSL_REDIRECT, SESSION_COOKIE_SECURE, CSRF_COOKIE_SECURE
+- SECURE_HSTS_SECONDS, SECURE_HSTS_INCLUDE_SUBDOMAINS, SECURE_HSTS_PRELOAD
+
+## Typical usage patterns
+
+As an embedded app in a host project:
+- Add 'endoreg_db' to INSTALLED_APPS in the host settings.
+- Define STORAGE_DIR in the host environment.
+- Run migrations in the host project (this app contributes its migrations).
+- Run the complete setup command: `python manage.py setup_endoreg_db`
+
+The `setup_endoreg_db` command performs all necessary initialization:
+1. Loads base database data (medical vocabularies, centers, etc.)
+2. Creates Django cache table for API functionality (only when using database-backed caching)
+3. Sets up AI models and labels (unless --skip-ai-setup is used)
+4. Creates AI model metadata with weights
+5. Verifies the setup was successful
+
+The command automatically detects your cache configuration:
+- For LocMemCache (default): Skips cache table creation
+- For database caching: Creates the required cache tables
+
+Use `--skip-ai-setup` if AI video processing features are not needed, or `--force-recreate` to recreate AI metadata.
+
+This repo standalone (local):
+- Development server: DJANGO_SETTINGS_MODULE=config.settings.dev python manage.py runserver
+- Tests (persistent test DB): pytest --reuse-db --create-db
+- Clean test DB: rm -f data/tests/db/test_db.sqlite3
+
+CI tips
+- Use DJANGO_SETTINGS_MODULE=config.settings.test
+- First run use --create-db to run migrations once; subsequent runs can cache the database file.
+- Override TEST_DB_NAME to a workspace cache path if needed.
+
+## Direnv/Devenv
+- Ensure devenv.nix and direnv don’t mutate repo files. Editor should inherit direnv env if used.
+
+## Removing legacy settings
+- Replace imports of prod_settings, dev/dev_settings.py, tests/test_settings.py with config.settings.prod/dev/test.
+- Update scripts: scripts/django_setup.py, check_video_files.py, etc., to default to config.settings.dev/test.
+
+## AI Model Setup (for video processing features)
+
+When using EndoReg DB's AI-powered video processing features, ensure model weights are available:
+
+### Model Weights Location
+The system looks for model weights in these locations (in order of preference):
+1. `STORAGE_DIR/model_weights/` (recommended for production)
+2. `tests/assets/` (for development/testing)
+3. `assets/` (fallback location)
+
+### Required Model Files
+For colonoscopy video processing, the following model file is required:
+- `colo_segmentation_RegNetX800MF_6.ckpt` - Multilabel classification model for colonoscopy
+
+### Automatic Setup
+The `setup_endoreg_db` command automatically:
+- Loads AI model definitions and labels
+- Creates model metadata with weights
+- Sets up the default AI model for video processing
+
+### Manual Setup (if needed)
+If automatic setup fails, run these commands individually:
+```bash
+python manage.py load_ai_model_data
+python manage.py load_ai_model_label_data
+python manage.py createcachetable
+python manage.py create_multilabel_model_meta --model_name image_multilabel_classification_colonoscopy_default --model_meta_version 1 --image_classification_labelset_name multilabel_classification_colonoscopy_default
+```
+
+### Troubleshooting AI Setup
+- **"Model file not found"**: Ensure model weights are in one of the expected locations
+- **"No model metadata found"**: Run the setup commands or use `--force-recreate`
+- **Import errors**: Check that the `EndoscopyProcessor` import fix is applied in `video_import.py`
+
+## Production checklist
+- Set DJANGO_SECRET_KEY to a strong random value (never commit). 
+- Set DJANGO_ALLOWED_HOSTS to your domains.
+- Enforce HTTPS: SECURE_SSL_REDIRECT=true, cookie secure flags true.
+- Consider HSTS: set SECURE_HSTS_SECONDS (e.g., 31536000) only when ready; include subdomains/preload as appropriate.

endoreg_db-0.8.4.2/PKG-INFO

@@ -0,0 +1,384 @@
+Metadata-Version: 2.4
+Name: endoreg-db
+Version: 0.8.4.2
+Summary: EndoReg Db Django App
+Project-URL: Homepage, https://info.coloreg.de
+Project-URL: Repository, https://github.com/wg-lux/endoreg-db
+Author: Max Hild, Hamza Zahid, Peter Kowalczyk
+Author-email: "Thomas J. Lux" <lux_t1@ukw.de>
+Maintainer: Max Hild, Hamza Zahid, Peter Kowalczyk
+Maintainer-email: "Thomas J. Lux" <lux_t1@ukw.de>
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
+Requires-Python: <3.13,>=3.12
+Requires-Dist: agl-report-reader>=0.4.0
+Requires-Dist: build>=1.3.0
+Requires-Dist: celery>=5.5.3
+Requires-Dist: cryptography>=44.0.2
+Requires-Dist: decord==0.6.0
+Requires-Dist: django-bootstrap5>=25.2
+Requires-Dist: django-cors-headers>=4.8.0
+Requires-Dist: django-extensions>=3.2.3
+Requires-Dist: django-filter>=25.1
+Requires-Dist: django-modeltranslation>=0.19.16
+Requires-Dist: django>=5.2.4
+Requires-Dist: djangorestframework>=3.16.1
+Requires-Dist: dotenv>=0.9.9
+Requires-Dist: faker>=37.6.0
+Requires-Dist: flake8>=7.3.0
+Requires-Dist: gunicorn>=23.0.0
+Requires-Dist: huggingface-hub>=0.35.3
+Requires-Dist: icecream>=2.1.4
+Requires-Dist: librosa==0.11.0
+Requires-Dist: llvmlite>=0.44.0
+Requires-Dist: lx-anonymizer[llm,ocr]>=0.8.8
+Requires-Dist: moviepy==2.2.1
+Requires-Dist: mypy>=1.16.0
+Requires-Dist: numpy>=2.2.3
+Requires-Dist: opencv-python>=4.11.0.86
+Requires-Dist: openpyxl>=3.1.5
+Requires-Dist: optimum==1.26.1
+Requires-Dist: pandas>=2.2.3
+Requires-Dist: pillow>=11.3.0
+Requires-Dist: pip>=25.2
+Requires-Dist: pydantic>=2.10.6
+Requires-Dist: pyjwt>=2.10.1
+Requires-Dist: pytesseract>=0.3.13
+Requires-Dist: pytest-cov>=6.1.1
+Requires-Dist: pytest-django>=4.11.1
+Requires-Dist: pytest-env>=1.1.5
+Requires-Dist: pytest>=8.3.5
+Requires-Dist: python-ffmpeg>=2.0.12
+Requires-Dist: python-magic>=0.4.27
+Requires-Dist: pytorch-lightning>=2.5.1
+Requires-Dist: pyyaml>=6.0.2
+Requires-Dist: regex>=2024.11.6
+Requires-Dist: requests>=2.32.3
+Requires-Dist: ruff>=0.14.0
+Requires-Dist: scikit-learn>=1.5.2
+Requires-Dist: scipy>=1.16.1
+Requires-Dist: soundfile==0.12.1
+Requires-Dist: sphinx-rtd-theme>=3.0.2
+Requires-Dist: sphinx>=8.2.3
+Requires-Dist: tesseract>=0.1.3
+Requires-Dist: torch==2.5.1
+Requires-Dist: torchaudio==2.5.1
+Requires-Dist: torchvision==0.20.1
+Requires-Dist: tqdm>=4.67.0
+Requires-Dist: transformers==4.44.2
+Requires-Dist: twine>=6.2.0
+Requires-Dist: urllib3>=2.2.3
+Requires-Dist: vector-quantize-pytorch==1.18.5
+Requires-Dist: vocos==0.1.0
+Requires-Dist: whitenoise>=6.10.0
+Provides-Extra: dev
+Description-Content-Type: text/markdown
+
+# EndoregDB - Professional Data Infrastructure for Clinical Research
+
+EndoregDB is a comprehensive database framework designed to manage medical and research-related data for clinical trials. This repository focuses on efficient data processing, automated deployment, security, and reproducibility, offering a flexible setup for local development environments as well as distributed systems. It supports the integration of AI/ML tools and advanced image and report processing.
+
+This infrastructure was originally designed for clinical research studies and is optimized for handling large data volumes, including:
+
+- Medical reports,
+- Patient imaging and video data,
+- Clinical product and treatment data,
+  and more.
+
+## 🚀 Key Features
+
+### System Architecture
+
+- **Modular Design**: Built on scalable and reusable components to simplify integration into various environments.
+- **Multi-System Support**: Manages configurations for local workstations and production servers.
+- **Role-Specific Configuration**: Predefined roles for common use cases:
+  - Medical data processing systems
+  - AI/ML model deployment
+  - Research workstation configuration
+
+### Security & Data Management
+
+- **Data Encryption**: All sensitive data is encrypted, and privacy policies are enforced.
+- **Impermanence**: Stateless system configuration with persistence for critical data.
+- **Access Control**: Role-based access and identity management integration.
+
+### Data and Processing Environment
+
+- **Data Processing**: Optimized for processing medical datasets with preprocessing tools.
+- **AI/ML Support**:
+  - Integration of machine learning tools for predictive analysis.
+  - TensorFlow, PyTorch, and other frameworks supported for model training.
+- **Image/Video Processing**: Support for analyzing patient images and clinical videos.
+
+### Development Tools & Infrastructure
+
+- **Data Science Toolchains**: Pre-configured environments for data processing, analysis, and visualization.
+- **Monitoring & Logging**: Setup for continuous monitoring and logging to ensure system stability and performance.
+
+---
+
+## 🛠 Getting Started
+
+### Prerequisites
+
+- A Linux-based system (Ubuntu/Debian recommended) or NixOS
+- Hardware with sufficient storage for data processing (at least 1 TB recommended)
+
+### Quick Start
+
+1. Clone the repository:
+
+   ```bash
+   git clone https://github.com/wg-lux/endoreg-db.git
+   cd endoreg-db
+   ```
+
+2. Set up your Python environment
+   We need to have a `devenv.nix` file.  
+   This Nix `devenv.nix` configuration sets up a Python development environment for a Django-based project using `uv` for dependency management. It defines project directories, environment variables, runtime packages, and several development tasks and scripts.
+
+   **Some available Test Shortcuts**
+
+   - `runtests`: Runs all tests — `uv run python runtests.py`
+   - `runtests-dataloader`: Runs dataloader tests — `uv run python runtests.py 'dataloader'`
+   - `runtests-other`: Runs other miscellaneous tests — `uv run python runtests.py 'other'`
+   - `runtests-helpers`: Runs helper module tests — `uv run python runtests.py 'helpers'`
+   - `runtests-administration`: Runs admin module tests — `uv run python runtests.py 'administration'`
+   - `runtests-medical`: Runs medical module tests — `uv run python runtests.py 'medical'`
+
+3. Then run 
+
+   ```bash
+   direnv allow
+   ```
+
+4. Run tests:
+   Call Devenv Script to run tests
+
+   ```bash
+   runtests
+   ```
+   Tests Overview
+   - These tests ensure the functionality of different models and scenarios.
+   - After running them, you can view the results as demonstrated in the image below:
+   
+   ![Test Results](Images/testscreenshort.png)
+
+5. Run 
+   ```python
+   python manage.py migrate
+   
+   ``` 
+   - It applies database migrations and make tables.
+   - It updates your database schema to match the current state of your Django models.
+
+6. To load the database data run 
+   ```
+   python manage.py load_base_db_data
+
+   ```
+   ![Data](Images/loadbasedata0.png)
+   ![Data](Images/loadbasedata1.png)
+   ![Data](Images/loadbasedata2.png)
+   ![Data](Images/loadbasedata3.png)
+   ![Data](Images/loadbasedata4.png)
+   ![Data](Images/loadbasedata4b.png)
+   ![Data](Images/loadbasedata5.png)
+
+7. Accessing the Django Shell
+   - To fetch or interact with data in the terminal, run the following command to run the Django shell:
+
+   ```bash
+      python manage.py shell
+   ```
+   - Using the Django shell, you can:
+      - Import database models
+      - Fetch data from the database
+      - Access related data through model relationships (e.g., foreign keys, one-to-many, many-to-many)
+      - Example is shown below
+
+   #### EXAMPLE # 1
+   ![Shell](Images/shell2.png)
+   - Explanation:
+      This script fetches a patient by ID and prints their related examination(s) using Django ORM. It retrieves the examination name linked to the patient from the PatientExamination table.
+      
+   #### EXAMPLE # 2
+   ![Shell](Images/shell0.png)
+   - Explanation:
+      In the Django shell, a specific ExaminationIndication named "colonoscopy_screening" was fetched, and its related FindingIntervention records were accessed using the reverse relation expected_interventions. The first intervention (colon_lesion_polypectomy_cold_snare) was then queried to confirm it is also linked to multiple indications, demonstrating a many-to-many relationship between indications and interventions.
+   
+   #### EXAMPLE # 3
+   ![Shell](Images/shell1.png)
+   - Explanation:
+      All required labels (polyp, instrument, digital_chromo_endoscopy, etc.) are confirmed to exist. The first available video (VideoFile) was loaded, with a valid frame_dir. Using the label "polyp", 8 labeled polyp segments were found in that video, with specific start and end frame numbers.
+
+   #### EXAMPLE # 4
+   ##### Image a
+
+   ![Shell](Images/shell3.png)
+
+   ##### Image b - All classifications with their choices together
+
+   ![Shell](Images/shell3b.png)
+
+   - Explanation: Using the Django shell to fetch all morphology classifications (e.g., NICE, Paris) and their related choices  from the database.
+
+
+
+
+---
+## 📦 Database Backup and Restore
+
+This project includes two shell scripts to **export** and **import** database data in JSON format using Django's management commands.
+
+###  Setup
+
+First, make the scripts executable:
+
+```bash
+chmod +x import_db.sh
+chmod +x export_db.sh
+```
+###  Export (Backup) the Database
+
+To export the current database into a JSON file:
+
+```bash
+./export_db.sh
+```
+This will create a backup file such as `endoreg_db_backup.json`.
+
+#### List of the comands in 'export_db.sh'
+1. `python manage.py dumpdata --indent 4 --output=endoreg_db_backup.json` (if migrate comand generates and stores data in database table then wee nee dto exclude those tables from dumping)
+
+2. `python manage.py shell < fix_endoreg_db_backup_json.py`
+
+###  Import (Restore) the Database
+
+To load the data back into the database
+```bash
+./import_db.sh
+```
+####  List of the comands in 'import_db.sh'
+1. `rm dev_db.sqlite3`
+2. `python manage.py migrate`
+3. `python manage.py shell < fix_endoreg_db_backup_json.py`
+4. `python manage.py loaddata endoreg_db_backup_fixed.json`
+
+---
+
+
+
+## 📁 Repository Structure
+
+```
+endoreg-db/
+├── endoreg_db/                # Main Django app for medical data
+│   ├── case_generator/        # Sample case generator
+│   ├── data/                  # Medical knowledge base
+│   ├── management/            # Data wrangling operations
+│   ├── models/                # Data models
+│   ├── migrations/            # Database migrations
+│   └── serializers/           # Serializers for data
+├── .gitignore                 # Git ignore file for unnecessary files
+└── README.md                  # Project description and setup instructions
+```
+
+---
+
+## 🔒 Security Features
+
+- **Data Encryption**: All sensitive patient data is encrypted.
+- **Role-Based Access Control**: Configurable roles for managing access to various parts of the system.
+- **Logging & Auditing**: Comprehensive logging system that tracks user activities and data changes.
+
+---
+
+## 🖥️ Supported Systems
+
+- **Workstations**: Local development or research workstations with low data processing demands.
+- **Servers**: Scalable server infrastructure for processing large data volumes, integrated with cloud services for scalability.
+
+---
+
+## 🛟 Support
+
+For issues and questions:
+
+- Create an issue in the repository
+- Review the Deployment Guide for common issues
+
+---
+
+## 📜 License
+
+MIT - see LICENSE
+
+---
+
+
+## 📖 Further Documentation
+
+All extended documentation lives in the project **Wiki** → **[Browse the Wiki »](https://github.com/wg-lux/endoreg-db/wiki)**
+
+### Optimization Documentation
+- [Complete Optimization Project Report](https://github.com/wg-lux/endoreg-db/wiki/Complete-Optimiztion-Project-Report)
+- [Test Performance Optimization Guide](https://github.com/wg-lux/endoreg-db/wiki/Test-Performance-Optimization-Guide)
+- [Test Performance Optimization - Success Summary](https://github.com/wg-lux/endoreg-db/wiki/Test-Performance-Optimization-‐-Succes-Summary)
+- [Test Performance Optimization: Complete Implementation Summary](https://github.com/wg-lux/endoreg-db/wiki/Test-Performance-Optimization:-Complete-Implementation-Summary)
+- [Test Suite Optimization - Final Status Report](https://github.com/wg-lux/endoreg-db/wiki/Test-Suite-Optimization-‐-Final-Status-Report)
+- [Test Suite Analysis & Optimization Plan](https://github.com/wg-lux/endoreg-db/wiki/Test-Suite-Analysis-&-Optimization-Plan)
+
+---
+
+### Models and Migration Documentation
+- [Models Documentation](https://github.com/wg-lux/endoreg-db/wiki/Models-Documentation)
+- [Test Migration & Optimization Report](https://github.com/wg-lux/endoreg-db/wiki/Test-Migration-&-Optimization-Report)
+- [Test Migration Success Summary](https://github.com/wg-lux/endoreg-db/wiki/Test-Migration-Success-Summary)
+- [Test Optimization Migration Guide](https://github.com/wg-lux/endoreg-db/wiki/Test-Optimization-Migration-Guide)
+
+---
+
+### API Documentation
+- [Upload API Documentation](https://github.com/wg-lux/endoreg-db/wiki/Upload-API-Documentation)
+
+---
+
+### Frame-Anonymisierung
+- [Frame-Anonymisierung](https://github.com/wg-lux/endoreg-db/wiki/Frame-Anonymisierung)
+
+---
+
+### Tutorials Documentation
+- [Run Production Server](https://github.com/wg-lux/endoreg-db/wiki/Run-Production-Server)
+- [Date and Time Standardization for Models](https://github.com/wg-lux/endoreg-db/wiki/Date-and-Time-Standardization-for-Models)
+
+---
+
+### Keycloak
+- [How to Create a New Account for Keycloak + Nextcloud](https://github.com/wg-lux/endoreg-db/wiki/How-to-Create-a-New-Account-for-Keycloak-+-Nextcloud)
+- [Integration with the frontend](https://github.com/wg-lux/endoreg-db/wiki/Integration-with-the-frontend)
+- [Merging Multi-User Accounts in Nextcloud // current options](https://github.com/wg-lux/endoreg-db/wiki/Merging-Multi-User-Accounts-in-Nextcloud-//-current-options)
+- [New user login steps for keycloak and nextcloud](https://github.com/wg-lux/endoreg-db/wiki/New-user-login-steps-for-keycloak-and-nextcloud)
+- [keycloak integration with backend endpoint](https://github.com/wg-lux/endoreg-db/wiki/keycloak-integration-with-backend-endpoint)
+
+---
+
+### Coding Principles & Practices
+- [Timestamp Naming Standard](https://github.com/wg-lux/endoreg-db/wiki/Timestamp-Naming-Standard)
+
+---
+
+### Figures
+- [Coloreg](https://github.com/wg-lux/endoreg-db/wiki/Coloreg)
+- [EndoReg Framework](https://github.com/wg-lux/endoreg-db/wiki/EndoReg-Framework)
+- [EndoReg Data Collection Workflow](https://github.com/wg-lux/endoreg-db/wiki/EndoReg-Data-Collection-Workflow)
+- [Eine gemeinsame Datenplattform für Klinik & Forschung](https://github.com/wg-lux/endoreg-db/wiki/Eine-gemeinsame-Datenplattform-für-Klinik-&-Forschung)
+
+---
+
+### Miscellaneous
+- [Requirement System Guide](https://github.com/wg-lux/endoreg-db/wiki/Requirement-System-Guide)
+- [Official Site Link](https://github.com/wg-lux/endoreg-db/wiki/Official-Site-Link)