endoreg-db 0.8.3.8__tar.gz → 0.8.6.3__tar.gz

endoreg_db-0.8.6.3/HOW_WE_KEEP_DATA_PRIVATE.md

@@ -0,0 +1,383 @@
+# Two-Phase Patient Data Pattern
+
+## Overview
+
+The endoreg-db system implements a **two-phase pattern** for handling patient data during video import and anonymization. This pattern allows the system to:
+
+1. **Create valid SensitiveMeta instances** even when patient data is not yet available
+2. **Update with real patient data** once it becomes available (e.g., from OCR extraction)
+3. **Maintain referential integrity** while recalculating hashes and pseudo-entities
+
+This document explains how this pattern works and why it's necessary.
+
+---
+
+## The Problem
+
+During video import, patient data is not immediately available:
+
+```python
+# Video import starts
+video = VideoFile.create_from_file_initialized(file_path, center_name, processor_name)
+video.initialize()  # ← Needs SensitiveMeta, but no patient data yet!
+```
+
+However, `SensitiveMeta` requires:
+- `patient_first_name` (for hash calculation)
+- `patient_last_name` (for hash calculation)  
+- `patient_dob` (for hash calculation)
+- `center` (for anonymization)
+
+**Without these fields, hash calculation fails:**
+```
+AssertionError: First name is required to calculate patient hash.
+```
+
+Patient data is only extracted **later** during frame cleaning:
+```python
+# lx_anonymizer extracts patient data from video frames
+extracted_metadata = {
+    "patient_first_name": "Max",
+    "patient_last_name": "Mustermann",
+    "patient_dob": date(1985, 3, 15),
+}
+```
+
+---
+
+## The Solution: Two-Phase Pattern
+
+### Phase 1: Initial Creation (with Defaults)
+
+When `SensitiveMeta` is first created, **default values** are set to allow hash calculation:
+
+```python
+# video_file.py - get_or_create_sensitive_meta()
+default_data = {
+    "patient_first_name": "Patient",      # Default
+    "patient_last_name": "Unknown",       # Default
+    "patient_dob": date(1990, 1, 1),      # Default
+    "examination_date": date.today(),
+    "center": video.center,
+}
+sensitive_meta = SensitiveMeta.create_from_dict(default_data)
+```
+
+**What happens internally:**
+1. `perform_save_logic()` is called
+2. Default names are already set (no replacement needed)
+3. Hash is calculated: `sha256("Patient Unknown 1990-01-01 ...")`
+4. Temporary pseudo-entities are created based on this hash
+
+**Result:** Video import can proceed without errors ✅
+
+---
+
+### Phase 2: Update (with Extracted Data)
+
+Later, when real patient data is extracted, it **overwrites** the defaults:
+
+```python
+# video_import.py - _update_sensitive_metadata()
+extracted_metadata = {
+    "patient_first_name": "Max",
+    "patient_last_name": "Mustermann",
+    "patient_dob": date(1985, 3, 15),
+}
+
+sm.update_from_dict(extracted_metadata)
+sm.save()  # ← Triggers perform_save_logic() AGAIN
+```
+
+**What happens internally:**
+1. `update_from_dict()` sets new values:
+   - `instance.patient_first_name = "Max"` (overwrites "Patient")
+   - `instance.patient_last_name = "Mustermann"` (overwrites "Unknown")
+2. `save()` is called → triggers `perform_save_logic()`
+3. Default-setting is **skipped** (names already exist):
+   ```python
+   if not instance.patient_first_name:  # ← FALSE (already "Max")
+       instance.patient_first_name = DEFAULT_UNKNOWN_NAME  # ← NOT EXECUTED
+   ```
+4. Hash is **RECALCULATED** with real data:
+   - `sha256("Max Mustermann 1985-03-15 ...")`
+5. **New pseudo-entities** are created/retrieved based on new hash
+
+**Result:** Correct anonymization with real patient data ✅
+
+---
+
+## Code Flow Diagram
+
+```mermaid
+graph TD
+    A[VideoFile.create_from_file_initialized] --> B[video.initialize]
+    B --> C[get_or_create_sensitive_meta]
+    C --> D[SensitiveMeta.create_from_dict default_data]
+    
+    D --> E[perform_save_logic]
+    E --> F{patient_first_name exists?}
+    F -->|NO| G[Set: patient_first_name = 'Patient']
+    F -->|YES| H[Skip default setting]
+    G --> I[Calculate Hash: sha256 'Patient Unknown...']
+    H --> I
+    I --> J[Create Pseudo-Patient A]
+    
+    J --> K[Frame Cleaning Extracts Data]
+    K --> L[extracted_metadata = 'Max Mustermann...']
+    L --> M[sm.update_from_dict extracted_metadata]
+    
+    M --> N[Set: patient_first_name = 'Max']
+    N --> O[sm.save]
+    O --> P[perform_save_logic AGAIN]
+    
+    P --> Q{patient_first_name exists?}
+    Q -->|YES Already 'Max'| R[Skip default setting]
+    R --> S[RECALCULATE Hash: sha256 'Max Mustermann...']
+    S --> T[Create/Get Pseudo-Patient B]
+    
+    T --> U[Correct Anonymization Complete]
+    
+    style D fill:#ffcccc
+    style E fill:#ffcccc
+    style I fill:#ffcccc
+    style J fill:#ffcccc
+    style M fill:#ccffcc
+    style P fill:#ccffcc
+    style S fill:#ccffcc
+    style T fill:#ccffcc
+```
+
+---
+
+## Key Implementation Details
+
+### 1. Default Name Setting (sensitive_meta_logic.py)
+
+```python
+@transaction.atomic
+def perform_save_logic(instance: "SensitiveMeta") -> "Examiner":
+    """
+    CRITICAL: Set default patient names BEFORE hash calculation.
+    
+    Two-phase approach:
+    - Phase 1: Set defaults if names are missing
+    - Phase 2: Skip if names already exist (real data)
+    """
+    
+    if not instance.patient_first_name:
+        instance.patient_first_name = DEFAULT_UNKNOWN_NAME  # "unknown"
+        
+    if not instance.patient_last_name:
+        instance.patient_last_name = DEFAULT_UNKNOWN_NAME  # "unknown"
+    
+    # Hash is ALWAYS recalculated (uses current field values)
+    instance.patient_hash = calculate_patient_hash(instance)
+    instance.examination_hash = calculate_examination_hash(instance)
+    
+    # Pseudo-entities created based on current hash
+    pseudo_patient = get_or_create_pseudo_patient_logic(instance)
+    # ...
+```
+
+### 2. Initial Creation (video_file.py)
+
+```python
+def get_or_create_sensitive_meta(self) -> "SensitiveMeta":
+    """Creates SensitiveMeta with default patient data."""
+    
+    if self.sensitive_meta is None:
+        default_data = {
+            "patient_first_name": "Patient",
+            "patient_last_name": "Unknown",
+            "patient_dob": date(1990, 1, 1),
+            "examination_date": date.today(),
+            "center": self.center,  # ← Can pass Center object OR "center_name": "hospital_name" string
+        }
+        self.sensitive_meta = SensitiveMeta.create_from_dict(default_data)
+```
+
+**Note:** `create_from_dict()` accepts **both** formats:
+- `"center": center_obj` (Center object)
+- `"center_name": "hospital_name"` (String)
+
+This flexibility allows different parts of the codebase to use whichever format is more convenient.
+
+### 3. Update with Real Data (sensitive_meta_logic.py)
+
+```python
+def update_sensitive_meta_from_dict(instance: "SensitiveMeta", data: Dict[str, Any]) -> "SensitiveMeta":
+    """Updates instance fields and triggers save (hash recalculation)."""
+    
+    # Update instance attributes
+    for k, v in selected_data.items():
+        setattr(instance, k, v)  # Overwrites defaults
+    
+    # Trigger save → perform_save_logic() → hash recalculation
+    instance.save()
+    return instance
+```
+
+### 4. Integration in Video Import (video_import.py)
+
+```python
+def _update_sensitive_metadata(self, extracted_metadata: Dict[str, Any]):
+    """Updates SensitiveMeta with extracted patient data."""
+    
+    sm = self.current_video.sensitive_meta
+    sm.update_from_dict(extracted_metadata)
+    # → Overwrites "Patient Unknown" with "Max Mustermann"
+    # → save() triggers hash recalculation
+    # → New pseudo-entities created
+```
+
+---
+
+## Example Timeline
+
+| Step | Action | patient_first_name | patient_hash | Pseudo-Patient |
+|------|--------|-------------------|--------------|----------------|
+| 1 | `video.initialize()` | - | - | - |
+| 2 | `get_or_create_sensitive_meta()` | "Patient" | `hash("Patient Unknown...")` | Pseudo-Patient A (temp) |
+| 3 | Frame cleaning extraction | "Patient" | `hash("Patient Unknown...")` | Pseudo-Patient A |
+| 4 | `update_from_dict({"patient_first_name": "Max"})` | "Max" | - | - |
+| 5 | `save()` → `perform_save_logic()` | "Max" | `hash("Max Mustermann...")` | Pseudo-Patient B (real) |
+
+**Final state:** Correctly anonymized with real patient data and proper pseudo-entities ✅
+
+---
+
+## Why This Pattern Works
+
+### 1. **Conditional Default Setting**
+```python
+if not instance.patient_first_name:  # Only sets if missing
+    instance.patient_first_name = DEFAULT_UNKNOWN_NAME
+```
+- **First save:** Field is empty → default is set
+- **Second save:** Field has "Max" → condition is False → skipped
+
+### 2. **Hash Always Recalculated**
+```python
+# This runs on EVERY save, regardless of phase
+instance.patient_hash = calculate_patient_hash(instance)
+```
+- **First save:** Hash of "Patient Unknown"
+- **Second save:** Hash of "Max Mustermann"
+
+### 3. **Pseudo-Entities Auto-Update**
+```python
+# get_or_create based on current hash
+pseudo_patient = get_or_create_pseudo_patient_logic(instance)
+```
+- **First save:** Creates/retrieves pseudo-patient with temp hash
+- **Second save:** Creates/retrieves **different** pseudo-patient with real hash
+
+---
+
+## Testing the Pattern
+
+### Test Initial Creation
+```python
+def test_initial_creation_with_defaults():
+    video = VideoFile.create_from_file_initialized(...)
+    sm = video.sensitive_meta
+    
+    assert sm.patient_first_name == "Patient"
+    assert sm.patient_last_name == "Unknown"
+    assert sm.patient_hash is not None  # Hash calculated successfully
+```
+
+### Test Update with Real Data
+```python
+def test_update_with_extracted_data():
+    video = VideoFile.create_from_file_initialized(...)
+    sm = video.sensitive_meta
+    initial_hash = sm.patient_hash
+    
+    # Extract real data
+    sm.update_from_dict({
+        "patient_first_name": "Max",
+        "patient_last_name": "Mustermann",
+        "patient_dob": date(1985, 3, 15),
+    })
+    
+    assert sm.patient_first_name == "Max"  # Updated
+    assert sm.patient_last_name == "Mustermann"  # Updated
+    assert sm.patient_hash != initial_hash  # Hash recalculated
+```
+
+---
+
+## Common Pitfalls (Avoided by This Pattern)
+
+### ❌ **Wrong: Direct `objects.create()` without defaults**
+```python
+# OLD CODE (BROKEN)
+self.sensitive_meta = SensitiveMeta.objects.create(center=self.center)
+# → save() called immediately
+# → perform_save_logic() tries to calculate hash
+# → ERROR: "First name is required to calculate patient hash"
+```
+
+### ❌ **Wrong: `create_from_dict()` without center parameter**
+```python
+# BROKEN - Missing center information
+default_data = {
+    "patient_first_name": "Patient",
+    "patient_last_name": "Unknown",
+}
+self.sensitive_meta = SensitiveMeta.create_from_dict(default_data)
+# → ERROR: "center_name is required in data dictionary"
+```
+
+### ✅ **Right: `create_from_dict()` with defaults and center**
+```python
+# NEW CODE (WORKING) - Option 1: Center object
+default_data = {
+    "patient_first_name": "Patient",
+    "patient_last_name": "Unknown",
+    "patient_dob": date(1990, 1, 1),
+    "center": self.center,  # ← Center object
+}
+self.sensitive_meta = SensitiveMeta.create_from_dict(default_data)
+
+# NEW CODE (WORKING) - Option 2: Center name string
+default_data = {
+    "patient_first_name": "Patient",
+    "patient_last_name": "Unknown",
+    "patient_dob": date(1990, 1, 1),
+    "center_name": "university_hospital_wuerzburg",  # ← String
+}
+self.sensitive_meta = SensitiveMeta.create_from_dict(default_data)
+# → Hash calculation succeeds
+# → Video import proceeds
+# → Real data can be updated later
+```
+
+**Why both formats are supported:**
+- `video_file.py` has access to the Center **object** (`self.center`)
+- `video_import.py` may only have the center **name** string
+- Both are valid and will work correctly
+
+---
+
+## Related Documentation
+
+- **Implementation:** `endoreg_db/models/metadata/sensitive_meta_logic.py`
+- **Video Integration:** `endoreg_db/models/media/video/video_file.py`
+- **Import Service:** `endoreg_db/services/video_import.py`
+- **Hash Calculation:** `endoreg_db/utils/hashs.py`
+
+---
+
+## Summary
+
+The two-phase pattern allows endoreg-db to:
+1. ✅ Create valid SensitiveMeta instances without patient data
+2. ✅ Import videos without blocking on missing information
+3. ✅ Update with real data once available
+4. ✅ Maintain proper anonymization with correct hashes
+5. ✅ Avoid "First name required" errors during hash calculation
+
+**Key principle:** Defaults are temporary placeholders that get replaced by real data, with hash recalculation ensuring correct anonymization.

{endoreg_db-0.8.3.8 → endoreg_db-0.8.6.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: endoreg-db
-Version: 0.8.3.8
+Version: 0.8.6.3
 Summary: EndoReg Db Django App
 Project-URL: Homepage, https://info.coloreg.de
 Project-URL: Repository, https://github.com/wg-lux/endoreg-db
@@ -29,7 +29,6 @@ Requires-Dist: dotenv>=0.9.9
 Requires-Dist: faker>=37.6.0
 Requires-Dist: flake8>=7.3.0
 Requires-Dist: gunicorn>=23.0.0
-Requires-Dist: huggingface-hub>=0.35.3
 Requires-Dist: icecream>=2.1.4
 Requires-Dist: librosa==0.11.0
 Requires-Dist: llvmlite>=0.44.0

endoreg_db-0.8.6.3/conf/example.env

@@ -0,0 +1,4 @@
+STORAGE_DIR=data/
+ENV_PATH=./.env
+KEYCLOAK_CLIENT_SECRET=your_keycloak_client_secret_here
+CENTER_NAME=university_hospital_wuerzburg

{endoreg_db-0.8.3.8 → endoreg_db-0.8.6.3}/devenv.nix

@@ -18,7 +18,7 @@ let
   confDir = "./conf"; # Define confDir here
 
   # Pin to specific Python 3.12 version to match pyproject.toml
-  python = pkgs.python3.withPackages(ps: with ps; [tkinter]); #known devenv issue with python3Packages since python3Full was deprecated
+  python = pkgs.python312; #known devenv issue with python3Packages since python3Full was deprecated
   uvPackage = pkgs.uv;
   
   buildInputs = with pkgs; [

endoreg_db-0.8.6.3/docs/AI_MODEL_CONFIGURATION_GUIDE.md

@@ -0,0 +1,292 @@
+# AI Model Configuration Guide
+
+## Overview
+
+This guide documents the comprehensive solution for EndoReg DB AI model configuration issues, including the integration between YAML-based configuration and automated setup processes.
+
+## Problem Context
+
+The EndoReg DB application was experiencing video processing failures with these specific errors:
+
+1. **"No model metadata found for this model"** - AI models lacked proper metadata configuration
+2. **"KeyError: 'file_path'"** - Video import service was not handling missing file paths defensively
+3. **YAML configuration conflicts** - YAML-loaded models were being overwritten by setup auto-generation
+
+## Solution Architecture
+
+### 1. YAML-Based Configuration System
+
+AI models and their metadata are now configured through structured YAML files:
+
+**Location**: `/endoreg_db/data/ai_model_meta/`
+
+**Example Configuration** (`default_multilabel_classification.yaml`):
+```yaml
+image_multilabel_classification_colonoscopy_default:
+  model: image_multilabel_classification_colonoscopy_default
+  labelset: multilabel_classification_colonoscopy_default
+  version: 1
+  name: image_multilabel_classification_colonoscopy_default
+  description: Default multilabel classification model for colonoscopy video analysis with ImageNet preprocessing
+  input_channels: 3
+  input_height: 224
+  input_width: 224
+  activation: softmax
+  normalization_mean: [0.485, 0.456, 0.406]
+  normalization_std: [0.229, 0.224, 0.225]
+  weights: model_weights/image_multilabel_classification_colonoscopy_default_v1_colo_segmentation_RegNetX800MF_6.ckpt
+  created_at: "2024-01-01T00:00:00Z"
+  updated_at: "2024-01-01T00:00:00Z"
+```
+
+### 2. Management Commands Integration
+
+#### Loading YAML Configurations
+
+**Command**: `python manage.py load_ai_model_data`
+
+This command loads AI models and metadata from YAML files with proper foreign key relationships:
+
+- **Models loaded**: AiModel, ModelMeta, VideoSegmentationLabel, VideoSegmentationLabelSet
+- **Foreign key handling**: Automatically resolves relationships between models and labelsets
+- **ModelMeta integration**: Re-enabled to support complete metadata loading
+
+**Key implementation details**:
+```python
+IMPORT_METADATA = {
+    ModelMeta.__name__: {
+        "dir": AI_MODEL_META_DATA_DIR,
+        "model": ModelMeta,
+        "foreign_keys": ["labelset", "model"],
+        "foreign_key_models": [VideoSegmentationLabelSet, AiModel],
+    },
+    # ... other models
+}
+```
+
+#### Setup Command Enhancement
+
+**Command**: `python manage.py setup_endoreg_db`
+
+Enhanced with smart metadata handling:
+
+- **YAML-aware**: Respects existing YAML-loaded configurations
+- **Auto-generation**: Creates missing metadata only when needed
+- **Weights handling**: Properly assigns FileField weights using `.name` property
+- **Validation**: Comprehensive validation of AI model configurations
+
+**New `--yaml-only` mode**:
+```bash
+python manage.py setup_endoreg_db --yaml-only
+```
+This mode skips auto-generation and relies entirely on YAML configurations.
+
+### 3. Validation System
+
+**Command**: `python manage.py validate_ai_models`
+
+Comprehensive validation system that:
+- Checks all AI models have proper metadata
+- Validates metadata accessibility
+- Reports detailed validation results
+- Can be used for troubleshooting
+
+### 4. Defensive Programming in Video Import
+
+Enhanced `VideoImportService` with robust error handling:
+
+```python
+# Defensive file path access
+file_path = getattr(video_file, 'file_path', None)
+if not file_path:
+    logger.warning(f"Video file {video_file.id} has no file_path, skipping...")
+    continue
+```
+
+## Setup Workflow
+
+### Option 1: YAML-First Setup (Recommended)
+
+1. **Load YAML configurations first**:
+   ```bash
+   python manage.py load_ai_model_data
+   ```
+
+2. **Run setup in YAML-only mode**:
+   ```bash
+   python manage.py setup_endoreg_db --yaml-only
+   ```
+
+3. **Validate configuration**:
+   ```bash
+   python manage.py validate_ai_models
+   ```
+
+### Option 2: Auto-Generation Setup
+
+1. **Run full setup** (will auto-generate missing metadata):
+   ```bash
+   python manage.py setup_endoreg_db
+   ```
+
+2. **Validate configuration**:
+   ```bash
+   python manage.py validate_ai_models
+   ```
+
+## File Structure
+
+```
+endoreg_db/
+├── data/
+│   └── ai_model_meta/
+│       └── default_multilabel_classification.yaml
+├── management/
+│   └── commands/
+│       ├── load_ai_model_data.py          # YAML loader
+│       ├── setup_endoreg_db.py            # Enhanced setup
+│       └── validate_ai_models.py          # Validation
+├── models/
+│   ├── administration/ai/
+│   │   └── ai_model.py                    # AiModel model
+│   └── metadata/
+│       └── model_meta.py                  # ModelMeta model
+└── services/
+    └── video_import.py                    # Enhanced with defensive coding
+```
+
+## Key Features
+
+### 1. Smart Weight Assignment
+
+The setup command now properly handles Django FileField assignments:
+```python
+# Uses .name property instead of direct assignment
+if weights_file.exists():
+    metadata.weights.name = f"model_weights/{weights_file.name}"
+    metadata.save()
+```
+
+### 2. Foreign Key Resolution
+
+YAML loading properly resolves foreign key relationships:
+- ModelMeta → AiModel (model field)
+- ModelMeta → VideoSegmentationLabelSet (labelset field)
+
+### 3. Conflict Prevention
+
+The `--yaml-only` flag prevents setup from overwriting YAML-defined configurations:
+```python
+if yaml_only:
+    self.stdout.write(f"  ⚠️  YAML-only mode: Skipping auto-generation for {model.name}")
+    continue
+```
+
+## Validation and Testing
+
+### Model Metadata Access Test
+
+```python
+from endoreg_db.models.administration.ai.ai_model import AiModel
+
+for model in AiModel.objects.all():
+    metadata = model.get_latest_version()
+    print(f"Model: {model.name}")
+    print(f"Metadata: {metadata}")
+    print(f"Weights: {metadata.weights.name}")
+```
+
+### Service Integration Test
+
+```python
+from endoreg_db.services.video_import import VideoImportService
+
+service = VideoImportService()  # Should initialize without errors
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **"No model metadata found"**
+   - Run: `python manage.py validate_ai_models`
+   - If missing, run: `python manage.py setup_endoreg_db`
+
+2. **"ValueError: The 'weights' attribute has no file associated"**
+   - Ensure weights files exist in `storage/model_weights/`
+   - Re-run setup to reassign weights properly
+
+3. **YAML loading errors**
+   - Check foreign key references in YAML match existing objects
+   - Ensure proper model loading order (dependencies first)
+
+### Validation Commands
+
+```bash
+# Check AI model configuration
+python manage.py validate_ai_models
+
+# Reload YAML configurations
+python manage.py load_ai_model_data --verbosity=2
+
+# Force recreate metadata
+python manage.py setup_endoreg_db --force-recreate
+```
+
+## Migration Guide
+
+### From Manual Setup to YAML Configuration
+
+1. **Backup existing data**:
+   ```bash
+   python manage.py dumpdata endoreg_db.AiModel endoreg_db.ModelMeta > ai_models_backup.json
+   ```
+
+2. **Clear existing metadata** (optional):
+   ```bash
+   python manage.py shell -c "from endoreg_db.models.metadata.model_meta import ModelMeta; ModelMeta.objects.all().delete()"
+   ```
+
+3. **Load YAML configurations**:
+   ```bash
+   python manage.py load_ai_model_data
+   ```
+
+4. **Validate new setup**:
+   ```bash
+   python manage.py validate_ai_models
+   ```
+
+## Best Practices
+
+1. **Always validate after changes**: Use `validate_ai_models` command
+2. **Use YAML-first approach**: Define configurations in YAML for consistency
+3. **Test service integration**: Ensure VideoImportService can access metadata
+4. **Monitor weights files**: Ensure model weight files exist and are accessible
+5. **Version control YAML**: Keep AI model configurations in version control
+
+## Performance Considerations
+
+- YAML loading is performed once during setup
+- Model metadata is cached in database
+- FileField weights are stored as paths, not loaded into memory
+- Validation commands are designed for development/debugging use
+
+## Security Notes
+
+- Model weight files should be stored securely
+- YAML configurations may contain sensitive model information
+- Ensure proper access controls on storage directories
+
+---
+
+## Summary
+
+This configuration system provides:
+- ✅ Robust AI model metadata management
+- ✅ YAML-based configuration with database integration
+- ✅ Defensive error handling in video processing
+- ✅ Comprehensive validation and troubleshooting tools
+- ✅ Flexible setup workflows for different deployment scenarios
+
+The integration successfully resolves the original video processing failures while providing a maintainable and extensible configuration system.