endoreg-db 0.8.4.2__tar.gz → 0.8.4.3__tar.gz

{endoreg_db-0.8.4.2 → endoreg_db-0.8.4.3}/CONFIGURATION_GUIDE.md

@@ -65,6 +65,33 @@ The command automatically detects your cache configuration:
 
 Use `--skip-ai-setup` if AI video processing features are not needed, or `--force-recreate` to recreate AI metadata.
 
+### AI Model Validation and Troubleshooting
+
+If you encounter "No model metadata found for this model" errors during video processing, use the dedicated validation command:
+
+```bash
+# Check AI model metadata status without making changes
+python manage.py validate_ai_models --dry-run
+
+# Fix AI model metadata issues automatically  
+python manage.py validate_ai_models
+
+# View detailed help
+python manage.py validate_ai_models --help
+```
+
+The validation command automatically:
+- Checks all AI models for proper metadata configuration
+- Creates missing metadata with sensible defaults (ImageNet normalization, etc.)
+- Sets active metadata for models that don't have it assigned
+- Validates that all models can access their latest versions
+
+Common scenarios this fixes:
+- **"No model metadata found"** errors in pipe_1 video processing
+- AI models exist but have no `active_meta` set
+- Models have metadata versions but none are marked as active
+- Missing ModelMeta records for classification/segmentation models
+
 This repo standalone (local):
 - Development server: DJANGO_SETTINGS_MODULE=config.settings.dev python manage.py runserver
 - Tests (persistent test DB): pytest --reuse-db --create-db

{endoreg_db-0.8.4.2 → endoreg_db-0.8.4.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: endoreg-db
-Version: 0.8.4.2
+Version: 0.8.4.3
 Summary: EndoReg Db Django App
 Project-URL: Homepage, https://info.coloreg.de
 Project-URL: Repository, https://github.com/wg-lux/endoreg-db

endoreg_db-0.8.4.3/docs/AI_MODEL_CONFIGURATION_GUIDE.md

@@ -0,0 +1,292 @@
+# AI Model Configuration Guide
+
+## Overview
+
+This guide documents the comprehensive solution for EndoReg DB AI model configuration issues, including the integration between YAML-based configuration and automated setup processes.
+
+## Problem Context
+
+The EndoReg DB application was experiencing video processing failures with these specific errors:
+
+1. **"No model metadata found for this model"** - AI models lacked proper metadata configuration
+2. **"KeyError: 'file_path'"** - Video import service was not handling missing file paths defensively
+3. **YAML configuration conflicts** - YAML-loaded models were being overwritten by setup auto-generation
+
+## Solution Architecture
+
+### 1. YAML-Based Configuration System
+
+AI models and their metadata are now configured through structured YAML files:
+
+**Location**: `/endoreg_db/data/ai_model_meta/`
+
+**Example Configuration** (`default_multilabel_classification.yaml`):
+```yaml
+image_multilabel_classification_colonoscopy_default:
+  model: image_multilabel_classification_colonoscopy_default
+  labelset: multilabel_classification_colonoscopy_default
+  version: 1
+  name: image_multilabel_classification_colonoscopy_default
+  description: Default multilabel classification model for colonoscopy video analysis with ImageNet preprocessing
+  input_channels: 3
+  input_height: 224
+  input_width: 224
+  activation: softmax
+  normalization_mean: [0.485, 0.456, 0.406]
+  normalization_std: [0.229, 0.224, 0.225]
+  weights: model_weights/image_multilabel_classification_colonoscopy_default_v1_colo_segmentation_RegNetX800MF_6.ckpt
+  created_at: "2024-01-01T00:00:00Z"
+  updated_at: "2024-01-01T00:00:00Z"
+```
+
+### 2. Management Commands Integration
+
+#### Loading YAML Configurations
+
+**Command**: `python manage.py load_ai_model_data`
+
+This command loads AI models and metadata from YAML files with proper foreign key relationships:
+
+- **Models loaded**: AiModel, ModelMeta, VideoSegmentationLabel, VideoSegmentationLabelSet
+- **Foreign key handling**: Automatically resolves relationships between models and labelsets
+- **ModelMeta integration**: Re-enabled to support complete metadata loading
+
+**Key implementation details**:
+```python
+IMPORT_METADATA = {
+    ModelMeta.__name__: {
+        "dir": AI_MODEL_META_DATA_DIR,
+        "model": ModelMeta,
+        "foreign_keys": ["labelset", "model"],
+        "foreign_key_models": [VideoSegmentationLabelSet, AiModel],
+    },
+    # ... other models
+}
+```
+
+#### Setup Command Enhancement
+
+**Command**: `python manage.py setup_endoreg_db`
+
+Enhanced with smart metadata handling:
+
+- **YAML-aware**: Respects existing YAML-loaded configurations
+- **Auto-generation**: Creates missing metadata only when needed
+- **Weights handling**: Properly assigns FileField weights using `.name` property
+- **Validation**: Comprehensive validation of AI model configurations
+
+**New `--yaml-only` mode**:
+```bash
+python manage.py setup_endoreg_db --yaml-only
+```
+This mode skips auto-generation and relies entirely on YAML configurations.
+
+### 3. Validation System
+
+**Command**: `python manage.py validate_ai_models`
+
+Comprehensive validation system that:
+- Checks all AI models have proper metadata
+- Validates metadata accessibility
+- Reports detailed validation results
+- Can be used for troubleshooting
+
+### 4. Defensive Programming in Video Import
+
+Enhanced `VideoImportService` with robust error handling:
+
+```python
+# Defensive file path access
+file_path = getattr(video_file, 'file_path', None)
+if not file_path:
+    logger.warning(f"Video file {video_file.id} has no file_path, skipping...")
+    continue
+```
+
+## Setup Workflow
+
+### Option 1: YAML-First Setup (Recommended)
+
+1. **Load YAML configurations first**:
+   ```bash
+   python manage.py load_ai_model_data
+   ```
+
+2. **Run setup in YAML-only mode**:
+   ```bash
+   python manage.py setup_endoreg_db --yaml-only
+   ```
+
+3. **Validate configuration**:
+   ```bash
+   python manage.py validate_ai_models
+   ```
+
+### Option 2: Auto-Generation Setup
+
+1. **Run full setup** (will auto-generate missing metadata):
+   ```bash
+   python manage.py setup_endoreg_db
+   ```
+
+2. **Validate configuration**:
+   ```bash
+   python manage.py validate_ai_models
+   ```
+
+## File Structure
+
+```
+endoreg_db/
+├── data/
+│   └── ai_model_meta/
+│       └── default_multilabel_classification.yaml
+├── management/
+│   └── commands/
+│       ├── load_ai_model_data.py          # YAML loader
+│       ├── setup_endoreg_db.py            # Enhanced setup
+│       └── validate_ai_models.py          # Validation
+├── models/
+│   ├── administration/ai/
+│   │   └── ai_model.py                    # AiModel model
+│   └── metadata/
+│       └── model_meta.py                  # ModelMeta model
+└── services/
+    └── video_import.py                    # Enhanced with defensive coding
+```
+
+## Key Features
+
+### 1. Smart Weight Assignment
+
+The setup command now properly handles Django FileField assignments:
+```python
+# Uses .name property instead of direct assignment
+if weights_file.exists():
+    metadata.weights.name = f"model_weights/{weights_file.name}"
+    metadata.save()
+```
+
+### 2. Foreign Key Resolution
+
+YAML loading properly resolves foreign key relationships:
+- ModelMeta → AiModel (model field)
+- ModelMeta → VideoSegmentationLabelSet (labelset field)
+
+### 3. Conflict Prevention
+
+The `--yaml-only` flag prevents setup from overwriting YAML-defined configurations:
+```python
+if yaml_only:
+    self.stdout.write(f"  ⚠️  YAML-only mode: Skipping auto-generation for {model.name}")
+    continue
+```
+
+## Validation and Testing
+
+### Model Metadata Access Test
+
+```python
+from endoreg_db.models.administration.ai.ai_model import AiModel
+
+for model in AiModel.objects.all():
+    metadata = model.get_latest_version()
+    print(f"Model: {model.name}")
+    print(f"Metadata: {metadata}")
+    print(f"Weights: {metadata.weights.name}")
+```
+
+### Service Integration Test
+
+```python
+from endoreg_db.services.video_import import VideoImportService
+
+service = VideoImportService()  # Should initialize without errors
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **"No model metadata found"**
+   - Run: `python manage.py validate_ai_models`
+   - If missing, run: `python manage.py setup_endoreg_db`
+
+2. **"ValueError: The 'weights' attribute has no file associated"**
+   - Ensure weights files exist in `storage/model_weights/`
+   - Re-run setup to reassign weights properly
+
+3. **YAML loading errors**
+   - Check foreign key references in YAML match existing objects
+   - Ensure proper model loading order (dependencies first)
+
+### Validation Commands
+
+```bash
+# Check AI model configuration
+python manage.py validate_ai_models
+
+# Reload YAML configurations
+python manage.py load_ai_model_data --verbosity=2
+
+# Force recreate metadata
+python manage.py setup_endoreg_db --force-recreate
+```
+
+## Migration Guide
+
+### From Manual Setup to YAML Configuration
+
+1. **Backup existing data**:
+   ```bash
+   python manage.py dumpdata endoreg_db.AiModel endoreg_db.ModelMeta > ai_models_backup.json
+   ```
+
+2. **Clear existing metadata** (optional):
+   ```bash
+   python manage.py shell -c "from endoreg_db.models.metadata.model_meta import ModelMeta; ModelMeta.objects.all().delete()"
+   ```
+
+3. **Load YAML configurations**:
+   ```bash
+   python manage.py load_ai_model_data
+   ```
+
+4. **Validate new setup**:
+   ```bash
+   python manage.py validate_ai_models
+   ```
+
+## Best Practices
+
+1. **Always validate after changes**: Use `validate_ai_models` command
+2. **Use YAML-first approach**: Define configurations in YAML for consistency
+3. **Test service integration**: Ensure VideoImportService can access metadata
+4. **Monitor weights files**: Ensure model weight files exist and are accessible
+5. **Version control YAML**: Keep AI model configurations in version control
+
+## Performance Considerations
+
+- YAML loading is performed once during setup
+- Model metadata is cached in database
+- FileField weights are stored as paths, not loaded into memory
+- Validation commands are designed for development/debugging use
+
+## Security Notes
+
+- Model weight files should be stored securely
+- YAML configurations may contain sensitive model information
+- Ensure proper access controls on storage directories
+
+---
+
+## Summary
+
+This configuration system provides:
+- ✅ Robust AI model metadata management
+- ✅ YAML-based configuration with database integration
+- ✅ Defensive error handling in video processing
+- ✅ Comprehensive validation and troubleshooting tools
+- ✅ Flexible setup workflows for different deployment scenarios
+
+The integration successfully resolves the original video processing failures while providing a maintainable and extensible configuration system.

endoreg_db-0.8.4.3/docs/CONFIGURABLE_SETUP_GUIDE.md

@@ -0,0 +1,305 @@
+# Configurable AI Model Setup Guide
+
+## Overview
+
+The EndoReg DB setup system is now fully configurable, eliminating hardcoded model names and making it easy to adapt for different model configurations. This guide explains how to customize the setup for your specific AI models.
+
+## Configuration Files
+
+### 1. Setup Configuration (`endoreg_db/data/setup_config.yaml`)
+
+This is the main configuration file that controls setup behavior:
+
+```yaml
+# Primary models used during setup
+default_models:
+  primary_classification_model: "image_multilabel_classification_colonoscopy_default"
+  primary_labelset: "multilabel_classification_colonoscopy_default"
+
+# HuggingFace fallback configuration
+huggingface_fallback:
+  enabled: true
+  repo_id: "wg-lux/colo_segmentation_RegNetX800MF_base"
+  filename: "colo_segmentation_RegNetX800MF_base.ckpt"
+  labelset_name: "multilabel_classification_colonoscopy_default"
+
+# Weight file search patterns (supports wildcards)
+weights_search_patterns:
+  - "colo_segmentation_RegNetX800MF_*.ckpt"
+  - "image_multilabel_classification_colonoscopy_default_*.ckpt"
+  - "*_colonoscopy_*.ckpt"
+  
+# Search directories for local weights
+weights_search_dirs:
+  - "tests/assets"
+  - "assets" 
+  - "data/storage/model_weights"
+  - "${STORAGE_DIR}/model_weights"  # Environment variable substitution
+
+# Default values for auto-generated metadata
+auto_generation_defaults:
+  activation: "sigmoid"
+  mean: "0.485,0.456,0.406"
+  std: "0.229,0.224,0.225"
+  size_x: 224
+  size_y: 224
+  axes: "CHW"
+  batchsize: 32
+  num_workers: 4
+```
+
+### 2. Enhanced Model YAML (`endoreg_db/data/ai_model_meta/*.yaml`)
+
+Model metadata files can now include setup-specific configuration:
+
+```yaml
+- model: endoreg_db.model_meta
+  fields:
+    name: "your_custom_model_name"
+    version: "1"
+    model: "your_custom_model_name"
+    labelset: "your_custom_labelset"
+    # ... other metadata fields
+  
+  # Setup-specific configuration for this model
+  setup_config:
+    is_primary_model: true  # Mark as primary model for setup
+    weight_filenames:       # Specific weight file patterns
+      - "your_model_weights_*.ckpt"
+      - "custom_model_v1_*.pth"
+    huggingface_fallback:   # HF configuration for this model
+      repo_id: "your-org/your-model-repo"
+      filename: "model_weights.ckpt"
+```
+
+## Configuration API
+
+### SetupConfig Class
+
+The `endoreg_db.utils.setup_config.SetupConfig` class provides programmatic access to configuration:
+
+```python
+from endoreg_db.utils.setup_config import setup_config
+
+# Get configured model names
+primary_model = setup_config.get_primary_model_name()
+primary_labelset = setup_config.get_primary_labelset_name()
+
+# Get HuggingFace configuration
+hf_config = setup_config.get_huggingface_config()
+
+# Find weight files using configured patterns
+weight_files = setup_config.find_model_weights_files()
+
+# Get auto-generation defaults
+defaults = setup_config.get_auto_generation_defaults()
+```
+
+## Customization Examples
+
+### Example 1: Different Model Type
+
+To use a segmentation model instead of classification:
+
+```yaml
+# setup_config.yaml
+default_models:
+  primary_classification_model: "video_segmentation_transformer"
+  primary_labelset: "segmentation_labels_v2"
+
+weights_search_patterns:
+  - "segmentation_transformer_*.pth"
+  - "video_seg_*.ckpt"
+
+auto_generation_defaults:
+  activation: "softmax"
+  mean: "0.5,0.5,0.5"  # Different normalization
+  std: "0.5,0.5,0.5"
+  size_x: 512  # Higher resolution
+  size_y: 512
+```
+
+### Example 2: Multiple Model Support
+
+For setups with multiple models:
+
+```yaml
+# setup_config.yaml - can reference any defined model as primary
+default_models:
+  primary_classification_model: "advanced_colonoscopy_classifier"
+  primary_labelset: "advanced_colonoscopy_labels"
+
+weights_search_patterns:
+  - "*_colonoscopy_*.ckpt"  # Catches all colonoscopy models
+  - "*_classification_*.pth"
+  - "model_*.weights"
+```
+
+### Example 3: Custom HuggingFace Repository
+
+```yaml
+# setup_config.yaml
+huggingface_fallback:
+  enabled: true
+  repo_id: "your-organization/custom-endoscopy-model"
+  filename: "best_model.ckpt"
+  labelset_name: "custom_endoscopy_labels"
+```
+
+### Example 4: Environment-Specific Configuration
+
+```yaml
+# setup_config.yaml
+weights_search_dirs:
+  - "${MODEL_CACHE_DIR}/weights"      # Custom environment variable
+  - "${HOME}/.endoreg/models"         # User-specific location
+  - "/shared/models/endoreg"          # Shared network location
+  - "data/storage/model_weights"      # Default fallback
+```
+
+## Migration from Hardcoded Setup
+
+### Step 1: Identify Current Models
+
+```bash
+# Check what models are currently configured
+python manage.py shell -c "
+from endoreg_db.models import AiModel
+for model in AiModel.objects.all():
+    print(f'Model: {model.name}')
+"
+```
+
+### Step 2: Create Setup Configuration
+
+Create `endoreg_db/data/setup_config.yaml` with your model names:
+
+```yaml
+default_models:
+  primary_classification_model: "your_existing_model_name"
+  primary_labelset: "your_existing_labelset"
+```
+
+### Step 3: Test Configuration
+
+```bash
+python manage.py shell -c "
+from endoreg_db.utils.setup_config import setup_config
+print('Primary model:', setup_config.get_primary_model_name())
+print('Found weights:', setup_config.find_model_weights_files())
+"
+```
+
+### Step 4: Run Setup
+
+```bash
+python manage.py setup_endoreg_db
+```
+
+## Advanced Features
+
+### 1. Dynamic Weight File Discovery
+
+The system automatically discovers weight files using glob patterns:
+
+```python
+# Finds files like:
+# - colo_segmentation_RegNetX800MF_v1.ckpt
+# - colo_segmentation_RegNetX800MF_best.ckpt
+# - image_multilabel_classification_colonoscopy_default_final.ckpt
+```
+
+### 2. Environment Variable Substitution
+
+Use environment variables in paths:
+
+```yaml
+weights_search_dirs:
+  - "${ENDOREG_MODEL_DIR}/weights"
+  - "${STORAGE_DIR}/custom_models"
+```
+
+### 3. Model-Specific Configuration Override
+
+Override global settings per model in the model YAML:
+
+```yaml
+setup_config:
+  weight_filenames:
+    - "special_model_*.ckpt"  # Override global patterns
+  huggingface_fallback:
+    repo_id: "different-org/special-model"  # Override global HF config
+```
+
+### 4. Fallback Chain
+
+The system follows this priority order:
+1. **Model-specific weight patterns** (from model YAML)
+2. **Global weight patterns** (from setup_config.yaml)  
+3. **HuggingFace download** (if enabled and no local files found)
+
+## Troubleshooting
+
+### Configuration Not Loading
+
+```bash
+# Check if configuration file exists and is valid
+python manage.py shell -c "
+from endoreg_db.utils.setup_config import SetupConfig
+config = SetupConfig()
+print('Config loaded:', config._config)
+"
+```
+
+### Weights Not Found
+
+```bash
+# Check weight discovery
+python manage.py shell -c "
+from endoreg_db.utils.setup_config import setup_config
+print('Search patterns:', setup_config.get_weights_search_patterns())
+print('Search dirs:', setup_config.get_weights_search_dirs())
+print('Found files:', setup_config.find_model_weights_files())
+"
+```
+
+### HuggingFace Issues
+
+```bash
+# Test HuggingFace configuration
+python manage.py shell -c "
+from endoreg_db.utils.setup_config import setup_config
+hf_config = setup_config.get_huggingface_config()
+print('HF enabled:', hf_config.get('enabled'))
+print('HF repo:', hf_config.get('repo_id'))
+"
+```
+
+## Benefits
+
+### ✅ **Fully Configurable**
+- No more hardcoded model names
+- Easy to adapt for new models
+- Environment-specific configurations
+
+### ✅ **Flexible Weight Discovery**
+- Supports wildcards and patterns
+- Multiple search directories
+- Environment variable substitution
+
+### ✅ **Robust Fallbacks**
+- Local files → HuggingFace → Graceful failure
+- Configurable at multiple levels
+- Smart priority handling
+
+### ✅ **Easy Migration**
+- Backward compatible with existing setups
+- Gradual migration path
+- Clear configuration structure
+
+### ✅ **Development Friendly**
+- Different configs for dev/test/prod
+- Easy to add new models
+- Clear separation of concerns
+
+This configurable approach makes the EndoReg DB setup system much more flexible and maintainable!

endoreg_db-0.8.4.3/endoreg_db/data/ai_model_meta/default_multilabel_classification.yaml

@@ -0,0 +1,27 @@
+# Model metadata configuration with setup hints
+- model: endoreg_db.model_meta
+  fields:
+    name: "image_multilabel_classification_colonoscopy_default"
+    version: "1"
+    model: "image_multilabel_classification_colonoscopy_default"
+    labelset: "multilabel_classification_colonoscopy_default"
+    activation: "sigmoid"
+    mean: "0.485,0.456,0.406" # ImageNet defaults
+    std: "0.229,0.224,0.225" # ImageNet defaults
+    size_x: 224
+    size_y: 224
+    axes: "CHW"
+    batchsize: 32
+    num_workers: 4
+    description: "Default colonoscopy image multilabel classification model from YAML"
+    # Note: weights field intentionally left empty - will be set by setup command or model meta logic
+
+  # Setup configuration hints for this model
+  setup_config:
+    is_primary_model: true # Mark this as the primary model for setup
+    weight_filenames: # Specific weight file patterns for this model
+      - "colo_segmentation_RegNetX800MF_6.ckpt"
+      - "image_multilabel_classification_colonoscopy_default_v1_*.ckpt"
+    huggingface_fallback: # HF configuration for this model
+      repo_id: "wg-lux/colo_segmentation_RegNetX800MF_base"
+      filename: "colo_segmentation_RegNetX800MF_base.ckpt"

endoreg_db-0.8.4.3/endoreg_db/data/setup_config.yaml

@@ -0,0 +1,38 @@
+# EndoReg DB Setup Configuration
+# This file defines which models should be used for setup and their fallback configurations
+
+default_models:
+  primary_classification_model: "image_multilabel_classification_colonoscopy_default"
+  primary_labelset: "multilabel_classification_colonoscopy_default"
+
+# HuggingFace fallback configuration
+huggingface_fallback:
+  enabled: true
+  repo_id: "wg-lux/colo_segmentation_RegNetX800MF_base"
+  filename: "colo_segmentation_RegNetX800MF_base.ckpt"
+  labelset_name: "multilabel_classification_colonoscopy_default"
+
+# Local weights search patterns
+weights_search_patterns:
+  # Primary model weight filename patterns (supports wildcards)
+  - "colo_segmentation_RegNetX800MF_*.ckpt"
+  - "image_multilabel_classification_colonoscopy_default_*.ckpt"
+  - "*_colonoscopy_*.ckpt"
+
+# Search directories for local weights (relative to project root)
+weights_search_dirs:
+  - "tests/assets"
+  - "assets"
+  - "data/storage/model_weights"
+  - "${STORAGE_DIR}/model_weights" # Environment variable substitution
+
+# Auto-generation defaults when creating metadata
+auto_generation_defaults:
+  activation: "sigmoid"
+  mean: "0.485,0.456,0.406" # ImageNet defaults
+  std: "0.229,0.224,0.225" # ImageNet defaults
+  size_x: 224
+  size_y: 224
+  axes: "CHW"
+  batchsize: 32
+  num_workers: 4