endoreg-db 0.8.3.2__tar.gz → 0.8.3.3__tar.gz

endoreg_db-0.8.3.3/AI_MODEL_SETUP.md

@@ -0,0 +1,84 @@
+## AI Model Metadata Setup Commands
+
+This section documents the complete setup process for AI model metadata in EndoReg DB. These commands must be run in sequence to properly initialize the AI model system for video processing.
+
+### 1. Load AI Models
+```bash
+uv run python manage.py load_ai_model_data
+```
+**Purpose**: Loads the base AI model definitions into the database.
+- Creates `AiModel` instances with model names, types, and descriptions
+- Required before creating model metadata
+- **Output**: "Loaded X AI models"
+
+### 2. Load AI Model Labels
+```bash
+uv run python manage.py load_ai_model_label_data
+```
+**Purpose**: Loads label sets and labels associated with AI models.
+- Creates `LabelSet` and `Label` instances for model outputs
+- Links labels to AI models for classification tasks
+- **Output**: "Loaded X model labels"
+
+### 3. Create Django Cache Table
+```bash
+uv run python manage.py createcachetable
+```
+**Purpose**: Creates the database table for Django's caching framework.
+- Required for API polling and caching functionality
+- Prevents `OperationalError` when accessing cached data
+- **Output**: "Cache table for database cache created."
+
+### 4. Create Model Metadata
+```bash
+uv run python manage.py create_multilabel_model_meta \
+    --model_name image_multilabel_classification_colonoscopy_default \
+    --model_meta_version 1 \
+    --image_classification_labelset_name multilabel_classification_colonoscopy_default
+```
+**Purpose**: Creates `ModelMeta` instances with model weights and configuration.
+- Copies model weights file to storage directory
+- Sets up model parameters (activation, normalization, input size)
+- Links metadata to AI models and label sets
+- **Output**: "Created new ModelMeta: [model_name] (v1)"
+
+
+### Verification Commands
+
+After running the setup commands, verify everything is working:
+
+```bash
+# Check AI models and metadata
+uv run python manage.py shell -c "
+from endoreg_db.models import AiModel, ModelMeta
+from endoreg_db.helpers.default_objects import get_latest_segmentation_model
+
+print('AI Models:', AiModel.objects.count())
+print('Model Metadata:', ModelMeta.objects.count())
+
+model_meta = get_latest_segmentation_model()
+print('Latest model meta:', model_meta)
+print('Weights exist:', model_meta.weights and model_meta.weights.path)
+"
+```
+
+### Troubleshooting
+
+- **"Model file not found"**: Ensure you're running commands from the correct project directory (`/home/admin/endoreg-db`)
+- **"No model metadata found"**: Run steps 1-4 in sequence
+- **"Processor is not of type EndoscopyProcessor"**: Apply the import fix in step 5
+- **Cache table errors**: Run step 3 to create the Django cache table
+
+### Alternative: Hugging Face Setup
+
+If local model files are not available, use the Hugging Face download command:
+
+```bash
+uv run python manage.py create_model_meta_from_huggingface \
+    --model_name image_multilabel_classification_colonoscopy_default \
+    --labelset_name multilabel_classification_colonoscopy_default \
+    --meta_version 1
+```
+
+**Note**: Requires the model to exist on Hugging Face Hub.</content>
+<parameter name="filePath">/home/admin/endoreg-db/AI_MODEL_SETUP.md

{endoreg_db-0.8.3.2 → endoreg_db-0.8.3.3}/CONFIGURATION_GUIDE.md

@@ -50,6 +50,20 @@ As an embedded app in a host project:
 - Add 'endoreg_db' to INSTALLED_APPS in the host settings.
 - Define STORAGE_DIR in the host environment.
 - Run migrations in the host project (this app contributes its migrations).
+- Run the complete setup command: `python manage.py setup_endoreg_db`
+
+The `setup_endoreg_db` command performs all necessary initialization:
+1. Loads base database data (medical vocabularies, centers, etc.)
+2. Creates Django cache table for API functionality (only when using database-backed caching)
+3. Sets up AI models and labels (unless --skip-ai-setup is used)
+4. Creates AI model metadata with weights
+5. Verifies the setup was successful
+
+The command automatically detects your cache configuration:
+- For LocMemCache (default): Skips cache table creation
+- For database caching: Creates the required cache tables
+
+Use `--skip-ai-setup` if AI video processing features are not needed, or `--force-recreate` to recreate AI metadata.
 
 This repo standalone (local):
 - Development server: DJANGO_SETTINGS_MODULE=config.settings.dev python manage.py runserver
@@ -68,12 +82,39 @@ CI tips
 - Replace imports of prod_settings, dev/dev_settings.py, tests/test_settings.py with config.settings.prod/dev/test.
 - Update scripts: scripts/django_setup.py, check_video_files.py, etc., to default to config.settings.dev/test.
 
-## Troubleshooting
-- If numpy/opencv import errors appear in VS Code discovery, ensure the editor inherits direnv env (Direnv extension) or use a pytest wrapper.
-- If tests fail with missing tables, recreate test DB with: pytest --reuse-db --create-db.
-- Run migrations and seed base data for tests when needed:
-  - DJANGO_SETTINGS_MODULE=config.settings.test python manage.py migrate
-  - DJANGO_SETTINGS_MODULE=config.settings.test python manage.py load_base_db_data
+## AI Model Setup (for video processing features)
+
+When using EndoReg DB's AI-powered video processing features, ensure model weights are available:
+
+### Model Weights Location
+The system looks for model weights in these locations (in order of preference):
+1. `STORAGE_DIR/model_weights/` (recommended for production)
+2. `tests/assets/` (for development/testing)
+3. `assets/` (fallback location)
+
+### Required Model Files
+For colonoscopy video processing, the following model file is required:
+- `colo_segmentation_RegNetX800MF_6.ckpt` - Multilabel classification model for colonoscopy
+
+### Automatic Setup
+The `setup_endoreg_db` command automatically:
+- Loads AI model definitions and labels
+- Creates model metadata with weights
+- Sets up the default AI model for video processing
+
+### Manual Setup (if needed)
+If automatic setup fails, run these commands individually:
+```bash
+python manage.py load_ai_model_data
+python manage.py load_ai_model_label_data
+python manage.py createcachetable
+python manage.py create_multilabel_model_meta --model_name image_multilabel_classification_colonoscopy_default --model_meta_version 1 --image_classification_labelset_name multilabel_classification_colonoscopy_default
+```
+
+### Troubleshooting AI Setup
+- **"Model file not found"**: Ensure model weights are in one of the expected locations
+- **"No model metadata found"**: Run the setup commands or use `--force-recreate`
+- **Import errors**: Check that the `EndoscopyProcessor` import fix is applied in `video_import.py`
 
 ## Production checklist
 - Set DJANGO_SECRET_KEY to a strong random value (never commit). 

{endoreg_db-0.8.3.2 → endoreg_db-0.8.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: endoreg-db
-Version: 0.8.3.2
+Version: 0.8.3.3
 Summary: EndoReg Db Django App
 Project-URL: Homepage, https://info.coloreg.de
 Project-URL: Repository, https://github.com/wg-lux/endoreg-db

endoreg_db-0.8.3.3/endoreg_db/management/commands/create_model_meta_from_huggingface.py

@@ -0,0 +1,116 @@
+"""
+Django management command to create ModelMeta from Hugging Face model.
+"""
+
+from pathlib import Path
+
+from django.core.files.base import ContentFile
+from django.core.management.base import BaseCommand
+from huggingface_hub import hf_hub_download
+
+from endoreg_db.models import AiModel, LabelSet, ModelMeta
+
+
+class Command(BaseCommand):
+    help = "Create ModelMeta by downloading model from Hugging Face"
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            "--model_id",
+            type=str,
+            default="wg-lux/colo_segmentation_RegNetX800MF_base",
+            help="Hugging Face model ID",
+        )
+        parser.add_argument(
+            "--model_name",
+            type=str,
+            default="image_multilabel_classification_colonoscopy_default",
+            help="Name for the AI model",
+        )
+        parser.add_argument(
+            "--labelset_name",
+            type=str,
+            default="multilabel_classification_colonoscopy_default",
+            help="Name of the labelset",
+        )
+        parser.add_argument(
+            "--meta_version",
+            type=str,
+            default="1",
+            help="Version for the model meta",
+        )
+
+    def handle(self, *args, **options):
+        model_id = options["model_id"]
+        model_name = options["model_name"]
+        labelset_name = options["labelset_name"]
+        version = options["meta_version"]
+
+        self.stdout.write(f"Downloading model {model_id} from Hugging Face...")
+
+        try:
+            # Download the model weights
+            weights_path = hf_hub_download(
+                repo_id=model_id,
+                filename="colo_segmentation_RegNetX800MF_base.ckpt",
+                local_dir="/tmp",
+            )
+            self.stdout.write(f"Downloaded weights to: {weights_path}")
+
+            # Get or create AI model
+            ai_model, created = AiModel.objects.get_or_create(
+                name=model_name, defaults={"description": f"Model from {model_id}"}
+            )
+            if created:
+                self.stdout.write(f"Created AI model: {ai_model.name}")
+
+            # Get labelset
+            try:
+                labelset = LabelSet.objects.get(name=labelset_name)
+            except LabelSet.DoesNotExist:
+                self.stdout.write(
+                    self.style.ERROR(f"LabelSet '{labelset_name}' not found")
+                )
+                return
+
+            # Create ModelMeta
+            model_meta, created = ModelMeta.objects.get_or_create(
+                name=model_name,
+                model=ai_model,
+                version=version,
+                defaults={
+                    "labelset": labelset,
+                    "activation": "sigmoid",
+                    "mean": "0.45211223,0.27139644,0.19264949",
+                    "std": "0.31418097,0.21088019,0.16059452",
+                    "size_x": 716,
+                    "size_y": 716,
+                    "axes": "2,0,1",
+                    "batchsize": 16,
+                    "num_workers": 0,
+                    "description": f"Downloaded from {model_id}",
+                },
+            )
+
+            # Save the weights file to the model
+            with open(weights_path, "rb") as f:
+                model_meta.weights.save(
+                    f"{model_name}_v{version}_colo_segmentation_RegNetX800MF_base.ckpt",
+                    ContentFile(f.read()),
+                )
+
+            # Set as active meta
+            ai_model.active_meta = model_meta
+            ai_model.save()
+
+            self.stdout.write(
+                self.style.SUCCESS(
+                    f"Successfully {'created' if created else 'updated'} ModelMeta: {model_meta}"
+                )
+            )
+
+        except Exception as e:
+            self.stdout.write(self.style.ERROR(f"Error creating ModelMeta: {e}"))
+            import traceback
+
+            traceback.print_exc()

endoreg_db-0.8.3.3/endoreg_db/management/commands/setup_endoreg_db.py

@@ -0,0 +1,196 @@
+"""
+Django management command to perform complete setup for EndoReg DB when used as an embedded app.
+This command ensures all necessary data and configurations are initialized.
+"""
+
+import os
+from pathlib import Path
+
+from django.core.management import call_command
+from django.core.management.base import BaseCommand
+
+
+class Command(BaseCommand):
+    help = """
+    Complete setup for EndoReg DB when used as an embedded app.
+    This command performs all necessary initialization steps:
+    1. Loads base database data
+    2. Sets up AI models and labels
+    3. Creates cache table
+    4. Initializes model metadata
+    """
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            "--skip-ai-setup",
+            action="store_true",
+            help="Skip AI model setup (for cases where AI features are not needed)",
+        )
+        parser.add_argument(
+            "--force-recreate",
+            action="store_true",
+            help="Force recreation of AI model metadata even if it exists",
+        )
+
+    def handle(self, *args, **options):
+        skip_ai = options.get("skip_ai_setup", False)
+        force_recreate = options.get("force_recreate", False)
+
+        self.stdout.write(self.style.SUCCESS("🚀 Starting EndoReg DB embedded app setup..."))
+
+        # Step 1: Load base database data
+        self.stdout.write("\n📊 Step 1: Loading base database data...")
+        try:
+            call_command("load_base_db_data")
+            self.stdout.write(self.style.SUCCESS("✅ Base database data loaded successfully"))
+        except Exception as e:
+            self.stdout.write(self.style.ERROR(f"❌ Failed to load base data: {e}"))
+            return
+
+            # Step 2: Create cache table (only if using database caching)
+        self.stdout.write("\n💾 Step 2: Setting up caching...")
+        from django.conf import settings
+
+        cache_backend = settings.CACHES.get("default", {}).get("BACKEND", "")
+        if "db" in cache_backend or "database" in cache_backend:
+            self.stdout.write("Using database caching - creating cache table...")
+            try:
+                call_command("createcachetable")
+                self.stdout.write(self.style.SUCCESS("✅ Cache table created successfully"))
+            except Exception as e:
+                self.stdout.write(self.style.ERROR(f"❌ Failed to create cache table: {e}"))
+                return
+        else:
+            self.stdout.write("Using in-memory caching - skipping cache table creation")
+
+        if skip_ai:
+            self.stdout.write(self.style.WARNING("\n⚠️  Skipping AI setup as requested"))
+        else:
+            # Step 3: Load AI model data
+            self.stdout.write("\n🤖 Step 3: Loading AI model data...")
+            try:
+                call_command("load_ai_model_data")
+                self.stdout.write(self.style.SUCCESS("✅ AI model data loaded successfully"))
+            except Exception as e:
+                self.stdout.write(self.style.ERROR(f"❌ Failed to load AI model data: {e}"))
+                return
+
+            # Step 4: Load AI model label data
+            self.stdout.write("\n🏷️  Step 4: Loading AI model label data...")
+            try:
+                call_command("load_ai_model_label_data")
+                self.stdout.write(self.style.SUCCESS("✅ AI model label data loaded successfully"))
+            except Exception as e:
+                self.stdout.write(self.style.ERROR(f"❌ Failed to load AI model label data: {e}"))
+                return
+
+            # Step 5: Create model metadata
+            self.stdout.write("\n📋 Step 5: Creating AI model metadata...")
+            try:
+                # Check if model metadata already exists
+                from endoreg_db.models import AiModel
+
+                default_model_name = "image_multilabel_classification_colonoscopy_default"
+                ai_model = AiModel.objects.filter(name=default_model_name).first()
+
+                if not ai_model:
+                    self.stdout.write(self.style.ERROR(f"❌ AI model '{default_model_name}' not found"))
+                    return
+
+                existing_meta = ai_model.metadata_versions.first()
+                if existing_meta and not force_recreate:
+                    self.stdout.write(self.style.SUCCESS("✅ Model metadata already exists (use --force-recreate to recreate)"))
+                else:
+                    # Try to create model metadata
+                    model_path = self._find_model_weights_file()
+                    if model_path:
+                        call_command(
+                            "create_multilabel_model_meta",
+                            model_name=default_model_name,
+                            model_meta_version=1,
+                            image_classification_labelset_name="multilabel_classification_colonoscopy_default",
+                            model_path=str(model_path),
+                        )
+                        self.stdout.write(self.style.SUCCESS("✅ AI model metadata created successfully"))
+                    else:
+                        self.stdout.write(self.style.WARNING("⚠️  Model weights file not found. AI features may not work properly."))
+
+            except Exception as e:
+                self.stdout.write(self.style.ERROR(f"❌ Failed to create AI model metadata: {e}"))
+                return
+
+        # Step 6: Verification
+        self.stdout.write("\n🔍 Step 6: Verifying setup...")
+        try:
+            self._verify_setup()
+            self.stdout.write(self.style.SUCCESS("✅ Setup verification completed successfully"))
+        except Exception as e:
+            self.stdout.write(self.style.ERROR(f"❌ Setup verification failed: {e}"))
+            return
+
+        self.stdout.write(self.style.SUCCESS("\n🎉 EndoReg DB embedded app setup completed successfully!"))
+        self.stdout.write("\nNext steps:")
+        self.stdout.write("1. Run migrations: python manage.py migrate")
+        self.stdout.write("2. Create superuser: python manage.py createsuperuser")
+        self.stdout.write("3. Start development server: python manage.py runserver")
+
+    def _find_model_weights_file(self):
+        """Find the model weights file in various possible locations."""
+        # Check common locations for model weights
+        possible_paths = [
+            # Test assets (for development)
+            Path("tests/assets/colo_segmentation_RegNetX800MF_6.ckpt"),
+            # Project root assets
+            Path("assets/colo_segmentation_RegNetX800MF_6.ckpt"),
+            # Storage directory
+            Path("data/storage/model_weights/colo_segmentation_RegNetX800MF_6.ckpt"),
+            # Absolute paths based on environment
+            Path(os.getenv("STORAGE_DIR", "storage")) / "model_weights" / "colo_segmentation_RegNetX800MF_6.ckpt",
+        ]
+
+        for path in possible_paths:
+            if path.exists():
+                self.stdout.write(f"Found model weights at: {path}")
+                return path
+
+        self.stdout.write("Model weights file not found in standard locations")
+        return None
+
+    def _verify_setup(self):
+        """Verify that the setup was successful."""
+        from django.conf import settings
+        from django.db import connection
+
+        # Check that required tables exist
+        required_tables = [
+            "endoreg_db_aimodel",
+            "endoreg_db_modelmeta",
+        ]
+
+        # Only check for cache table if using database caching
+        cache_backend = settings.CACHES.get("default", {}).get("BACKEND", "")
+        if "db" in cache_backend or "database" in cache_backend:
+            required_tables.append("django_cache_table")
+
+        cursor = connection.cursor()
+        cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
+        existing_tables = [row[0] for row in cursor.fetchall()]
+
+        missing_tables = [table for table in required_tables if table not in existing_tables]
+        if missing_tables:
+            raise Exception(f"Missing required tables: {missing_tables}")
+
+        # Check that AI models exist (if AI setup was performed)
+        from endoreg_db.models import AiModel
+
+        if AiModel.objects.exists():
+            ai_model_count = AiModel.objects.count()
+            self.stdout.write(f"Found {ai_model_count} AI model(s)")
+
+            # Check for model metadata
+            from endoreg_db.models import ModelMeta
+
+            meta_count = ModelMeta.objects.count()
+            self.stdout.write(f"Found {meta_count} model metadata record(s)")
+
+        self.stdout.write("Setup verification passed")