endoreg-db 0.4.5__py3-none-any.whl → 0.8.6.3__py3-none-any.whl

endoreg_db/utils/pipelines/Readme.md

@@ -0,0 +1,235 @@
+# Pipeline Documentation for `process_video_dir.py`
+
+**File Path:** `endoreg_db/utils/pipelines/process_video_dir.py`
+
+---
+
+> This document outlines the multi-stage video processing pipeline implemented in the `process_video_dir.py` module, covering registration, validation, anonymization, and final state outcomes.
+
+
+
+## 0. Start: Registering a New Video (before any pipeline)
+
+- A new `VideoFile` record is created to register the video file.
+- Linked to a `Center` and an `EndoscopyProcessor`.
+- A `VideoMeta` object is created (or updated) by reading the technical information from the raw video (e.g., **fps**, **resolution**, **frame count**, **duration**).
+- `Frame` objects are initialized in the database — one for each expected frame, but they are not yet extracted.
+- A `VideoState` object is created to track the processing progress for this video.
+- **No frames are extracted yet. No sensitive information or AI predictions exist yet.**
+### Tables Involved
+
+| Table Name   |
+|--------------|
+| VideoFile    |
+| VideoMeta    |
+| Frame        |
+| VideoState   |
+
+## Pipeline 1 Steps
+
+---
+
+### Step 1: Ensure VideoState Exists (`_pipe_1_step_ensure_state`)
+- Ensures a tracking object (`VideoState`) exists for this video.
+
+---
+
+### Step 2: Update or Create VideoMeta (`_pipe_1_step_update_meta`)
+- Extract technical metadata from the raw file.
+- Update or create the `VideoMeta` object.
+
+---
+
+### Step 3: Extract Frames (`_pipe_1_step_extract_frames`)
+- Using `ffmpeg`, the raw video is split into frames.
+- Frames are saved to disk.
+- `Frame` database entries are updated: `is_extracted=True`.
+- `VideoState.frames_extracted` is set to `True`.
+
+---
+
+### Step 4: Extract Text (OCR) (`_pipe_1_step_extract_text`)
+- A random sample of frames is passed through OCR.
+- Text such as **Patient Name**, **Date of Birth (DOB)**, **IDs** is extracted.
+- A `SensitiveMeta` object is created and filled with this information.
+- `VideoState.text_meta_extracted` is set to `True`.
+
+---
+
+### Step 5: Predict Labels (AI Prediction) (`_pipe_1_step_predict`)
+- An AI model is run on frames to predict medical labels (e.g., **polyp**, **normal**, **outside**).
+- Smoothed predictions are generated over frames.
+- A `VideoPredictionMeta` is created to store AI model info and predictions.
+- `VideoState.initial_prediction_completed` is set to `True`.
+
+---
+
+### Step 6: Create Label Segments (`_pipe_1_step_create_segments`)
+- Predictions are converted into labeled segments (`LabelVideoSegment`).
+- Each segment is linked to frame ranges and specific labels.
+- A `LabelVideoSegmentState` is created for each segment to track validation.
+- `VideoState.lvs_created` is set to `True`.
+
+---
+## End of PIPE 1
+
+At this point in the pipeline:
+
+- Frames are **extracted** and **saved**.
+- OCR **text is extracted** and **stored**.
+- AI **predictions are made**.
+- **Label segments are created** based on predictions.
+- **Sensitive information** (e.g. patient identifiers) is stored.
+- `VideoState` is updated to reflect all completed steps.
+
+---
+
+### Tables Involved
+
+| Table Name               |
+|--------------------------|
+| `VideoFile`              |
+| `VideoMeta`              |
+| `Frame`                  |
+| `VideoState`             |
+| `SensitiveMeta`          |
+| `VideoPredictionMeta`    |
+| `LabelVideoSegment`      |
+| `LabelVideoSegmentState` |
+
+---
+## 2. VALIDATION Phase: Review and Approval
+
+**Goal:** Confirm that automated results are correct before anonymization.
+
+### Steps
+
+- A human reviewer (or automated test) opens each `LabelVideoSegment` and reviews it.
+- Each segment’s state is updated:  
+  `LabelVideoSegmentState.is_validated = True`
+- The extracted sensitive metadata (e.g., names, DOB) in `SensitiveMeta` is reviewed.
+- Sensitive metadata state fields are set:  
+  `dob_verified = True`  
+  `names_verified = True`
+- If **any** of these validations fail, processing **must stop** and be corrected.
+- **Only** when **all** validations are successfully marked, the video is allowed to proceed to **PIPE 2**.
+
+---
+
+### Tables Involved
+
+| Table Name               |
+|--------------------------|
+| `LabelVideoSegmentState` |
+| `SensitiveMeta`          |
+
+---
+## 3. PIPELINE 2: Anonymization and Cleanup
+
+**Goal:** Prepare a safe, anonymized version of the video.
+
+---
+
+### Pipeline 2 Major Steps
+
+---
+
+#### Step 1: Ensure VideoState Exists Again (`_pipe_2_step_ensure_state`)
+- Confirm that the `VideoState` tracking object is still present.
+
+---
+
+#### Step 2: Ensure Frames Available (`_pipe_2_step_extract_frames`)
+- Check if extracted frames are available.
+- If frames were deleted or are missing, **re-extract** them from the raw video file.
+
+---
+
+#### Step 3: Anonymize Video (`_pipe_2_step_anonymize`)
+- Frames labeled as `"outside"` are blacked out.
+- Remaining frames are masked using **endoscopy ROI** (Region of Interest).
+- New anonymized frames are created and saved temporarily.
+- A new anonymized video is assembled from these frames.
+- New video file path is saved in:  
+  `VideoFile.processed_file`
+- Video hash is calculated and saved in:  
+  `VideoFile.processed_video_hash`
+- `VideoState.anonymized = True`
+
+---
+
+#### Step 4: Delete Raw Files and Frames
+- Raw video file and extracted frames are **permanently deleted**.
+- `VideoState.frames_extracted = False`
+
+---
+
+#### Step 5: Delete SensitiveMeta (`_pipe_2_step_delete_sensitive_meta`)
+- The `SensitiveMeta` object is deleted from the database.
+- `VideoFile.sensitive_meta = None`
+
+---
+
+### At the End of PIPE 2:
+
+- Only the **anonymized processed video** remains.
+- **All sensitive data** and **raw video/frames** are permanently deleted.
+- **Technical metadata** and **AI prediction audit data** are retained.
+
+---
+
+### Tables Involved
+
+| Table Name      | Notes                            |
+|------------------|----------------------------------|
+| `VideoFile`      | Updated with processed file info |
+| `Frame`          | Frames used, then deleted        |
+| `VideoState`     | Tracks anonymization status      |
+| `SensitiveMeta`  | **Deleted after anonymization**  |
+
+---
+## 4. FINAL STATE
+
+### Database Contains:
+
+- The **processed, safe, anonymized video**:  
+  `VideoFile.processed_file`
+- **Technical information** about the video:  
+  `VideoMeta`
+- **Medical label segments** for audit:  
+  `LabelVideoSegment` and `LabelVideoSegmentState`
+- **AI prediction information** for audit:  
+  `VideoPredictionMeta`
+- **Sensitive patient information** has been **permanently deleted**
+- **Raw extracted frames** and the **original raw video file** have been **permanently deleted**
+
+---
+
+### `VideoState` Summary:
+
+- `frames_extracted = False`  *(raw frames are gone)*
+- `anonymized = True`  *(anonymized video is ready)*
+
+---
+
+### Tables Overview
+
+| Table Name              | Purpose                                                                 |
+|--------------------------|-------------------------------------------------------------------------|
+| `VideoFile`              | Main entry linking video paths, processed file, and metadata references |
+| `VideoMeta`              | Technical information (fps, duration, frame count, etc.)                |
+| `Frame`                  | Individual frame images from the video *(deleted in PIPE 2)*            |
+| `VideoState`             | Tracks pipeline processing steps and their completion                   |
+| `SensitiveMeta`          | Patient-sensitive data from OCR *(deleted in PIPE 2)*                   |
+| `VideoPredictionMeta`    | Information about the AI model and predictions                          |
+| `LabelVideoSegment`      | Video segments created from AI predictions                              |
+| `LabelVideoSegmentState` | Tracks human validation status for each segment                         |
+
+---
+### Note: on Frame Deletion and Data Integrity
+
+Although the frames are permanently deleted after the entire execution of Pipeline 1 and Pipeline 2, the necessary data and information about these frames are still populated in the database. Therefore, a script is required to utilize the correct imported data and **re-extract potential frames** from the original video.
+
+This process ensures that the required frame data can be retrieved and used even after deletion.
+
+

endoreg_db/utils/pipelines/process_video_dir.py

@@ -0,0 +1,120 @@
+from endoreg_db.models import (
+    VideoFile, Center, EndoscopyProcessor, AiModel, ModelMeta,
+    LabelVideoSegment, SensitiveMeta
+)
+from pathlib import Path
+from icecream import ic
+from tqdm import tqdm
+import os
+from django.core.management import call_command
+import logging
+from django.core.exceptions import ObjectDoesNotExist
+
+logger = logging.getLogger(__name__)
+# Usage
+# from endoreg_db.utils.pipelines.process_video_dir import process_video_dir
+# (video_dir=Path("/path/to/your/videos"))
+
+# DEFAULT_DIR = Path(__file__).resolve().parent.parent.parent.parent / "data/raw_videos"
+DEFAULT_DIR = Path(__file__).resolve().parent.parent.parent.parent / "data/study_data"
+CENTER_NAME = os.environ.get("CENTER_NAME", "university_hospital_wuerzburg")
+ENDOSCOPY_PROCESSOR_NAME = os.environ.get("ENDOSCOPY_PROCESSOR_NAME", "olympus_cv_1500")
+MODEL_NAME = "image_multilabel_classification_colonoscopy_default"
+MODEL_WEIGHTS_PATH = "./tests/assets/colo_segmentation_RegNetX800MF_6.ckpt"
+ic(DEFAULT_DIR)
+
+
+
+def process_video_dir(video_dir: Path=DEFAULT_DIR, center_name: str = CENTER_NAME, endoscopy_processor_name: str = ENDOSCOPY_PROCESSOR_NAME):
+    """
+    Process a directory of video files and create VideoFile objects in the database.
+    
+    Args:
+        video_dir (Path): Path to the directory containing video files.
+        center_name (str): Name of the center to associate with the video files.
+        endoscopy_processor_name (str): Name of the endoscopy processor to associate with the video files.
+    """
+    # Make sure the ai model exists:
+    ai_model = AiModel.objects.filter(name=MODEL_NAME)
+    if ai_model:
+        ai_model = ai_model.first()
+    try:
+        model_meta = ai_model.get_latest_version()
+        assert isinstance(model_meta, ModelMeta), "No ModelMeta found in the database."
+    except Exception:
+        logger.warning(f"ModelMeta not found for {MODEL_NAME}. Creating a new one.")
+        # Create a new ModelMeta instance
+        call_command(
+            "create_multilabel_model_meta",
+            "--model_path",
+            MODEL_WEIGHTS_PATH,
+        )
+
+
+
+    # Get the center and endoscopy processor objects
+
+    # Iterate through all files in the directory
+    for file_path in tqdm(video_dir.iterdir()):
+        logger.info(f"Processing file: {file_path}")
+        if file_path.is_file() and file_path.suffix in ['.mp4', '.avi', '.mov']:
+            # Create a VideoFile object
+            video_file = VideoFile.create_from_file_initialized(
+                file_path=file_path,
+                center_name=center_name,
+                processor_name=endoscopy_processor_name,
+            )
+            logger.warning(f"Processing video file: {video_file}")
+            try:
+                success_pipe_1 = video_file.pipe_1(
+                    model_name = MODEL_NAME,
+                )
+                assert success_pipe_1, f"Pipe 1 failed for video {video_file.uuid}"
+            except Exception as e:
+                logger.error(f"Pipe 1 failed for video {video_file.uuid}: {e}", exc_info=True)
+                raise e
+
+            video_file.refresh_from_db()
+
+            ####### SIMULATION OF VALIDATION #######
+            # Simulate the validation process for video segments
+            outside_segments = video_file.get_outside_segments()
+            logger.warning(f"Outside segments found for simulation: {outside_segments}")
+            if outside_segments:
+                for segment in outside_segments:
+                    try:
+                        seg_state = segment.state
+                        seg_state.is_validated = True
+                        seg_state.save()
+                        logger.info(f"Simulated validation for segment {segment.id}")
+                    except LabelVideoSegment.state.RelatedObjectDoesNotExist:
+                        logger.warning(f"Cannot simulate validation for segment {segment.id}: Related state object does not exist.")
+                    except Exception as e:
+                        logger.error(f"Error during segment state simulation for segment {segment.id}: {e}", exc_info=True)
+
+
+            # Simulate the validation process for the video files sensitive meta obj
+            if video_file.sensitive_meta:
+                try:
+                    sm_state = video_file.sensitive_meta.state
+                    sm_state.dob_verified = True
+                    sm_state.names_verified = True
+                    sm_state.save()
+                    logger.info(f"Simulated validation for sensitive meta of video {video_file.uuid}")
+                except Exception as e:
+                    logger.error(f"Error during sensitive meta state simulation for video {video_file.uuid}: {e}", exc_info=True)
+
+            else:
+                logger.warning(f"Cannot simulate validation for sensitive meta of video {video_file.uuid}: SensitiveMeta object does not exist (likely due to pipe_1 failure or incomplete text extraction).")
+
+            ######## SIMULATION OF VALIDATION ########
+
+            video_file.refresh_from_db()
+            if success_pipe_1:
+                try:
+                    video_file.pipe_2()
+                except Exception as e:
+                    logger.error(f"Pipe 2 failed for video {video_file.uuid}: {e}", exc_info=True)
+            else:
+                logger.warning(f"Skipping Pipe 2 for video {video_file.uuid} because Pipe 1 did not complete successfully.")
+

endoreg_db/utils/product/sum_emissions.py

@@ -0,0 +1,20 @@
+from endoreg_db.models.administration.product.product_material import ProductMaterial
+
+
+from typing import List
+
+
+def sum_emissions(product_materials:List["ProductMaterial"]):
+    # sum up the emissions
+    emission = 0
+    reference_unit = None
+    for product_material in product_materials:
+        if not reference_unit:
+            reference_unit = product_material.unit
+        else:
+            assert reference_unit == product_material.unit, "ProductMaterial units do not match"
+        em_value, emission_unit = product_material.get_emission()
+        assert reference_unit == emission_unit, "ProductMaterial units do not match"
+        emission += em_value
+
+    return emission, reference_unit

endoreg_db/utils/product/sum_weights.py

@@ -0,0 +1,18 @@
+from endoreg_db.models.administration.product.product_material import ProductMaterial
+
+
+from typing import List
+
+
+def sum_weights(product_materials: List["ProductMaterial"]):
+    # sum up the weights
+    weight = 0
+    reference_unit = None
+    for product_material in product_materials:
+        if not reference_unit:
+            reference_unit = product_material.unit
+        else:
+            assert reference_unit == product_material.unit, "ProductMaterial units do not match"
+        weight += product_material.quantity
+
+    return weight, reference_unit

endoreg_db/utils/pydantic_models/__init__.py

@@ -0,0 +1,6 @@
+'''Module for Pydantic models.'''
+from .db_config import DbConfig
+
+__all__ = [
+    'DbConfig'
+]

endoreg_db/utils/pydantic_models/db_config.py

@@ -0,0 +1,57 @@
+from pathlib import Path
+from typing import Optional, Union
+import yaml
+from icecream import ic
+from pydantic import BaseModel
+
+
+class DbConfig(BaseModel):
+    """Database configuration model."""
+
+    engine: str = "django.db.backends.postgresql"
+    host: str
+    port: int
+    user: str
+    password: Optional[str] = None
+    password_file: str = "/etc/secrets/vault/SCRT_local_password_maintenance_password"
+    name: str
+
+    # class options should allow for unused excess data
+
+    @classmethod
+    def from_file(cls, path: Union[Path, str]) -> "DbConfig":
+        """Create a DbConfig instance from a YAML configuration file."""
+        if isinstance(path, str):
+            filepath = Path(path)
+        else:
+            filepath = path
+        assert filepath.exists(), f"Missing Config {filepath}"
+
+        with open(filepath, "r", encoding="utf-8") as f:
+            cfg = yaml.safe_load(f)
+
+        obj = cls(**cfg)
+
+        return obj
+
+    def sync_password(self):
+        """Load the password from the configured password file."""
+        with open(self.password_file, "r", encoding="utf-8") as f:
+            self.password = f.read().strip()
+
+    def custom_validate(self):
+        """Fully validate all database configuration fields."""
+        self.sync_password()
+
+        assert self.host, "Missing Host"
+        assert self.port, "Missing Port"
+        assert self.user, "Missing User"
+        assert self.password, "Missing Password"
+        assert self.name, "Missing Database"
+
+    def to_file(self, target: str = "./conf/db.yml", ask_override: bool = True):
+        """Export the configuration to a YAML file."""
+        ic(target)
+
+        with open(target, "w") as f:
+            yaml.safe_dump(self.model_dump(), f)