endoreg-db 0.8.2__tar.gz → 0.8.2.2__tar.gz

{endoreg_db-0.8.2 → endoreg_db-0.8.2.2}/.env.example

@@ -10,7 +10,7 @@ TIME_ZONE=Europe/Berlin
 # -------------------------------------------------------------------
 # Paths (relative to repo root unless absolute)
 # -------------------------------------------------------------------
-STORAGE_DIR=storage
+STORAGE_DIR=data/storage
 ASSET_DIR=tests/assets
 STATIC_URL=/static/
 MEDIA_URL=/media/
@@ -86,4 +86,14 @@ TEST_RUN_FRAME_NUMBER=1000
 RUST_BACKTRACE=1
 DJANGO_FFMPEG_EXTRACT_FRAME_BATCHSIZE=500
 VIDEO_ALLOW_FPS_FALLBACK=True
-VIDEO_DEFAULT_FPS=50
+VIDEO_DEFAULT_FPS=50
+
+# Caching optimization for huggingface, ollama, etc. 
+# CAUTION: SETUP ASSUMES LUXNIX USER 'admin' WITH HOME /home/admin
+
+HF_HOME=/home/admin/.cache/huggingface
+HF_HUB_CACHE=/home/admin/.cache/huggingface/hub
+HF_HUB_ENABLE_HF_TRANSFER=1
+OLLAMA_KEEP_ALIVE=4h
+OLLAMA_MODELS=/home/admin/.ollama/models
+TRANSFORMERS_CACHE=/home/admin/.cache/huggingface/hub

endoreg_db-0.8.2.2/.git

@@ -0,0 +1 @@
+gitdir: ../../.git/modules/libs/endoreg-db

endoreg_db-0.8.2.2/.github/copilot-instructions.md

@@ -0,0 +1,39 @@
+## Big Picture
+- **Core app**: Django project lives in `endoreg_db/`, `config/settings/*` drive environments via `endoreg_db.config.env`; root URLs mount the entire API under `/api/`.
+- **Domain**: Focus on ingesting and anonymizing endoscopy videos and clinical reports; `services/video_import.py` handles the full pipeline, mirrored by management commands for CLI workflows.
+- **Data layout**: Working assets stay under `storage/` (media root) and `data/` (raw/test fixtures); `endoreg_db/utils/paths.py` centralizes directories—respect existing folder contracts when writing files.
+- **Knowledge base**: Medical vocabularies live in YAML under `endoreg_db/data/`; management commands in `endoreg_db/management/commands/load_*` hydrate models from those files—keep YAML schema changes aligned with the commands.
+- **Async flows**: `celery_app.py` wires Celery but operational imports use Redis Queue (`endoreg_db/tasks/video_ingest.py`); align new background work with that queue unless migrating everything to Celery.
+
+## Environment & Tooling
+- **Platform baseline**: Most development happens on NixOS; run `devenv up` (or allow via `direnv`) before any command so the pinned toolchain and `uv` environment from `devenv.nix` activate.
+- **Dev shell**: Once inside the shell, use `devenv task run env:build` to refresh `.env` via `env_setup.py`; the task relies on `.devenv-vars.json` generated during shell entry.
+- **Python entry**: Always call `uv run ...` (e.g. `uv run python manage.py migrate`) so dependencies resolve inside `.devenv/state/venv`.
+- **Settings switch**: `DJANGO_SETTINGS_MODULE` defaults to `config.settings.dev`; tests use `config.settings.test` which persists an SQLite DB at `data/tests/db/test_db.sqlite3`.
+- **External repos**: LX anonymizer support is expected (`lx-anonymizer>=0.8.2.1`); the Nix enter hook can clone it—check this before debugging anonymization failures.
+
+## Testing
+- **Test runner**: Prefer `uv run python runtests.py` to execute curated suites; arguments map to subpackages under `tests/` (e.g. `runtests helpers`).
+- **Pytest direct**: For ad-hoc cases run `uv run pytest tests/path::TestClass -k keyword`; `pytest.ini` already points at the Django settings.
+- **Video guards**: GPU/ffmpeg-heavy specs are behind `RUN_VIDEO_TESTS`; default false—set `RUN_VIDEO_TESTS=true` when required assets in `tests/assets/` are available.
+- **Fixtures**: `tests/conftest.py` configures storage dirs and shared fixtures; extend `endoreg_db/factories/` instead of hand-building ORM objects in tests.
+
+## Data & Management Commands
+- **Database seed**: `load_base_db_data` (and other `load_*` commands) populate clinical taxonomies from CSVs under `data/` and `models_table.*`.
+- **Knowledge YAML**: Commands like `load_disease_data`, `load_finding_data`, etc. expect curated YAML in `endoreg_db/data/`; keep keys stable and update factories/tests when the YAML schema shifts.
+- **Import scripts**: Video/report ingestion commands delegate to `services/video_import.py` and `services/pdf_import.py`; reuse the service layer when adding new commands.
+- **Backups**: `export_db.sh` and `import_db.sh` wrap Django `dumpdata/loaddata`—update them instead of reinventing backup flows.
+- **Paths**: `env_setup.py` writes `STORAGE_DIR` into `.env`; new code should resolve paths via `endoreg_db.utils.data_paths` or `config.env_path` to stay relocatable.
+
+## API & Presentation
+- **Routing**: `endoreg_db/root_urls.py` includes `endoreg_db/urls/`; keep new REST endpoints inside `endoreg_db/api/views` + serializers and register them in `api_urls.py`.
+- **Translations**: `django-modeltranslation` is preloaded (languages `de`/`en`); text fields need `TranslationOptions` under `endoreg_db/models/**/translation*.py`.
+- **Forms/Admin**: Custom admin and form logic lives in `endoreg_db/admin.py` and `endoreg_db/forms/`; follow existing Bootstrap 5 widgets in `endoreg_db/templates/`.
+- **Security**: Sensitive metadata models in `endoreg_db/models/metadata/` use state objects—preserve their transitions (see `VideoImportService._process_frames_and_metadata`).
+
+## Observability & Utilities
+- **Logging**: `config/settings/base.py` defines `TEST_LOGGER_NAMES`; use `logging.getLogger(__name__)` so unit tests capture module-level logs consistently.
+- **Pipelines**: `endoreg_db/utils/pipelines/process_video_dir.py` shows the orchestration pattern—refresh model state after `pipe_1/pipe_2` and reuse its helpers for batch jobs.
+- **Permissions**: Storage-aware helpers live in `endoreg_db/utils/permissions.py` and related services—reuse them instead of manual `os.path` checks.
+- **Path hygiene**: File movement bugs usually stem from bypassing `endoreg_db.utils.paths` or `endoreg_db.config.env`; always derive locations from `data_paths[...]`/`env_path(...)` so raw/anonymized video & PDF paths stay consistent.
+- **RQ usage**: Enqueue jobs via `endoreg_db/tasks/video_ingest.enqueue_video_import`; ensure `RQ_REDIS_URL` is configured and Django is initialized before background execution.

{endoreg_db-0.8.2 → endoreg_db-0.8.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: endoreg-db
-Version: 0.8.2
+Version: 0.8.2.2
 Summary: EndoReg Db Django App
 Project-URL: Homepage, https://info.coloreg.de
 Project-URL: Repository, https://github.com/wg-lux/endoreg-db
@@ -33,7 +33,7 @@ Requires-Dist: gunicorn>=23.0.0
 Requires-Dist: icecream>=2.1.4
 Requires-Dist: librosa==0.11.0
 Requires-Dist: llvmlite>=0.44.0
-Requires-Dist: lx-anonymizer[llm,ocr]>=0.8.2
+Requires-Dist: lx-anonymizer[llm,ocr]>=0.8.2.2
 Requires-Dist: moviepy==2.2.1
 Requires-Dist: mypy>=1.16.0
 Requires-Dist: numpy>=2.2.3

{endoreg_db-0.8.2 → endoreg_db-0.8.2.2}/devenv.nix

@@ -23,7 +23,6 @@ let
   
   buildInputs = with pkgs; [
     python312
-    python312Packages.tkinter
     stdenv.cc.cc
     tesseract
     glib
@@ -37,7 +36,6 @@ let
   runtimePackages = with pkgs; [
     stdenv.cc.cc
     ffmpeg-headless.bin
-    python312Packages.tkinter
     tesseract
     uvPackage
     libglvnd # Add libglvnd for libGL.so.1
@@ -48,16 +46,16 @@ let
 
   _module.args.buildInputs = baseBuildInputs;
 
-  lx-anonymizer-src = pkgs.fetchGit {
-    url = "https://github.com/wg-lux/lx-anonymizer";
-    ref = "prototype";
-    # If you know the specific revision, it's better to use rev for reproducibility
-    # rev = "abcdef1234567890"; 
-  };
+  # lx-anonymizer-src = pkgs.fetchGit {
+  #   url = "https://github.com/wg-lux/lx-anonymizer";
+  #   ref = "prototype";
+  #   # If you know the specific revision, it's better to use rev for reproducibility
+  #   # rev = "abcdef1234567890"; 
+  # };
 
-  imports = [ 
-    "${lx-anonymizer-src}/devenv.nix"
-  ]; 
+  # imports = [ 
+  #   "${lx-anonymizer-src}/devenv.nix"
+  # ]; 
 
 in 
 {

endoreg_db-0.8.2.2/docs/test_suite_blueprint.md

@@ -0,0 +1,58 @@
+# Test Suite Standardization Blueprint (Draft)
+
+_Date: 2025-10-18_
+
+## Goals
+- Consistent test layout and naming across all domains.
+- Predictable fixture hierarchy that makes scope explicit and minimises global state.
+- Single cache interface with explicit lifecycle for expensive resources (DB seeds, video assets, ML models).
+- Clear integration guidance for new tests and suites.
+
+## Directory & Naming Conventions
+- **Top-level folders:** maintain domain grouping (`tests/<domain>/`) but enforce suffixes: `test_*.py` files only.
+- **Class naming:** use `Test<Subject>` for unit-style suites, `<Feature>Tests` reserved for integration/system cases.
+- **Function naming:** `test_<scenario>_<expected>` pattern; leverage parametrization for variant coverage instead of bespoke fixtures.
+- **Helper modules:** move shared builders/factories under `tests/helpers/` with explicit exports through `__all__`.
+
+## Fixture Hierarchy
+- **Session scope (`conftest.py` or plugin):**
+  - `base_data`: loads taxonomy/seed data exactly once, returns immutable context object.
+  - `cache`: provides namespaced cache manager (see Caching Layer) with `get/set/clear` API.
+  - `env_settings`: applies test-specific overrides (MEDIA_ROOT, Celery eager) in a context manager.
+- **Module scope:** domain-specific heavy resources (e.g., processed video file) keyed via cache fixture to avoid duplication.
+- **Function scope:** lightweight factories, API clients, data builders; avoid mutating session objects directly.
+- Use `pytest_plugins = ["tests.plugins.cache", "tests.plugins.fixtures"]` to centralise definitions.
+
+## Parametrization & Markers
+- Prefer `@pytest.mark.parametrize` over hand-written loops.
+- Define common markers: `slow`, `video`, `integration`, `requires_gpu`. Document skip conditions in `pytest.ini`.
+- Implement marker-aware skip logic in fixtures rather than environment variable spaghetti.
+
+## Caching Layer Design
+- Introduce `tests/plugins/cache.py` defining a `CacheManager` with:
+  - `get(namespace, key, default=None)` / `set(namespace, key, value)` APIs.
+  - Lazy initialisation backed by `functools.lru_cache` for in-memory usage plus optional `diskcache` fallback if available.
+  - `invalidate(namespace, key=None)` hooks invoked via pytest `request.addfinalizer` to avoid cross-test leaks.
+- Session-scoped `cache` fixture returns a manager instance; module/functional fixtures request namespaces (`cache.namespace("video")`).
+- Provide helper `cache.memoize(namespace)` decorator to wrap expensive factory functions (e.g., processed video creation).
+- Replace module-level globals (`_base_data_loaded`, `_session_video_file`) by storing state under deterministic keys, e.g. `cache.set("video", "processed_default", value)`.
+- Emit debug logging when cache hits/misses occur during `-vv` runs to aid troubleshooting.
+
+## Documentation & Governance
+- Document expectations in `docs/testing/TEST_GUIDELINES.md` (to be created).
+- Add pre-commit lint to fail if non-conforming test module names detected.
+- Establish codeowners for `tests/` to review adherence.
+
+## Next Steps
+1. Implement prototype cache manager and fixtures (Step 3 of action plan).
+  - Create `tests/plugins/cache.py` and `tests/plugins/__init__.py` exporting `pytest_plugins`.
+  - Update `pytest.ini` to register default markers (`slow`, `video`, `integration`, `requires_gpu`).
+  - Refactor `tests/conftest.py` to consume the new `cache` fixture and remove global caches.
+2. Migrate `tests/media/video` to new fixture hierarchy as pilot (Step 4).
+  - Introduce module-scoped fixtures leveraging `cache.namespace("video")` and ensure cleanup via invalidation hooks.
+  - Wire helper modules such as `tests/helpers/optimized_video_fixtures.py` into dedicated namespaces via `configure_cache` to eliminate legacy globals.
+  - ✅ Segment CRUD suite now runs against a lightweight cached video stub; validate similar patterns for services tests next.
+  - ✅ Base data loader now provisions stub AI model metadata (behind `USE_STUB_MODEL_META=true`) so `load_default_ai_model` no longer copies large checkpoints during fast suites.
+  - Current end-to-end runtime still ~660 s; profile durations to locate the next hotspots.
+  - Measure runtime improvements and document findings in `docs/testing/TEST_GUIDELINES.md`.
+3. Update developer guide with blueprint summary once pilot validated.

endoreg_db-0.8.2.2/docs/test_suite_review.md

@@ -0,0 +1,31 @@
+# Test Suite Review (2025-10-18)
+
+## Module Inventory Snapshot
+- `tests/administration/*`: CRUD/admin scenarios for centers, shifts, qualifications.
+- `tests/assets/*`: Large artefact fixtures (models, JSON) consumed by video/import suites.
+- `tests/dataloader/test_dataloader.py`: Exercises custom dataset ingestion helpers.
+- `tests/environment/test_env.py`: Validates env wiring and settings overrides.
+- `tests/examination/*`: Domain tests for examination models and serializers.
+- `tests/fileserver/*`: Covers WhiteNoise static file handling.
+- `tests/finding/*`: Disease/finding taxonomy loading and serializer behaviour.
+- `tests/helpers/*`: Fixture factories plus integration smoke tests for helper modules.
+- `tests/legacy/*`: Regression coverage for legacy data migration utilities.
+- `tests/media/video/*`: Heavy video import/anonymization flows (long runtime, GPU/ffmpeg dependencies).
+- `tests/models/*`: ORM and state machine regression coverage.
+- `tests/pipelines/*`: Orchestration pipelines and process directory flows.
+- `tests/requirement/*`: Requirement rule matrix validation; high combinatorial count.
+- `tests/services/*`: Service-layer orchestration smoke tests (RQ, anonymizer, imports).
+
+## Shared Fixture & Caching Landscape
+- `tests/conftest.py` defines global caches (`_session_cache`, `_session_video_file`, `_base_data_loaded`) implemented via module-level globals; invalidation is implicit and hard to reason about during parametrised runs.
+- Base taxonomy loaders (`load_*`) execute inside `base_db_data` fixture with a function scope guard backed by `global _base_data_loaded`, mixing fixture scopes and custom caching.
+- Video fixtures (`sample_video_file`, `processed_video_file`) key off environment flags (`RUN_VIDEO_TESTS`, `SKIP_EXPENSIVE_TESTS`) with ad-hoc globals; repeated suite invocations in the same process can leak cached state.
+- Session-level DB optimisations (`optimize_database_queries`) execute per-test (autouse) rather than establishing a single shared connection configuration.
+- `_session_cache` remains unused across the suite, and several modules re-create expensive assets (e.g., YAML loads, JSON fixtures) without a shared cache hook.
+
+## Pain Points Observed During Full-Suite Runs
+- Environment flag handling (`RUN_VIDEO_TESTS`) is inconsistent: default expression evaluates to `False` even when explicitly enabling, leading to skipped media tests locally but executed in CI.
+- Global caches are not namespaced by test parameters; when parametrised tests mutate cached video objects, downstream tests see dirty state, causing non-deterministic failures in full runs.
+- Multiple suites duplicate fixture logic (e.g., default center/processor creation) instead of relying on `tests.helpers.default_objects`, increasing setup time and risking drift.
+- Several long-running suites (`tests/media/video`, `tests/pipelines`) hit the filesystem repeatedly because cached paths live outside a central storage coordinator; temp directories are re-created per module.
+- Lack of a documented test layout encourages bespoke fixtures per module, making it difficult to standardise teardown behaviour and to introduce suite-wide caching.

{endoreg_db-0.8.2 → endoreg_db-0.8.2.2}/endoreg_db/helpers/default_objects.py

@@ -1,4 +1,5 @@
 import random
+from typing import Optional
 from endoreg_db.models import (
     Center, 
     Gender, 
@@ -17,6 +18,7 @@ import shutil
 from pathlib import Path
 from django.conf import settings # Import settings
 from django.core.files.storage import default_storage # Import default storage
+from django.db.models.fields.files import FieldFile
 
 from endoreg_db.utils import (
     create_mock_patient_name,
@@ -44,6 +46,10 @@ DEFAULT_INDICATIONS = [
 DEFAULT_SEGMENTATION_MODEL_NAME = "image_multilabel_classification_colonoscopy_default"
 
 DEFAULT_GENDER = "unknown"
+DEFAULT_PATIENT_FIRST_NAME = "TestFirst"
+DEFAULT_PATIENT_LAST_NAME = "TestLast"
+DEFAULT_PATIENT_GENDER_NAME = "female"
+DEFAULT_PATIENT_BIRTH_DATE = date(1970, 1, 1)
 
 def get_information_source_prediction():
     """
@@ -170,33 +176,41 @@ def get_default_center() -> Center:
     return center
 
 def generate_patient(**kwargs) -> Patient:
-    """
-    Creates a Patient instance with randomized or specified attributes.
-    
-    Randomly generates first name, last name, date of birth, gender, and center for the patient unless overridden by keyword arguments. Raises ValueError if the provided gender is invalid.
-    
-    Args:
-        **kwargs: Optional overrides for patient attributes such as 'first_name', 'last_name', 'birth_date', 'gender', and 'center'.
-    
-    Returns:
-        A Patient instance with the specified or randomly generated attributes.
-    """
-    # Set default values
-    gender = kwargs.get("gender", get_random_gender())
-    if not isinstance(gender, Gender):
+    """Create a Patient with deterministic defaults unless ``randomize=True`` is supplied."""
+
+    randomize = kwargs.pop("randomize", False)
+
+    gender = kwargs.get("gender")
+    if gender is None:
+        if randomize:
+            gender = get_random_gender()
+        else:
+            gender = Gender.objects.get(name=DEFAULT_PATIENT_GENDER_NAME)
+    elif not isinstance(gender, Gender):
         gender = Gender.objects.get(name=gender)
 
-    if not isinstance(gender, Gender):
-        raise ValueError("No Gender Found")
-    first_name, last_name = create_mock_patient_name(gender = gender.name)
-    first_name = kwargs.get("first_name", first_name)
-    last_name = kwargs.get("last_name", last_name)
-    birth_date = kwargs.get("birth_date", "1970-01-01")
-    dob = date.fromisoformat(birth_date)
-    center = kwargs.get("center", None)
+    first_name = kwargs.get("first_name")
+    last_name = kwargs.get("last_name")
+    if first_name is None or last_name is None:
+        if randomize:
+            generated_first, generated_last = create_mock_patient_name(gender=gender.name)
+        else:
+            generated_first, generated_last = DEFAULT_PATIENT_FIRST_NAME, DEFAULT_PATIENT_LAST_NAME
+        first_name = first_name or generated_first
+        last_name = last_name or generated_last
+
+    dob = kwargs.get("dob")
+    if dob is None:
+        birth_date = kwargs.get("birth_date", DEFAULT_PATIENT_BIRTH_DATE)
+        if isinstance(birth_date, date):
+            dob = birth_date
+        else:
+            dob = date.fromisoformat(str(birth_date))
+
+    center = kwargs.get("center")
     if center is None:
         center = get_default_center()
-    else:
+    elif not isinstance(center, Center):
         center = Center.objects.get(name=center)
 
     patient = Patient(
@@ -261,6 +275,7 @@ def get_default_egd_pdf():
     shutil.copy(egd_path, temp_file_path)
 
     pdf_file = None
+    file_field: Optional[FieldFile] = None
     try:
         # Create the PDF record using the temporary file.
         # delete_source=True will ensure temp_file_path is deleted by create_from_file
@@ -275,8 +290,11 @@ def get_default_egd_pdf():
             raise RuntimeError("Failed to create PDF file object")
         
         # Use storage API to check existence
-        if not default_storage.exists(pdf_file.file.path):
-            raise RuntimeError(f"PDF file does not exist in storage at {pdf_file.file.path}")
+        file_field = pdf_file.file
+        if not isinstance(file_field, FieldFile):
+            raise RuntimeError("RawPdfFile.file did not return a FieldFile instance")
+        if not default_storage.exists(file_field.path):
+            raise RuntimeError(f"PDF file does not exist in storage at {file_field.path}")
         
         # Check that the source temp file was deleted
         if temp_file_path.exists():
@@ -308,10 +326,11 @@ def get_default_egd_pdf():
 
     # pdf_file.file.path might fail if storage doesn't support direct paths (like S3)
     # Prefer using storage API for checks. Logging path if available.
-    try:
-        logger.info(f"PDF file created: {pdf_file.file.name}, Path: {pdf_file.file.path}")
-    except NotImplementedError:
-        logger.info(f"PDF file created: {pdf_file.file.name}, Path: (Not available from storage)")
+    if file_field is not None:
+        try:
+            logger.info(f"PDF file created: {file_field.name}, Path: {file_field.path}")
+        except NotImplementedError:
+            logger.info(f"PDF file created: {file_field.name}, Path: (Not available from storage)")
 
 
     return pdf_file

{endoreg_db-0.8.2 → endoreg_db-0.8.2.2}/endoreg_db/management/commands/import_video.py

@@ -294,10 +294,12 @@ class Command(BaseCommand):
                 
                 # Updated to handle new return signature (path, metadata)
                 cleaned_video_path, extracted_metadata = frame_cleaner.clean_video(
-                    Path(video_file_obj.raw_file.path),
+                    video_path=Path(video_file_obj.raw_file.path),
                     video_file_obj=video_file_obj,  # Pass VideoFile object to store metadata
-                    report_reader=report_reader,
-                    device_name=processor_name
+                    device_name=processor_name,
+                    endoscope_roi=video_file_obj.processor.get_roi_endoscope_image if video_file_obj.processor else None,
+                    output_path=video_file_obj.get_processed_file_path(),
+                    technique="mask_overlay"  # Use mask overlay technique as default, if not set this will be inferred.
                 )
                 
                 # Save the cleaned video using Django's FileField

endoreg_db-0.8.2.2/endoreg_db/migrations/0003_add_center_display_name.py

@@ -0,0 +1,30 @@
+from django.db import migrations, models
+
+
+def populate_display_name(apps, schema_editor):
+    Center = apps.get_model('endoreg_db', 'Center')
+    for center in Center.objects.all():
+        if not center.display_name:
+            center.display_name = center.name
+            center.save(update_fields=['display_name'])
+
+
+def reset_display_name(apps, schema_editor):
+    Center = apps.get_model('endoreg_db', 'Center')
+    Center.objects.update(display_name='')
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('endoreg_db', '0002_add_video_correction_models'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='center',
+            name='display_name',
+            field=models.CharField(blank=True, default='', max_length=255),
+        ),
+        migrations.RunPython(populate_display_name, reset_display_name),
+    ]

{endoreg_db-0.8.2 → endoreg_db-0.8.2.2}/endoreg_db/models/administration/center/center.py

@@ -19,6 +19,7 @@ class Center(models.Model):
 
     # import_id = models.IntegerField(primary_key=True)
     name = models.CharField(max_length=255)
+    display_name = models.CharField(max_length=255, blank=True, default="")
 
     first_names = models.ManyToManyField(
         to="FirstName",
@@ -45,8 +46,13 @@ class Center(models.Model):
     def natural_key(self) -> tuple[str]:
         return (self.name,)
 
+    def save(self, *args, **kwargs):
+        if not self.display_name:
+            self.display_name = self.name
+        super().save(*args, **kwargs)
+
     def __str__(self) -> str:
-        return str(object=self.name)
+        return str(object=self.display_name or self.name)
 
     def get_first_names(self):
         return self.first_names.all()

{endoreg_db-0.8.2 → endoreg_db-0.8.2.2}/endoreg_db/models/media/pdf/raw_pdf.py

@@ -383,37 +383,42 @@ class RawPdfFile(models.Model):
         new_file_name, _uuid = get_uuid_filename(file_path)
         logger.info(f"Generated new filename: {new_file_name}")
 
-        # Create model instance (without file initially)
-        raw_pdf = cls(
-            pdf_hash=pdf_hash,
-            center=center,
-        )
-
-        # Assign file using Django's File wrapper and save
+        # Create model instance via manager so creation can be intercepted/mocked during tests
         try:
             with file_path.open("rb") as f:
                 django_file = File(f, name=new_file_name)
-                raw_pdf.file = django_file  # type: ignore # Assign the file object
-                # Save the instance - Django storage handles the file copy/move
-                raw_pdf.save()
-                _file = raw_pdf.file
-                assert _file is not None
-                logger.info(f"Created and saved new RawPdfFile {raw_pdf.pk} with file {_file.name}")
-
-                # Verify file exists in storage after save
-                if not _file.storage.exists(_file.name):
-                    logger.error(f"File was not saved correctly to storage path {_file.name} after model save.")
-                    raise IOError(f"File not found at expected storage path after save: {_file.name}")
-                # Log the absolute path for debugging if possible (depends on storage)
-                try:
-                    logger.info(f"File saved to absolute path: {_file.path}")
-                except NotImplementedError:
-                    logger.info(f"File saved to storage path: {_file.name} (Absolute path not available from storage)")
+                raw_pdf = cls.objects.create(
+                    pdf_hash=pdf_hash,
+                    center=center,
+                    file=django_file,
+                )
+
+            _file = raw_pdf.file
+            assert _file is not None
+            logger.info(
+                "Created and saved new RawPdfFile %s with file %s", raw_pdf.pk, _file.name
+            )
+
+            if not _file.storage.exists(_file.name):
+                logger.error(
+                    "File was not saved correctly to storage path %s after model save.",
+                    _file.name,
+                )
+                raise IOError(
+                    f"File not found at expected storage path after save: {_file.name}"
+                )
+
+            try:
+                logger.info("File saved to absolute path: %s", _file.path)
+            except NotImplementedError:
+                logger.info(
+                    "File saved to storage path: %s (Absolute path not available from storage)",
+                    _file.name,
+                )
 
         except Exception as e:
-            logger.error(f"Error processing or saving file {file_path} for new record: {e}")
-            # If save failed, the instance might be partially created but not fully saved.
-            raise  # Re-raise the exception
+            logger.error("Error processing or saving file %s for new record: %s", file_path, e)
+            raise
 
         # Delete source file *after* successful save and verification
         if delete_source:

{endoreg_db-0.8.2 → endoreg_db-0.8.2.2}/endoreg_db/models/media/video/create_from_file.py

@@ -6,7 +6,8 @@ from typing import TYPE_CHECKING, Optional, Type
 
 # Import the new exceptions from the correct path
 from endoreg_db.exceptions import InsufficientStorageError, TranscodingError
-from ...utils import VIDEO_DIR, TMP_VIDEO_DIR, data_paths
+from ...utils import VIDEO_DIR, TMP_VIDEO_DIR
+from importlib import import_module
 
 if TYPE_CHECKING:
     from endoreg_db.models import VideoFile
@@ -170,6 +171,22 @@ def atomic_move_with_fallback(src_path: Path, dst_path: Path) -> bool:
         raise
 
 
+def _get_data_paths():
+    """Return the current data_paths mapping (supports patched instances in tests)."""
+    utils_module = import_module("endoreg_db.utils")
+    return getattr(utils_module, "data_paths")
+
+
+def _get_path(mapping, key, default):
+    """Access mapping by key using __getitem__ so MagicMocks with side effects work."""
+    if mapping is None:
+        return default
+    try:
+        return mapping[key]
+    except (KeyError, TypeError):
+        return default
+
+
 def _create_from_file(
     cls_model: Type["VideoFile"],
     file_path: Path,
@@ -199,8 +216,12 @@ def _create_from_file(
 
     try:
         # Ensure we operate under the canonical video path root
-        video_dir = data_paths.get("video", video_dir)
-        storage_root = Path(video_dir).parent
+        data_paths = _get_data_paths()
+        resolved_video_dir = _get_path(data_paths, "video", video_dir)
+        video_dir = Path(resolved_video_dir)
+        storage_root_default = Path(video_dir).parent
+        resolved_storage_root = _get_path(data_paths, "storage", storage_root_default)
+        storage_root = Path(resolved_storage_root)
         storage_root.mkdir(parents=True, exist_ok=True)
         
         # Check storage capacity before starting any work
@@ -300,7 +321,8 @@ def _create_from_file(
         # 8. Create the VideoFile instance
         logger.info("Creating new VideoFile instance with UUID: %s", uuid_val)
         # Store FileField path relative to storage root including the videos prefix
-        relative_name = (final_storage_path.relative_to(data_paths['storage'])).as_posix()
+        storage_base = Path(_get_path(data_paths, "storage", final_storage_path.parent))
+        relative_name = (final_storage_path.relative_to(storage_base)).as_posix()
         video = cls_model(
             uuid=uuid_val,
             raw_file=relative_name,