endoreg-db 0.8.6.4__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of endoreg-db might be problematic. Click here for more details.

Files changed (793) hide show
  1. endoreg_db/__init__.py +0 -0
  2. endoreg_db/admin.py +92 -0
  3. endoreg_db/api/serializers/finding_descriptions.py +0 -0
  4. endoreg_db/api/views/finding_descriptions.py +0 -0
  5. endoreg_db/api_urls.py +4 -0
  6. endoreg_db/apps.py +18 -0
  7. endoreg_db/assets/dummy_model.ckpt +1 -0
  8. endoreg_db/codemods/readme.md +88 -0
  9. endoreg_db/codemods/rename_datetime_fields.py +92 -0
  10. endoreg_db/config/__init__.py +0 -0
  11. endoreg_db/config/env.py +101 -0
  12. endoreg_db/data/__init__.py +144 -0
  13. endoreg_db/data/ai_model/data.yaml +7 -0
  14. endoreg_db/data/ai_model_label/label/data.yaml +88 -0
  15. endoreg_db/data/ai_model_label/label/polyp_classification.yaml +52 -0
  16. endoreg_db/data/ai_model_label/label-set/data.yaml +40 -0
  17. endoreg_db/data/ai_model_label/label-set/polyp_classifications.yaml +25 -0
  18. endoreg_db/data/ai_model_label/label-type/data.yaml +7 -0
  19. endoreg_db/data/ai_model_meta/default_multilabel_classification.yaml +27 -0
  20. endoreg_db/data/ai_model_type/data.yaml +7 -0
  21. endoreg_db/data/ai_model_video_segmentation_label/base_segmentation.yaml +176 -0
  22. endoreg_db/data/ai_model_video_segmentation_labelset/data.yaml +20 -0
  23. endoreg_db/data/case_template/rule/00_patient_lab_sample_add_default_value.yaml +167 -0
  24. endoreg_db/data/case_template/rule/01_patient-set-age.yaml +8 -0
  25. endoreg_db/data/case_template/rule/01_patient-set-gender.yaml +9 -0
  26. endoreg_db/data/case_template/rule/11_create_patient_lab_sample.yaml +23 -0
  27. endoreg_db/data/case_template/rule/12_create-patient_medication-anticoagulation.yaml +19 -0
  28. endoreg_db/data/case_template/rule/13_create-patient_medication_schedule-anticoagulation.yaml +19 -0
  29. endoreg_db/data/case_template/rule/19_create_patient.yaml +17 -0
  30. endoreg_db/data/case_template/rule_type/base_types.yaml +35 -0
  31. endoreg_db/data/case_template/rule_value/.init +0 -0
  32. endoreg_db/data/case_template/rule_value_type/base_types.yaml +59 -0
  33. endoreg_db/data/case_template/template/base.yaml +8 -0
  34. endoreg_db/data/case_template/template_type/pre_endoscopy.yaml +3 -0
  35. endoreg_db/data/case_template/tmp/_rule_value +13 -0
  36. endoreg_db/data/case_template/tmp/rule/01_atrial_fibrillation.yaml +21 -0
  37. endoreg_db/data/case_template/tmp/rule/02_create_object.yaml +10 -0
  38. endoreg_db/data/case_template/tmp/template/atrial_fibrillation_low_risk.yaml +7 -0
  39. endoreg_db/data/center/data.yaml +91 -0
  40. endoreg_db/data/center_resource/green_endoscopy_dashboard_CenterResource.yaml +144 -0
  41. endoreg_db/data/center_shift/ukw.yaml +9 -0
  42. endoreg_db/data/center_waste/green_endoscopy_dashboard_CenterWaste.yaml +48 -0
  43. endoreg_db/data/contraindication/bleeding.yaml +11 -0
  44. endoreg_db/data/db_summary.csv +58 -0
  45. endoreg_db/data/db_summary.xlsx +0 -0
  46. endoreg_db/data/disease/cardiovascular.yaml +37 -0
  47. endoreg_db/data/disease/hepatology.yaml +5 -0
  48. endoreg_db/data/disease/misc.yaml +5 -0
  49. endoreg_db/data/disease/renal.yaml +5 -0
  50. endoreg_db/data/disease_classification/chronic_kidney_disease.yaml +6 -0
  51. endoreg_db/data/disease_classification/coronary_vessel_disease.yaml +6 -0
  52. endoreg_db/data/disease_classification_choice/chronic_kidney_disease.yaml +41 -0
  53. endoreg_db/data/disease_classification_choice/coronary_vessel_disease.yaml +20 -0
  54. endoreg_db/data/distribution/date/patient.yaml +7 -0
  55. endoreg_db/data/distribution/multiple_categorical/.init +0 -0
  56. endoreg_db/data/distribution/numeric/data.yaml +14 -0
  57. endoreg_db/data/distribution/single_categorical/patient.yaml +7 -0
  58. endoreg_db/data/emission_factor/green_endoscopy_dashboard_EmissionFactor.yaml +132 -0
  59. endoreg_db/data/endoscope/data.yaml +93 -0
  60. endoreg_db/data/endoscope_type/data.yaml +11 -0
  61. endoreg_db/data/endoscopy_processor/data.yaml +50 -0
  62. endoreg_db/data/event/cardiology.yaml +15 -0
  63. endoreg_db/data/event/neurology.yaml +14 -0
  64. endoreg_db/data/event/surgery.yaml +13 -0
  65. endoreg_db/data/event/thrombembolism.yaml +20 -0
  66. endoreg_db/data/examination/examinations/data.yaml +72 -0
  67. endoreg_db/data/examination/time/data.yaml +48 -0
  68. endoreg_db/data/examination/time-type/data.yaml +8 -0
  69. endoreg_db/data/examination/type/data.yaml +17 -0
  70. endoreg_db/data/examination_indication/endoscopy.yaml +424 -0
  71. endoreg_db/data/examination_indication_classification/endoscopy.yaml +160 -0
  72. endoreg_db/data/examination_indication_classification_choice/endoscopy.yaml +101 -0
  73. endoreg_db/data/examination_requirement_set/colonoscopy.yaml +15 -0
  74. endoreg_db/data/finding/anatomy_colon.yaml +128 -0
  75. endoreg_db/data/finding/colonoscopy.yaml +40 -0
  76. endoreg_db/data/finding/colonoscopy_bowel_prep.yaml +56 -0
  77. endoreg_db/data/finding/complication.yaml +16 -0
  78. endoreg_db/data/finding/data.yaml +105 -0
  79. endoreg_db/data/finding/examination_setting.yaml +16 -0
  80. endoreg_db/data/finding/medication_related.yaml +18 -0
  81. endoreg_db/data/finding/outcome.yaml +12 -0
  82. endoreg_db/data/finding_classification/colonoscopy_bowel_preparation.yaml +95 -0
  83. endoreg_db/data/finding_classification/colonoscopy_jnet.yaml +22 -0
  84. endoreg_db/data/finding_classification/colonoscopy_kudo.yaml +25 -0
  85. endoreg_db/data/finding_classification/colonoscopy_lesion_circularity.yaml +20 -0
  86. endoreg_db/data/finding_classification/colonoscopy_lesion_planarity.yaml +24 -0
  87. endoreg_db/data/finding_classification/colonoscopy_lesion_size.yaml +68 -0
  88. endoreg_db/data/finding_classification/colonoscopy_lesion_surface.yaml +20 -0
  89. endoreg_db/data/finding_classification/colonoscopy_location.yaml +80 -0
  90. endoreg_db/data/finding_classification/colonoscopy_lst.yaml +21 -0
  91. endoreg_db/data/finding_classification/colonoscopy_nice.yaml +20 -0
  92. endoreg_db/data/finding_classification/colonoscopy_paris.yaml +26 -0
  93. endoreg_db/data/finding_classification/colonoscopy_sano.yaml +22 -0
  94. endoreg_db/data/finding_classification/colonoscopy_summary.yaml +53 -0
  95. endoreg_db/data/finding_classification/complication_generic.yaml +25 -0
  96. endoreg_db/data/finding_classification/examination_setting_generic.yaml +40 -0
  97. endoreg_db/data/finding_classification/histology_colo.yaml +51 -0
  98. endoreg_db/data/finding_classification/intervention_required.yaml +26 -0
  99. endoreg_db/data/finding_classification/medication_related.yaml +23 -0
  100. endoreg_db/data/finding_classification/visualized.yaml +33 -0
  101. endoreg_db/data/finding_classification_choice/bowel_preparation.yaml +78 -0
  102. endoreg_db/data/finding_classification_choice/colon_lesion_circularity_default.yaml +32 -0
  103. endoreg_db/data/finding_classification_choice/colon_lesion_jnet.yaml +15 -0
  104. endoreg_db/data/finding_classification_choice/colon_lesion_kudo.yaml +23 -0
  105. endoreg_db/data/finding_classification_choice/colon_lesion_lst.yaml +15 -0
  106. endoreg_db/data/finding_classification_choice/colon_lesion_nice.yaml +17 -0
  107. endoreg_db/data/finding_classification_choice/colon_lesion_paris.yaml +57 -0
  108. endoreg_db/data/finding_classification_choice/colon_lesion_planarity_default.yaml +49 -0
  109. endoreg_db/data/finding_classification_choice/colon_lesion_sano.yaml +14 -0
  110. endoreg_db/data/finding_classification_choice/colon_lesion_surface_intact_default.yaml +36 -0
  111. endoreg_db/data/finding_classification_choice/colonoscopy_location.yaml +229 -0
  112. endoreg_db/data/finding_classification_choice/colonoscopy_not_complete_reason.yaml +19 -0
  113. endoreg_db/data/finding_classification_choice/colonoscopy_size.yaml +82 -0
  114. endoreg_db/data/finding_classification_choice/colonoscopy_summary_worst_finding.yaml +15 -0
  115. endoreg_db/data/finding_classification_choice/complication_generic_types.yaml +15 -0
  116. endoreg_db/data/finding_classification_choice/examination_setting_generic_types.yaml +15 -0
  117. endoreg_db/data/finding_classification_choice/histology.yaml +24 -0
  118. endoreg_db/data/finding_classification_choice/histology_polyp.yaml +20 -0
  119. endoreg_db/data/finding_classification_choice/outcome.yaml +19 -0
  120. endoreg_db/data/finding_classification_choice/yes_no_na.yaml +11 -0
  121. endoreg_db/data/finding_classification_type/colonoscopy_basic.yaml +48 -0
  122. endoreg_db/data/finding_intervention/endoscopy.yaml +43 -0
  123. endoreg_db/data/finding_intervention/endoscopy_colonoscopy.yaml +168 -0
  124. endoreg_db/data/finding_intervention/endoscopy_egd.yaml +128 -0
  125. endoreg_db/data/finding_intervention/endoscopy_ercp.yaml +32 -0
  126. endoreg_db/data/finding_intervention/endoscopy_eus_lower.yaml +9 -0
  127. endoreg_db/data/finding_intervention/endoscopy_eus_upper.yaml +36 -0
  128. endoreg_db/data/finding_intervention_type/endoscopy.yaml +15 -0
  129. endoreg_db/data/finding_morphology_classification_type/colonoscopy.yaml +79 -0
  130. endoreg_db/data/finding_type/data.yaml +43 -0
  131. endoreg_db/data/gender/data.yaml +42 -0
  132. endoreg_db/data/information_source/annotation.yaml +6 -0
  133. endoreg_db/data/information_source/data.yaml +30 -0
  134. endoreg_db/data/information_source/endoscopy_guidelines.yaml +7 -0
  135. endoreg_db/data/information_source/medication.yaml +6 -0
  136. endoreg_db/data/information_source/prediction.yaml +7 -0
  137. endoreg_db/data/information_source_type/data.yaml +8 -0
  138. endoreg_db/data/lab_value/cardiac_enzymes.yaml +37 -0
  139. endoreg_db/data/lab_value/coagulation.yaml +54 -0
  140. endoreg_db/data/lab_value/electrolytes.yaml +228 -0
  141. endoreg_db/data/lab_value/gastrointestinal_function.yaml +133 -0
  142. endoreg_db/data/lab_value/hematology.yaml +184 -0
  143. endoreg_db/data/lab_value/hormones.yaml +59 -0
  144. endoreg_db/data/lab_value/lipids.yaml +53 -0
  145. endoreg_db/data/lab_value/misc.yaml +76 -0
  146. endoreg_db/data/lab_value/renal_function.yaml +12 -0
  147. endoreg_db/data/log_type/data.yaml +57 -0
  148. endoreg_db/data/lx_client_tag/base.yaml +54 -0
  149. endoreg_db/data/lx_client_type/base.yaml +30 -0
  150. endoreg_db/data/lx_permission/base.yaml +24 -0
  151. endoreg_db/data/lx_permission/endoreg.yaml +52 -0
  152. endoreg_db/data/material/material.yaml +91 -0
  153. endoreg_db/data/medication/anticoagulation.yaml +65 -0
  154. endoreg_db/data/medication/tah.yaml +70 -0
  155. endoreg_db/data/medication_indication/anticoagulation.yaml +115 -0
  156. endoreg_db/data/medication_indication_type/data.yaml +11 -0
  157. endoreg_db/data/medication_indication_type/thrombembolism.yaml +41 -0
  158. endoreg_db/data/medication_intake_time/base.yaml +31 -0
  159. endoreg_db/data/medication_schedule/apixaban.yaml +95 -0
  160. endoreg_db/data/medication_schedule/ass.yaml +12 -0
  161. endoreg_db/data/medication_schedule/enoxaparin.yaml +26 -0
  162. endoreg_db/data/names_first/first_names.yaml +54 -0
  163. endoreg_db/data/names_last/last_names.yaml +51 -0
  164. endoreg_db/data/network_device/data.yaml +59 -0
  165. endoreg_db/data/network_device_type/data.yaml +12 -0
  166. endoreg_db/data/organ/data.yaml +29 -0
  167. endoreg_db/data/patient_lab_sample_type/generic.yaml +6 -0
  168. endoreg_db/data/pdf_type/data.yaml +46 -0
  169. endoreg_db/data/product/green_endoscopy_dashboard_Product.yaml +66 -0
  170. endoreg_db/data/product_group/green_endoscopy_dashboard_ProductGroup.yaml +33 -0
  171. endoreg_db/data/product_material/green_endoscopy_dashboard_ProductMaterial.yaml +308 -0
  172. endoreg_db/data/product_weight/green_endoscopy_dashboard_ProductWeight.yaml +88 -0
  173. endoreg_db/data/profession/data.yaml +70 -0
  174. endoreg_db/data/qualification/endoscopy.yaml +36 -0
  175. endoreg_db/data/qualification/m2.yaml +39 -0
  176. endoreg_db/data/qualification/outpatient_clinic.yaml +35 -0
  177. endoreg_db/data/qualification/sonography.yaml +36 -0
  178. endoreg_db/data/qualification_type/base.yaml +29 -0
  179. endoreg_db/data/reference_product/green_endoscopy_dashboard_ReferenceProduct.yaml +55 -0
  180. endoreg_db/data/report_reader_flag/rkh-histology-generic.yaml +10 -0
  181. endoreg_db/data/report_reader_flag/ukw-examination-generic.yaml +30 -0
  182. endoreg_db/data/report_reader_flag/ukw-histology-generic.yaml +24 -0
  183. endoreg_db/data/requirement/age.yaml +26 -0
  184. endoreg_db/data/requirement/colonoscopy_baseline_austria.yaml +45 -0
  185. endoreg_db/data/requirement/disease_cardiovascular.yaml +79 -0
  186. endoreg_db/data/requirement/disease_classification_choice_cardiovascular.yaml +41 -0
  187. endoreg_db/data/requirement/disease_hepatology.yaml +12 -0
  188. endoreg_db/data/requirement/disease_misc.yaml +12 -0
  189. endoreg_db/data/requirement/disease_renal.yaml +96 -0
  190. endoreg_db/data/requirement/endoscopy_bleeding_risk.yaml +59 -0
  191. endoreg_db/data/requirement/event_cardiology.yaml +251 -0
  192. endoreg_db/data/requirement/event_requirements.yaml +145 -0
  193. endoreg_db/data/requirement/finding_colon_polyp.yaml +50 -0
  194. endoreg_db/data/requirement/gender.yaml +25 -0
  195. endoreg_db/data/requirement/lab_value.yaml +441 -0
  196. endoreg_db/data/requirement/medication.yaml +93 -0
  197. endoreg_db/data/requirement_operator/age.yaml +13 -0
  198. endoreg_db/data/requirement_operator/lab_operators.yaml +129 -0
  199. endoreg_db/data/requirement_operator/model_operators.yaml +96 -0
  200. endoreg_db/data/requirement_set/01_endoscopy_generic.yaml +48 -0
  201. endoreg_db/data/requirement_set/colonoscopy_austria_screening.yaml +57 -0
  202. endoreg_db/data/requirement_set/endoscopy_bleeding_risk.yaml +52 -0
  203. endoreg_db/data/requirement_set_type/data.yaml +20 -0
  204. endoreg_db/data/requirement_type/requirement_types.yaml +165 -0
  205. endoreg_db/data/resource/green_endoscopy_dashboard_Resource.yaml +15 -0
  206. endoreg_db/data/risk/bleeding.yaml +26 -0
  207. endoreg_db/data/risk/thrombosis.yaml +37 -0
  208. endoreg_db/data/risk_type/data.yaml +27 -0
  209. endoreg_db/data/setup_config.yaml +38 -0
  210. endoreg_db/data/shift/endoscopy.yaml +21 -0
  211. endoreg_db/data/shift/m2.yaml +0 -0
  212. endoreg_db/data/shift_type/base.yaml +35 -0
  213. endoreg_db/data/tag/requirement_set_tags.yaml +11 -0
  214. endoreg_db/data/tmp/chronic_kidney_disease.yaml +0 -0
  215. endoreg_db/data/tmp/congestive_heart_failure.yaml +0 -0
  216. endoreg_db/data/transport_route/green_endoscopy_dashboard_TransportRoute.yaml +12 -0
  217. endoreg_db/data/unit/concentration.yaml +115 -0
  218. endoreg_db/data/unit/data.yaml +17 -0
  219. endoreg_db/data/unit/length.yaml +31 -0
  220. endoreg_db/data/unit/misc.yaml +20 -0
  221. endoreg_db/data/unit/rate.yaml +6 -0
  222. endoreg_db/data/unit/time.yaml +48 -0
  223. endoreg_db/data/unit/volume.yaml +35 -0
  224. endoreg_db/data/unit/weight.yaml +38 -0
  225. endoreg_db/data/waste/data.yaml +12 -0
  226. endoreg_db/exceptions.py +19 -0
  227. endoreg_db/factories/__init__.py +0 -0
  228. endoreg_db/forms/__init__.py +5 -0
  229. endoreg_db/forms/examination_form.py +11 -0
  230. endoreg_db/forms/patient_finding_intervention_form.py +18 -0
  231. endoreg_db/forms/patient_form.py +27 -0
  232. endoreg_db/forms/questionnaires/__init__.py +1 -0
  233. endoreg_db/forms/questionnaires/tto_questionnaire.py +23 -0
  234. endoreg_db/forms/settings/__init__.py +8 -0
  235. endoreg_db/forms/unit.py +6 -0
  236. endoreg_db/helpers/__init__.py +0 -0
  237. endoreg_db/helpers/count_db.py +45 -0
  238. endoreg_db/helpers/data_loader.py +208 -0
  239. endoreg_db/helpers/default_objects.py +378 -0
  240. endoreg_db/helpers/download_segmentation_model.py +31 -0
  241. endoreg_db/helpers/interact.py +6 -0
  242. endoreg_db/helpers/test_video_helper.py +119 -0
  243. endoreg_db/logger_conf.py +140 -0
  244. endoreg_db/management/__init__.py +1 -0
  245. endoreg_db/management/commands/__init__.py +1 -0
  246. endoreg_db/management/commands/anonymize_video.py +0 -0
  247. endoreg_db/management/commands/check_auth.py +125 -0
  248. endoreg_db/management/commands/create_model_meta_from_huggingface.py +115 -0
  249. endoreg_db/management/commands/create_multilabel_model_meta.py +214 -0
  250. endoreg_db/management/commands/fix_missing_patient_data.py +172 -0
  251. endoreg_db/management/commands/fix_video_paths.py +165 -0
  252. endoreg_db/management/commands/import_fallback_video.py +203 -0
  253. endoreg_db/management/commands/import_report.py +298 -0
  254. endoreg_db/management/commands/import_video.py +423 -0
  255. endoreg_db/management/commands/import_video_with_classification.py +367 -0
  256. endoreg_db/management/commands/init_default_ai_model.py +112 -0
  257. endoreg_db/management/commands/load_ai_model_data.py +77 -0
  258. endoreg_db/management/commands/load_ai_model_label_data.py +59 -0
  259. endoreg_db/management/commands/load_base_db_data.py +192 -0
  260. endoreg_db/management/commands/load_center_data.py +68 -0
  261. endoreg_db/management/commands/load_contraindication_data.py +41 -0
  262. endoreg_db/management/commands/load_disease_classification_choices_data.py +41 -0
  263. endoreg_db/management/commands/load_disease_classification_data.py +41 -0
  264. endoreg_db/management/commands/load_disease_data.py +62 -0
  265. endoreg_db/management/commands/load_distribution_data.py +66 -0
  266. endoreg_db/management/commands/load_endoscope_data.py +68 -0
  267. endoreg_db/management/commands/load_event_data.py +41 -0
  268. endoreg_db/management/commands/load_examination_data.py +75 -0
  269. endoreg_db/management/commands/load_examination_indication_data.py +86 -0
  270. endoreg_db/management/commands/load_finding_data.py +128 -0
  271. endoreg_db/management/commands/load_gender_data.py +44 -0
  272. endoreg_db/management/commands/load_green_endoscopy_wuerzburg_data.py +132 -0
  273. endoreg_db/management/commands/load_information_source.py +51 -0
  274. endoreg_db/management/commands/load_lab_value_data.py +50 -0
  275. endoreg_db/management/commands/load_medication_data.py +103 -0
  276. endoreg_db/management/commands/load_name_data.py +37 -0
  277. endoreg_db/management/commands/load_organ_data.py +43 -0
  278. endoreg_db/management/commands/load_pdf_type_data.py +61 -0
  279. endoreg_db/management/commands/load_profession_data.py +44 -0
  280. endoreg_db/management/commands/load_qualification_data.py +59 -0
  281. endoreg_db/management/commands/load_report_reader_flag_data.py +46 -0
  282. endoreg_db/management/commands/load_requirement_data.py +180 -0
  283. endoreg_db/management/commands/load_risk_data.py +56 -0
  284. endoreg_db/management/commands/load_shift_data.py +60 -0
  285. endoreg_db/management/commands/load_tag_data.py +57 -0
  286. endoreg_db/management/commands/load_unit_data.py +46 -0
  287. endoreg_db/management/commands/load_user_groups.py +28 -0
  288. endoreg_db/management/commands/register_ai_model.py +64 -0
  289. endoreg_db/management/commands/reset_celery_schedule.py +9 -0
  290. endoreg_db/management/commands/setup_endoreg_db.py +381 -0
  291. endoreg_db/management/commands/start_filewatcher.py +106 -0
  292. endoreg_db/management/commands/storage_management.py +548 -0
  293. endoreg_db/management/commands/summarize_db_content.py +189 -0
  294. endoreg_db/management/commands/validate_video.py +204 -0
  295. endoreg_db/management/commands/validate_video_files.py +161 -0
  296. endoreg_db/management/commands/video_validation.py +22 -0
  297. endoreg_db/mermaid/Overall_flow_patient_finding_intervention.md +10 -0
  298. endoreg_db/mermaid/anonymized_image_annotation.md +20 -0
  299. endoreg_db/mermaid/binary_classification_annotation.md +50 -0
  300. endoreg_db/mermaid/classification.md +8 -0
  301. endoreg_db/mermaid/examination.md +8 -0
  302. endoreg_db/mermaid/findings.md +7 -0
  303. endoreg_db/mermaid/image_classification.md +28 -0
  304. endoreg_db/mermaid/interventions.md +8 -0
  305. endoreg_db/mermaid/morphology.md +8 -0
  306. endoreg_db/mermaid/patient_creation.md +14 -0
  307. endoreg_db/mermaid/video_segmentation_annotation.md +17 -0
  308. endoreg_db/migrations/0001_initial.py +1857 -0
  309. endoreg_db/migrations/0002_add_video_correction_models.py +52 -0
  310. endoreg_db/migrations/0003_add_center_display_name.py +30 -0
  311. endoreg_db/migrations/__init__.py +0 -0
  312. endoreg_db/models/__init__.py +359 -0
  313. endoreg_db/models/administration/__init__.py +116 -0
  314. endoreg_db/models/administration/ai/__init__.py +9 -0
  315. endoreg_db/models/administration/ai/active_model.py +35 -0
  316. endoreg_db/models/administration/ai/ai_model.py +156 -0
  317. endoreg_db/models/administration/ai/model_type.py +41 -0
  318. endoreg_db/models/administration/case/__init__.py +19 -0
  319. endoreg_db/models/administration/case/case.py +114 -0
  320. endoreg_db/models/administration/case/case_template/__init__.py +15 -0
  321. endoreg_db/models/administration/case/case_template/case_template.py +125 -0
  322. endoreg_db/models/administration/case/case_template/case_template_rule.py +269 -0
  323. endoreg_db/models/administration/case/case_template/case_template_rule_value.py +86 -0
  324. endoreg_db/models/administration/case/case_template/case_template_type.py +26 -0
  325. endoreg_db/models/administration/center/__init__.py +13 -0
  326. endoreg_db/models/administration/center/center.py +67 -0
  327. endoreg_db/models/administration/center/center_product.py +64 -0
  328. endoreg_db/models/administration/center/center_resource.py +49 -0
  329. endoreg_db/models/administration/center/center_shift.py +88 -0
  330. endoreg_db/models/administration/center/center_waste.py +30 -0
  331. endoreg_db/models/administration/permissions/__init__.py +44 -0
  332. endoreg_db/models/administration/person/__init__.py +24 -0
  333. endoreg_db/models/administration/person/employee/__init__.py +3 -0
  334. endoreg_db/models/administration/person/employee/employee.py +35 -0
  335. endoreg_db/models/administration/person/employee/employee_qualification.py +39 -0
  336. endoreg_db/models/administration/person/employee/employee_type.py +42 -0
  337. endoreg_db/models/administration/person/examiner/__init__.py +4 -0
  338. endoreg_db/models/administration/person/examiner/examiner.py +54 -0
  339. endoreg_db/models/administration/person/names/__init__.py +0 -0
  340. endoreg_db/models/administration/person/names/first_name.py +18 -0
  341. endoreg_db/models/administration/person/names/last_name.py +19 -0
  342. endoreg_db/models/administration/person/patient/__init__.py +5 -0
  343. endoreg_db/models/administration/person/patient/patient.py +460 -0
  344. endoreg_db/models/administration/person/person.py +31 -0
  345. endoreg_db/models/administration/person/profession/__init__.py +24 -0
  346. endoreg_db/models/administration/person/user/__init__.py +5 -0
  347. endoreg_db/models/administration/person/user/portal_user_information.py +37 -0
  348. endoreg_db/models/administration/product/__init__.py +14 -0
  349. endoreg_db/models/administration/product/product.py +97 -0
  350. endoreg_db/models/administration/product/product_group.py +39 -0
  351. endoreg_db/models/administration/product/product_material.py +54 -0
  352. endoreg_db/models/administration/product/product_weight.py +47 -0
  353. endoreg_db/models/administration/product/reference_product.py +130 -0
  354. endoreg_db/models/administration/qualification/__init__.py +7 -0
  355. endoreg_db/models/administration/qualification/qualification.py +37 -0
  356. endoreg_db/models/administration/qualification/qualification_type.py +35 -0
  357. endoreg_db/models/administration/shift/__init__.py +9 -0
  358. endoreg_db/models/administration/shift/scheduled_days.py +69 -0
  359. endoreg_db/models/administration/shift/shift.py +51 -0
  360. endoreg_db/models/administration/shift/shift_type.py +108 -0
  361. endoreg_db/models/label/__init__.py +24 -0
  362. endoreg_db/models/label/annotation/__init__.py +12 -0
  363. endoreg_db/models/label/annotation/image_classification.py +84 -0
  364. endoreg_db/models/label/annotation/video_segmentation_annotation.py +66 -0
  365. endoreg_db/models/label/label.py +83 -0
  366. endoreg_db/models/label/label_set.py +53 -0
  367. endoreg_db/models/label/label_type.py +29 -0
  368. endoreg_db/models/label/label_video_segment/__init__.py +3 -0
  369. endoreg_db/models/label/label_video_segment/_create_from_video.py +41 -0
  370. endoreg_db/models/label/label_video_segment/label_video_segment.py +511 -0
  371. endoreg_db/models/label/video_segmentation_label.py +31 -0
  372. endoreg_db/models/label/video_segmentation_labelset.py +27 -0
  373. endoreg_db/models/media/__init__.py +16 -0
  374. endoreg_db/models/media/frame/__init__.py +3 -0
  375. endoreg_db/models/media/frame/frame.py +111 -0
  376. endoreg_db/models/media/pdf/__init__.py +11 -0
  377. endoreg_db/models/media/pdf/raw_pdf.py +757 -0
  378. endoreg_db/models/media/pdf/report_file.py +162 -0
  379. endoreg_db/models/media/pdf/report_reader/__init__.py +7 -0
  380. endoreg_db/models/media/pdf/report_reader/report_reader_config.py +77 -0
  381. endoreg_db/models/media/pdf/report_reader/report_reader_flag.py +20 -0
  382. endoreg_db/models/media/video/__init__.py +8 -0
  383. endoreg_db/models/media/video/create_from_file.py +358 -0
  384. endoreg_db/models/media/video/pipe_1.py +213 -0
  385. endoreg_db/models/media/video/pipe_2.py +105 -0
  386. endoreg_db/models/media/video/refactor_plan.md +0 -0
  387. endoreg_db/models/media/video/video_file.py +825 -0
  388. endoreg_db/models/media/video/video_file_ai.py +443 -0
  389. endoreg_db/models/media/video/video_file_anonymize.py +349 -0
  390. endoreg_db/models/media/video/video_file_frames/__init__.py +47 -0
  391. endoreg_db/models/media/video/video_file_frames/_bulk_create_frames.py +22 -0
  392. endoreg_db/models/media/video/video_file_frames/_create_frame_object.py +23 -0
  393. endoreg_db/models/media/video/video_file_frames/_delete_frames.py +104 -0
  394. endoreg_db/models/media/video/video_file_frames/_extract_frames.py +174 -0
  395. endoreg_db/models/media/video/video_file_frames/_get_frame.py +28 -0
  396. endoreg_db/models/media/video/video_file_frames/_get_frame_number.py +27 -0
  397. endoreg_db/models/media/video/video_file_frames/_get_frame_path.py +20 -0
  398. endoreg_db/models/media/video/video_file_frames/_get_frame_paths.py +27 -0
  399. endoreg_db/models/media/video/video_file_frames/_get_frame_range.py +34 -0
  400. endoreg_db/models/media/video/video_file_frames/_get_frames.py +27 -0
  401. endoreg_db/models/media/video/video_file_frames/_initialize_frames.py +129 -0
  402. endoreg_db/models/media/video/video_file_frames/_manage_frame_range.py +141 -0
  403. endoreg_db/models/media/video/video_file_frames/_mark_frames_extracted_status.py +65 -0
  404. endoreg_db/models/media/video/video_file_frames.py +0 -0
  405. endoreg_db/models/media/video/video_file_io.py +168 -0
  406. endoreg_db/models/media/video/video_file_meta/__init__.py +22 -0
  407. endoreg_db/models/media/video/video_file_meta/get_crop_template.py +45 -0
  408. endoreg_db/models/media/video/video_file_meta/get_endo_roi.py +39 -0
  409. endoreg_db/models/media/video/video_file_meta/get_fps.py +147 -0
  410. endoreg_db/models/media/video/video_file_meta/initialize_video_specs.py +143 -0
  411. endoreg_db/models/media/video/video_file_meta/text_meta.py +134 -0
  412. endoreg_db/models/media/video/video_file_meta/video_meta.py +70 -0
  413. endoreg_db/models/media/video/video_file_segments.py +209 -0
  414. endoreg_db/models/media/video/video_metadata.py +65 -0
  415. endoreg_db/models/media/video/video_processing.py +152 -0
  416. endoreg_db/models/medical/__init__.py +146 -0
  417. endoreg_db/models/medical/contraindication/__init__.py +17 -0
  418. endoreg_db/models/medical/disease.py +156 -0
  419. endoreg_db/models/medical/event.py +137 -0
  420. endoreg_db/models/medical/examination/__init__.py +9 -0
  421. endoreg_db/models/medical/examination/examination.py +148 -0
  422. endoreg_db/models/medical/examination/examination_indication.py +278 -0
  423. endoreg_db/models/medical/examination/examination_time.py +49 -0
  424. endoreg_db/models/medical/examination/examination_time_type.py +41 -0
  425. endoreg_db/models/medical/examination/examination_type.py +48 -0
  426. endoreg_db/models/medical/finding/__init__.py +18 -0
  427. endoreg_db/models/medical/finding/finding.py +96 -0
  428. endoreg_db/models/medical/finding/finding_classification.py +142 -0
  429. endoreg_db/models/medical/finding/finding_intervention.py +52 -0
  430. endoreg_db/models/medical/finding/finding_type.py +35 -0
  431. endoreg_db/models/medical/hardware/__init__.py +8 -0
  432. endoreg_db/models/medical/hardware/endoscope.py +65 -0
  433. endoreg_db/models/medical/hardware/endoscopy_processor.py +182 -0
  434. endoreg_db/models/medical/laboratory/__init__.py +5 -0
  435. endoreg_db/models/medical/laboratory/lab_value.py +419 -0
  436. endoreg_db/models/medical/medication/__init__.py +19 -0
  437. endoreg_db/models/medical/medication/medication.py +31 -0
  438. endoreg_db/models/medical/medication/medication_indication.py +50 -0
  439. endoreg_db/models/medical/medication/medication_indication_type.py +39 -0
  440. endoreg_db/models/medical/medication/medication_intake_time.py +44 -0
  441. endoreg_db/models/medical/medication/medication_schedule.py +45 -0
  442. endoreg_db/models/medical/organ/__init__.py +35 -0
  443. endoreg_db/models/medical/patient/__init__.py +56 -0
  444. endoreg_db/models/medical/patient/medication_examples.py +38 -0
  445. endoreg_db/models/medical/patient/patient_disease.py +63 -0
  446. endoreg_db/models/medical/patient/patient_event.py +75 -0
  447. endoreg_db/models/medical/patient/patient_examination.py +249 -0
  448. endoreg_db/models/medical/patient/patient_examination_indication.py +44 -0
  449. endoreg_db/models/medical/patient/patient_finding.py +357 -0
  450. endoreg_db/models/medical/patient/patient_finding_classification.py +207 -0
  451. endoreg_db/models/medical/patient/patient_finding_intervention.py +40 -0
  452. endoreg_db/models/medical/patient/patient_lab_sample.py +148 -0
  453. endoreg_db/models/medical/patient/patient_lab_value.py +222 -0
  454. endoreg_db/models/medical/patient/patient_medication.py +104 -0
  455. endoreg_db/models/medical/patient/patient_medication_schedule.py +136 -0
  456. endoreg_db/models/medical/risk/__init__.py +7 -0
  457. endoreg_db/models/medical/risk/risk.py +72 -0
  458. endoreg_db/models/medical/risk/risk_type.py +51 -0
  459. endoreg_db/models/metadata/__init__.py +19 -0
  460. endoreg_db/models/metadata/frame_ocr_result.py +0 -0
  461. endoreg_db/models/metadata/model_meta.py +206 -0
  462. endoreg_db/models/metadata/model_meta_logic.py +343 -0
  463. endoreg_db/models/metadata/pdf_meta.py +89 -0
  464. endoreg_db/models/metadata/sensitive_meta.py +288 -0
  465. endoreg_db/models/metadata/sensitive_meta_logic.py +1048 -0
  466. endoreg_db/models/metadata/video_meta.py +332 -0
  467. endoreg_db/models/metadata/video_prediction_logic.py +190 -0
  468. endoreg_db/models/metadata/video_prediction_meta.py +270 -0
  469. endoreg_db/models/other/__init__.py +40 -0
  470. endoreg_db/models/other/distribution/__init__.py +44 -0
  471. endoreg_db/models/other/distribution/base_value_distribution.py +20 -0
  472. endoreg_db/models/other/distribution/date_value_distribution.py +89 -0
  473. endoreg_db/models/other/distribution/multiple_categorical_value_distribution.py +32 -0
  474. endoreg_db/models/other/distribution/numeric_value_distribution.py +125 -0
  475. endoreg_db/models/other/distribution/single_categorical_value_distribution.py +22 -0
  476. endoreg_db/models/other/emission/__init__.py +5 -0
  477. endoreg_db/models/other/emission/emission_factor.py +94 -0
  478. endoreg_db/models/other/gender.py +27 -0
  479. endoreg_db/models/other/information_source.py +159 -0
  480. endoreg_db/models/other/material.py +28 -0
  481. endoreg_db/models/other/resource.py +22 -0
  482. endoreg_db/models/other/tag.py +27 -0
  483. endoreg_db/models/other/transport_route.py +33 -0
  484. endoreg_db/models/other/unit.py +32 -0
  485. endoreg_db/models/other/waste.py +27 -0
  486. endoreg_db/models/requirement/__init__.py +11 -0
  487. endoreg_db/models/requirement/requirement.py +767 -0
  488. endoreg_db/models/requirement/requirement_evaluation/__init__.py +6 -0
  489. endoreg_db/models/requirement/requirement_evaluation/get_values.py +40 -0
  490. endoreg_db/models/requirement/requirement_evaluation/operator_evaluation_models.py +9 -0
  491. endoreg_db/models/requirement/requirement_evaluation/requirement_type_parser.py +95 -0
  492. endoreg_db/models/requirement/requirement_operator.py +176 -0
  493. endoreg_db/models/requirement/requirement_set.py +287 -0
  494. endoreg_db/models/rule/__init__.py +13 -0
  495. endoreg_db/models/rule/rule.py +27 -0
  496. endoreg_db/models/rule/rule_applicator.py +224 -0
  497. endoreg_db/models/rule/rule_attribute_dtype.py +17 -0
  498. endoreg_db/models/rule/rule_type.py +20 -0
  499. endoreg_db/models/rule/ruleset.py +17 -0
  500. endoreg_db/models/state/__init__.py +12 -0
  501. endoreg_db/models/state/abstract.py +11 -0
  502. endoreg_db/models/state/audit_ledger.py +150 -0
  503. endoreg_db/models/state/label_video_segment.py +22 -0
  504. endoreg_db/models/state/raw_pdf.py +187 -0
  505. endoreg_db/models/state/sensitive_meta.py +46 -0
  506. endoreg_db/models/state/video.py +232 -0
  507. endoreg_db/models/upload_job.py +99 -0
  508. endoreg_db/models/utils.py +135 -0
  509. endoreg_db/queries/__init__.py +5 -0
  510. endoreg_db/queries/annotations/__init__.py +3 -0
  511. endoreg_db/queries/annotations/legacy.py +158 -0
  512. endoreg_db/queries/sanity/__init_.py +0 -0
  513. endoreg_db/renames.yml +8 -0
  514. endoreg_db/root_urls.py +9 -0
  515. endoreg_db/schemas/__init__.py +0 -0
  516. endoreg_db/schemas/examination_evaluation.py +27 -0
  517. endoreg_db/serializers/Frames_NICE_and_PARIS_classifications.py +775 -0
  518. endoreg_db/serializers/__init__.py +118 -0
  519. endoreg_db/serializers/_old/raw_pdf_meta_validation.py +223 -0
  520. endoreg_db/serializers/_old/raw_video_meta_validation.py +179 -0
  521. endoreg_db/serializers/_old/video.py +71 -0
  522. endoreg_db/serializers/administration/__init__.py +14 -0
  523. endoreg_db/serializers/administration/ai/__init__.py +10 -0
  524. endoreg_db/serializers/administration/ai/active_model.py +10 -0
  525. endoreg_db/serializers/administration/ai/ai_model.py +18 -0
  526. endoreg_db/serializers/administration/ai/model_type.py +10 -0
  527. endoreg_db/serializers/administration/center.py +9 -0
  528. endoreg_db/serializers/administration/gender.py +9 -0
  529. endoreg_db/serializers/anonymization.py +69 -0
  530. endoreg_db/serializers/evaluation/examination_evaluation.py +1 -0
  531. endoreg_db/serializers/examination/__init__.py +10 -0
  532. endoreg_db/serializers/examination/base.py +46 -0
  533. endoreg_db/serializers/examination/dropdown.py +21 -0
  534. endoreg_db/serializers/examination_serializer.py +12 -0
  535. endoreg_db/serializers/finding/__init__.py +5 -0
  536. endoreg_db/serializers/finding/finding.py +54 -0
  537. endoreg_db/serializers/finding_classification/__init__.py +7 -0
  538. endoreg_db/serializers/finding_classification/choice.py +19 -0
  539. endoreg_db/serializers/finding_classification/classification.py +13 -0
  540. endoreg_db/serializers/label/__init__.py +7 -0
  541. endoreg_db/serializers/label/image_classification_annotation.py +62 -0
  542. endoreg_db/serializers/label/label.py +15 -0
  543. endoreg_db/serializers/label_video_segment/__init__.py +7 -0
  544. endoreg_db/serializers/label_video_segment/_lvs_create.py +149 -0
  545. endoreg_db/serializers/label_video_segment/_lvs_update.py +138 -0
  546. endoreg_db/serializers/label_video_segment/_lvs_validate.py +149 -0
  547. endoreg_db/serializers/label_video_segment/label_video_segment.py +344 -0
  548. endoreg_db/serializers/label_video_segment/label_video_segment_annotation.py +99 -0
  549. endoreg_db/serializers/label_video_segment/label_video_segment_update.py +163 -0
  550. endoreg_db/serializers/meta/__init__.py +19 -0
  551. endoreg_db/serializers/meta/pdf_file_meta_extraction.py +115 -0
  552. endoreg_db/serializers/meta/report_meta.py +53 -0
  553. endoreg_db/serializers/meta/sensitive_meta_detail.py +162 -0
  554. endoreg_db/serializers/meta/sensitive_meta_update.py +148 -0
  555. endoreg_db/serializers/meta/sensitive_meta_verification.py +59 -0
  556. endoreg_db/serializers/meta/video_meta.py +39 -0
  557. endoreg_db/serializers/misc/__init__.py +14 -0
  558. endoreg_db/serializers/misc/file_overview.py +182 -0
  559. endoreg_db/serializers/misc/sensitive_patient_data.py +120 -0
  560. endoreg_db/serializers/misc/stats.py +33 -0
  561. endoreg_db/serializers/misc/translatable_field_mix_in.py +44 -0
  562. endoreg_db/serializers/misc/upload_job.py +71 -0
  563. endoreg_db/serializers/patient/__init__.py +11 -0
  564. endoreg_db/serializers/patient/patient.py +86 -0
  565. endoreg_db/serializers/patient/patient_dropdown.py +27 -0
  566. endoreg_db/serializers/patient_examination/__init__.py +7 -0
  567. endoreg_db/serializers/patient_examination/patient_examination.py +141 -0
  568. endoreg_db/serializers/patient_finding/__init__.py +15 -0
  569. endoreg_db/serializers/patient_finding/patient_finding.py +31 -0
  570. endoreg_db/serializers/patient_finding/patient_finding_classification.py +39 -0
  571. endoreg_db/serializers/patient_finding/patient_finding_detail.py +53 -0
  572. endoreg_db/serializers/patient_finding/patient_finding_intervention.py +26 -0
  573. endoreg_db/serializers/patient_finding/patient_finding_list.py +41 -0
  574. endoreg_db/serializers/patient_finding/patient_finding_write.py +126 -0
  575. endoreg_db/serializers/pdf/__init__.py +5 -0
  576. endoreg_db/serializers/pdf/anony_text_validation.py +85 -0
  577. endoreg_db/serializers/report/__init__.py +9 -0
  578. endoreg_db/serializers/report/mixins.py +45 -0
  579. endoreg_db/serializers/report/report.py +105 -0
  580. endoreg_db/serializers/report/report_list.py +22 -0
  581. endoreg_db/serializers/report/secure_file_url.py +26 -0
  582. endoreg_db/serializers/requirements/requirement_schema.py +25 -0
  583. endoreg_db/serializers/requirements/requirement_sets.py +29 -0
  584. endoreg_db/serializers/sensitive_meta_serializer.py +282 -0
  585. endoreg_db/serializers/video/__init__.py +7 -0
  586. endoreg_db/serializers/video/segmentation.py +263 -0
  587. endoreg_db/serializers/video/video_file_brief.py +10 -0
  588. endoreg_db/serializers/video/video_file_detail.py +83 -0
  589. endoreg_db/serializers/video/video_file_list.py +67 -0
  590. endoreg_db/serializers/video/video_metadata.py +105 -0
  591. endoreg_db/serializers/video/video_processing_history.py +153 -0
  592. endoreg_db/serializers/video_examination.py +198 -0
  593. endoreg_db/services/__init__.py +5 -0
  594. endoreg_db/services/anonymization.py +223 -0
  595. endoreg_db/services/examination_evaluation.py +149 -0
  596. endoreg_db/services/finding_description_service.py +0 -0
  597. endoreg_db/services/lookup_service.py +411 -0
  598. endoreg_db/services/lookup_store.py +266 -0
  599. endoreg_db/services/pdf_import.py +1382 -0
  600. endoreg_db/services/polling_coordinator.py +288 -0
  601. endoreg_db/services/pseudonym_service.py +89 -0
  602. endoreg_db/services/requirements_object.py +147 -0
  603. endoreg_db/services/segment_sync.py +155 -0
  604. endoreg_db/services/storage_aware_video_processor.py +344 -0
  605. endoreg_db/services/video_import.py +1259 -0
  606. endoreg_db/tasks/upload_tasks.py +207 -0
  607. endoreg_db/tasks/video_ingest.py +157 -0
  608. endoreg_db/tasks/video_processing_tasks.py +327 -0
  609. endoreg_db/templates/admin/patient_finding_intervention.html +253 -0
  610. endoreg_db/templates/admin/start_examination.html +12 -0
  611. endoreg_db/templates/timeline.html +176 -0
  612. endoreg_db/urls/__init__.py +83 -0
  613. endoreg_db/urls/anonymization.py +32 -0
  614. endoreg_db/urls/auth.py +16 -0
  615. endoreg_db/urls/classification.py +39 -0
  616. endoreg_db/urls/examination.py +54 -0
  617. endoreg_db/urls/files.py +6 -0
  618. endoreg_db/urls/label_video_segment_validate.py +33 -0
  619. endoreg_db/urls/label_video_segments.py +46 -0
  620. endoreg_db/urls/media.py +227 -0
  621. endoreg_db/urls/patient.py +19 -0
  622. endoreg_db/urls/report.py +48 -0
  623. endoreg_db/urls/requirements.py +13 -0
  624. endoreg_db/urls/sensitive_meta.py +0 -0
  625. endoreg_db/urls/stats.py +46 -0
  626. endoreg_db/urls/upload.py +20 -0
  627. endoreg_db/urls/video.py +61 -0
  628. endoreg_db/urls.py +9 -0
  629. endoreg_db/utils/__init__.py +88 -0
  630. endoreg_db/utils/ai/__init__.py +9 -0
  631. endoreg_db/utils/ai/get.py +5 -0
  632. endoreg_db/utils/ai/inference_dataset.py +52 -0
  633. endoreg_db/utils/ai/multilabel_classification_net.py +159 -0
  634. endoreg_db/utils/ai/postprocess.py +63 -0
  635. endoreg_db/utils/ai/predict.py +291 -0
  636. endoreg_db/utils/ai/preprocess.py +68 -0
  637. endoreg_db/utils/calc_duration_seconds.py +24 -0
  638. endoreg_db/utils/case_generator/__init__.py +0 -0
  639. endoreg_db/utils/case_generator/case_generator.py +159 -0
  640. endoreg_db/utils/case_generator/lab_sample_factory.py +33 -0
  641. endoreg_db/utils/case_generator/utils.py +30 -0
  642. endoreg_db/utils/check_video_files.py +148 -0
  643. endoreg_db/utils/cropping.py +29 -0
  644. endoreg_db/utils/dataloader.py +175 -0
  645. endoreg_db/utils/dates.py +60 -0
  646. endoreg_db/utils/env.py +33 -0
  647. endoreg_db/utils/extract_specific_frames.py +72 -0
  648. endoreg_db/utils/file_operations.py +58 -0
  649. endoreg_db/utils/fix_video_path_direct.py +141 -0
  650. endoreg_db/utils/frame_anonymization_utils.py +463 -0
  651. endoreg_db/utils/hashs.py +153 -0
  652. endoreg_db/utils/links/__init__.py +0 -0
  653. endoreg_db/utils/links/requirement_link.py +193 -0
  654. endoreg_db/utils/mime_types.py +0 -0
  655. endoreg_db/utils/names.py +76 -0
  656. endoreg_db/utils/ocr.py +190 -0
  657. endoreg_db/utils/parse_and_generate_yaml.py +46 -0
  658. endoreg_db/utils/paths.py +95 -0
  659. endoreg_db/utils/permissions.py +143 -0
  660. endoreg_db/utils/pipelines/Readme.md +235 -0
  661. endoreg_db/utils/pipelines/__init__.py +0 -0
  662. endoreg_db/utils/pipelines/process_video_dir.py +120 -0
  663. endoreg_db/utils/product/__init__.py +0 -0
  664. endoreg_db/utils/product/sum_emissions.py +20 -0
  665. endoreg_db/utils/product/sum_weights.py +18 -0
  666. endoreg_db/utils/pydantic_models/__init__.py +6 -0
  667. endoreg_db/utils/pydantic_models/db_config.py +57 -0
  668. endoreg_db/utils/requirement_helpers.py +0 -0
  669. endoreg_db/utils/requirement_operator_logic/__init__.py +0 -0
  670. endoreg_db/utils/requirement_operator_logic/lab_value_operators.py +578 -0
  671. endoreg_db/utils/requirement_operator_logic/model_evaluators.py +368 -0
  672. endoreg_db/utils/setup_config.py +177 -0
  673. endoreg_db/utils/translation.py +27 -0
  674. endoreg_db/utils/uuid.py +4 -0
  675. endoreg_db/utils/validate_endo_roi.py +19 -0
  676. endoreg_db/utils/validate_subcategory_dict.py +91 -0
  677. endoreg_db/utils/validate_video_detailed.py +357 -0
  678. endoreg_db/utils/video/__init__.py +26 -0
  679. endoreg_db/utils/video/extract_frames.py +88 -0
  680. endoreg_db/utils/video/ffmpeg_wrapper.py +835 -0
  681. endoreg_db/utils/video/names.py +42 -0
  682. endoreg_db/utils/video/streaming_processor.py +312 -0
  683. endoreg_db/utils/video/video_splitter.py +94 -0
  684. endoreg_db/views/Frames_NICE_and_PARIS_classifications_views.py +238 -0
  685. endoreg_db/views/__init__.py +274 -0
  686. endoreg_db/views/anonymization/__init__.py +27 -0
  687. endoreg_db/views/anonymization/media_management.py +454 -0
  688. endoreg_db/views/anonymization/overview.py +216 -0
  689. endoreg_db/views/anonymization/validate.py +107 -0
  690. endoreg_db/views/auth/__init__.py +13 -0
  691. endoreg_db/views/auth/keycloak.py +113 -0
  692. endoreg_db/views/examination/__init__.py +33 -0
  693. endoreg_db/views/examination/examination.py +37 -0
  694. endoreg_db/views/examination/examination_manifest_cache.py +26 -0
  695. endoreg_db/views/examination/get_finding_classification_choices.py +59 -0
  696. endoreg_db/views/examination/get_finding_classifications.py +36 -0
  697. endoreg_db/views/examination/get_findings.py +41 -0
  698. endoreg_db/views/examination/get_instruments.py +18 -0
  699. endoreg_db/views/examination/get_interventions.py +14 -0
  700. endoreg_db/views/finding/__init__.py +9 -0
  701. endoreg_db/views/finding/finding.py +112 -0
  702. endoreg_db/views/finding/get_classifications.py +14 -0
  703. endoreg_db/views/finding/get_interventions.py +17 -0
  704. endoreg_db/views/finding_classification/__init__.py +13 -0
  705. endoreg_db/views/finding_classification/base.py +0 -0
  706. endoreg_db/views/finding_classification/finding_classification.py +42 -0
  707. endoreg_db/views/finding_classification/get_classification_choices.py +55 -0
  708. endoreg_db/views/label/__init__.py +5 -0
  709. endoreg_db/views/label/label.py +15 -0
  710. endoreg_db/views/label_video_segment/__init__.py +16 -0
  711. endoreg_db/views/label_video_segment/create_lvs_from_annotation.py +44 -0
  712. endoreg_db/views/label_video_segment/get_lvs_by_name_and_video.py +50 -0
  713. endoreg_db/views/label_video_segment/label_video_segment.py +77 -0
  714. endoreg_db/views/label_video_segment/label_video_segment_by_label.py +174 -0
  715. endoreg_db/views/label_video_segment/label_video_segment_detail.py +73 -0
  716. endoreg_db/views/label_video_segment/update_lvs_from_annotation.py +46 -0
  717. endoreg_db/views/label_video_segment/validate.py +226 -0
  718. endoreg_db/views/media/__init__.py +45 -0
  719. endoreg_db/views/media/pdf_media.py +388 -0
  720. endoreg_db/views/media/segments.py +71 -0
  721. endoreg_db/views/media/sensitive_metadata.py +314 -0
  722. endoreg_db/views/media/video_media.py +272 -0
  723. endoreg_db/views/media/video_segments.py +524 -0
  724. endoreg_db/views/meta/__init__.py +15 -0
  725. endoreg_db/views/meta/available_files_list.py +146 -0
  726. endoreg_db/views/meta/report_meta.py +53 -0
  727. endoreg_db/views/meta/sensitive_meta_detail.py +148 -0
  728. endoreg_db/views/meta/sensitive_meta_list.py +104 -0
  729. endoreg_db/views/meta/sensitive_meta_verification.py +71 -0
  730. endoreg_db/views/misc/__init__.py +63 -0
  731. endoreg_db/views/misc/center.py +13 -0
  732. endoreg_db/views/misc/csrf.py +7 -0
  733. endoreg_db/views/misc/gender.py +14 -0
  734. endoreg_db/views/misc/secure_file_serving_view.py +80 -0
  735. endoreg_db/views/misc/secure_file_url_view.py +84 -0
  736. endoreg_db/views/misc/secure_url_validate.py +79 -0
  737. endoreg_db/views/misc/stats.py +220 -0
  738. endoreg_db/views/misc/translation.py +182 -0
  739. endoreg_db/views/misc/upload_views.py +240 -0
  740. endoreg_db/views/patient/__init__.py +5 -0
  741. endoreg_db/views/patient/patient.py +210 -0
  742. endoreg_db/views/patient_examination/DEPRECATED_video_backup.py +164 -0
  743. endoreg_db/views/patient_examination/__init__.py +11 -0
  744. endoreg_db/views/patient_examination/patient_examination.py +140 -0
  745. endoreg_db/views/patient_examination/patient_examination_create.py +63 -0
  746. endoreg_db/views/patient_examination/patient_examination_detail.py +66 -0
  747. endoreg_db/views/patient_examination/patient_examination_list.py +68 -0
  748. endoreg_db/views/patient_examination/video.py +194 -0
  749. endoreg_db/views/patient_finding/__init__.py +7 -0
  750. endoreg_db/views/patient_finding/base.py +0 -0
  751. endoreg_db/views/patient_finding/patient_finding.py +64 -0
  752. endoreg_db/views/patient_finding/patient_finding_optimized.py +259 -0
  753. endoreg_db/views/patient_finding_classification/__init__.py +5 -0
  754. endoreg_db/views/patient_finding_classification/pfc_create.py +67 -0
  755. endoreg_db/views/patient_finding_location/__init__.py +5 -0
  756. endoreg_db/views/patient_finding_location/pfl_create.py +70 -0
  757. endoreg_db/views/patient_finding_morphology/__init__.py +5 -0
  758. endoreg_db/views/patient_finding_morphology/pfm_create.py +70 -0
  759. endoreg_db/views/pdf/__init__.py +8 -0
  760. endoreg_db/views/pdf/pdf_stream.py +186 -0
  761. endoreg_db/views/pdf/reimport.py +177 -0
  762. endoreg_db/views/report/__init__.py +9 -0
  763. endoreg_db/views/report/report_list.py +112 -0
  764. endoreg_db/views/report/report_with_secure_url.py +28 -0
  765. endoreg_db/views/report/start_examination.py +7 -0
  766. endoreg_db/views/requirement/__init__.py +10 -0
  767. endoreg_db/views/requirement/evaluate.py +279 -0
  768. endoreg_db/views/requirement/lookup.py +367 -0
  769. endoreg_db/views/requirement/lookup_store.py +252 -0
  770. endoreg_db/views/requirement_lookup/lookup.py +0 -0
  771. endoreg_db/views/requirement_lookup/lookup_store.py +0 -0
  772. endoreg_db/views/stats/__init__.py +13 -0
  773. endoreg_db/views/stats/stats_views.py +229 -0
  774. endoreg_db/views/video/__init__.py +59 -0
  775. endoreg_db/views/video/correction.py +530 -0
  776. endoreg_db/views/video/reimport.py +195 -0
  777. endoreg_db/views/video/segmentation.py +274 -0
  778. endoreg_db/views/video/task_status.py +49 -0
  779. endoreg_db/views/video/timeline.py +46 -0
  780. endoreg_db/views/video/video_analyze.py +52 -0
  781. endoreg_db/views/video/video_apply_mask.py +48 -0
  782. endoreg_db/views/video/video_correction.py +21 -0
  783. endoreg_db/views/video/video_download_processed.py +58 -0
  784. endoreg_db/views/video/video_examination_viewset.py +242 -0
  785. endoreg_db/views/video/video_meta.py +29 -0
  786. endoreg_db/views/video/video_processing_history.py +24 -0
  787. endoreg_db/views/video/video_remove_frames.py +48 -0
  788. endoreg_db/views/video/video_stream.py +306 -0
  789. endoreg_db/views.py +0 -0
  790. endoreg_db-0.8.6.4.dist-info/METADATA +383 -0
  791. endoreg_db-0.8.6.4.dist-info/RECORD +793 -0
  792. endoreg_db-0.8.6.4.dist-info/WHEEL +4 -0
  793. endoreg_db-0.8.6.4.dist-info/licenses/LICENSE +674 -0
@@ -0,0 +1,306 @@
1
+ """
2
+ Video Streaming Views (Phase 3.2)
3
+
4
+ Separate view for streaming raw and processed video files.
5
+ Extracted from segmentation.py for better code organization.
6
+
7
+ Created: October 9, 2025
8
+ Updated: October 15, 2025 - Added HTTP 206 Range Request Support
9
+ """
10
+
11
+ import os
12
+ import re
13
+ import logging
14
+ import mimetypes
15
+ from pathlib import Path
16
+ from typing import Tuple, Optional
17
+ from django.http import FileResponse, Http404, StreamingHttpResponse
18
+ from rest_framework.views import APIView
19
+
20
+ from ...models import VideoFile
21
+ from ...utils.permissions import EnvironmentAwarePermission
22
+ from ...utils.paths import STORAGE_DIR # Import STORAGE_DIR for path resolution
23
+
24
+ logger = logging.getLogger(__name__)
25
+
26
+
27
+ def parse_range_header(range_header: str, file_size: int) -> Tuple[int, int]:
28
+ """
29
+ Parse HTTP Range header and return (start, end) byte positions.
30
+
31
+ Args:
32
+ range_header: HTTP Range header value (e.g., "bytes=0-1023")
33
+ file_size: Total file size in bytes
34
+
35
+ Returns:
36
+ Tuple of (start_byte, end_byte) inclusive
37
+
38
+ Raises:
39
+ ValueError: If range header is invalid
40
+ """
41
+ # Expected format: "bytes=start-end" or "bytes=start-"
42
+ match = re.match(r'bytes=(\d+)-(\d*)', range_header)
43
+
44
+ if not match:
45
+ raise ValueError(f"Invalid Range header format: {range_header}")
46
+
47
+ start = int(match.group(1))
48
+ end_str = match.group(2)
49
+
50
+ # If end is not specified, use file size - 1
51
+ end = int(end_str) if end_str else file_size - 1
52
+
53
+ # Validate range
54
+ if start >= file_size or start < 0:
55
+ raise ValueError(f"Start byte {start} is out of range (file size: {file_size})")
56
+
57
+ if end >= file_size:
58
+ end = file_size - 1
59
+
60
+ if start > end:
61
+ raise ValueError(f"Invalid range: start ({start}) > end ({end})")
62
+
63
+ return start, end
64
+
65
+
66
+ def stream_file_chunk(file_path: Path, start: int, end: int, chunk_size: int = 8192):
67
+ """
68
+ Generator that yields chunks of a file within the specified byte range.
69
+
70
+ Args:
71
+ file_path: Path to the file
72
+ start: Start byte position (inclusive)
73
+ end: End byte position (inclusive)
74
+ chunk_size: Size of each chunk to yield
75
+
76
+ Yields:
77
+ Bytes chunks from the file
78
+ """
79
+ with open(file_path, 'rb') as f:
80
+ f.seek(start)
81
+ remaining = end - start + 1 # +1 because end is inclusive
82
+
83
+ while remaining > 0:
84
+ chunk = f.read(min(chunk_size, remaining))
85
+ if not chunk:
86
+ break
87
+ yield chunk
88
+ remaining -= len(chunk)
89
+
90
+
91
+ def _stream_video_file(vf: VideoFile, frontend_origin: str, file_type: str = 'raw', range_header: Optional[str] = None) -> FileResponse | StreamingHttpResponse:
92
+ """
93
+ Helper function to stream a video file with proper headers, CORS support, and HTTP Range Requests.
94
+
95
+ Args:
96
+ vf: VideoFile model instance
97
+ frontend_origin: Frontend origin URL for CORS headers
98
+ file_type: Either 'raw' (original video) or 'processed' (anonymized video)
99
+ range_header: HTTP Range header value (e.g., "bytes=0-1023") for partial content requests
100
+
101
+ Returns:
102
+ FileResponse: HTTP 200 response streaming the entire file (no range header)
103
+ StreamingHttpResponse: HTTP 206 response streaming partial content (with range header)
104
+
105
+ Raises:
106
+ Http404: If video file not found or cannot be accessed
107
+
108
+ Note:
109
+ Permissions are handled by the calling view, not in this helper function.
110
+ HTTP 206 Partial Content support is critical for video seeking in browsers.
111
+ """
112
+ try:
113
+ # Determine which file to stream based on file_type
114
+ if file_type == 'raw':
115
+ if hasattr(vf, 'active_raw_file') and vf.active_raw_file and hasattr(vf.active_raw_file, 'name'):
116
+ file_ref = vf.active_raw_file
117
+ else:
118
+ raise Http404("No raw video file available for this entry")
119
+
120
+ elif file_type == 'processed':
121
+ if hasattr(vf, 'processed_file') and vf.processed_file and hasattr(vf.processed_file, 'name'):
122
+ file_ref = vf.processed_file
123
+ else:
124
+ raise Http404("No processed video file available for this entry")
125
+ else:
126
+ raise ValueError(f"Invalid file_type: {file_type}. Must be 'raw' or 'processed'.")
127
+
128
+ # FIX: Handle both relative and absolute paths
129
+ # Django FileField.path returns .name if MEDIA_ROOT is not set
130
+ # Import services store relative paths like "videos/UUID.mp4"
131
+ # We need to resolve to absolute path: STORAGE_DIR / "videos/UUID.mp4"
132
+ file_name = file_ref.name
133
+
134
+ if file_name.startswith('/'):
135
+ # Already absolute path
136
+ path = Path(file_name)
137
+ else:
138
+ # Relative path - make absolute by prepending STORAGE_DIR
139
+ path = STORAGE_DIR / file_name
140
+ logger.debug("Resolved relative path '%s' to absolute: %s", file_name, path)
141
+
142
+ # Validate file exists on disk
143
+ if not path.exists():
144
+ raise Http404(f"Video file not found on disk: {path}")
145
+
146
+ # Validate file size before streaming
147
+ try:
148
+ file_size = path.stat().st_size
149
+ if file_size == 0:
150
+ raise Http404("Video file is empty")
151
+ except OSError as e:
152
+ raise Http404(f"Cannot access video file: {str(e)}")
153
+
154
+ # Determine MIME type
155
+ mime, _ = mimetypes.guess_type(str(path))
156
+ content_type = mime or 'video/mp4' # Default to mp4 if detection fails
157
+
158
+ # βœ… NEW: HTTP Range Request support for video seeking
159
+ if range_header:
160
+ try:
161
+ # Parse Range header
162
+ start, end = parse_range_header(range_header, file_size)
163
+ logger.debug("Range request: bytes=%d-%d (total: %d)", start, end, file_size)
164
+
165
+ # Stream partial content (HTTP 206)
166
+ response = StreamingHttpResponse(
167
+ stream_file_chunk(path, start, end),
168
+ status=206, # Partial Content
169
+ content_type=content_type
170
+ )
171
+
172
+ # Set Range-specific headers
173
+ response['Content-Range'] = f'bytes {start}-{end}/{file_size}'
174
+ response['Content-Length'] = str(end - start + 1)
175
+ response['Accept-Ranges'] = 'bytes'
176
+ response['Content-Disposition'] = f'inline; filename="{path.name}"'
177
+
178
+ except ValueError as e:
179
+ # Invalid range header - return 416 Range Not Satisfiable
180
+ logger.warning("Invalid Range header: %s", str(e))
181
+ response = StreamingHttpResponse(
182
+ status=416, # Range Not Satisfiable
183
+ content_type=content_type
184
+ )
185
+ response['Content-Range'] = f'bytes */{file_size}'
186
+
187
+ else:
188
+ # No Range header - stream entire file (HTTP 200)
189
+ try:
190
+ # Open file in binary mode - FileResponse will handle closing
191
+ file_handle = open(path, 'rb')
192
+ response = FileResponse(file_handle, content_type=content_type)
193
+
194
+ # Set HTTP headers for video streaming
195
+ response['Content-Length'] = str(file_size)
196
+ response['Accept-Ranges'] = 'bytes' # Enable HTTP range requests for seeking
197
+ response['Content-Disposition'] = f'inline; filename="{path.name}"'
198
+
199
+ except IOError as e:
200
+ raise Http404(f"Cannot open video file: {str(e)}")
201
+
202
+ # CORS headers for frontend access (both HTTP 200 and 206)
203
+ response["Access-Control-Allow-Origin"] = frontend_origin
204
+ response["Access-Control-Allow-Credentials"] = "true"
205
+
206
+ return response
207
+
208
+ except Exception as e:
209
+ # Log unexpected errors but don't expose internal details
210
+ logger.error("Unexpected error in _stream_video_file: %s", str(e))
211
+ raise Http404("Video file cannot be streamed")
212
+
213
+
214
+ class VideoStreamView(APIView):
215
+ """
216
+ Separate view for video streaming to avoid DRF content negotiation issues.
217
+
218
+ Supports streaming both raw (original) and processed (anonymized) videos.
219
+
220
+ Query Parameters:
221
+ type: 'raw' (default) or 'processed' - Selects which video file to stream
222
+ file_type: (deprecated, use 'type') - Legacy parameter for backwards compatibility
223
+
224
+ Examples:
225
+ GET /api/media/videos/1/?type=raw - Stream original raw video
226
+ GET /api/media/videos/1/?type=processed - Stream anonymized/masked video
227
+ GET /api/videostream/1/ - Default to raw video (legacy endpoint)
228
+
229
+ Phase 3.2 Implementation:
230
+ - Supports dual video comparison (raw vs processed)
231
+ - Backward compatible with legacy ?file_type= parameter
232
+ - Proper error handling with Http404
233
+ - CORS support for frontend access
234
+ - HTTP range support for video seeking
235
+ """
236
+ permission_classes = [EnvironmentAwarePermission]
237
+
238
+ def get(self, request, pk=None):
239
+ """
240
+ Stream raw or anonymized video file with HTTP Range Request and CORS support.
241
+
242
+ Supports HTTP 206 Partial Content for video seeking functionality.
243
+
244
+ Args:
245
+ request: HTTP request object
246
+ pk: Video ID (primary key)
247
+
248
+ Returns:
249
+ FileResponse: HTTP 200 streaming entire video file (no range header)
250
+ StreamingHttpResponse: HTTP 206 streaming partial content (with range header)
251
+
252
+ Raises:
253
+ Http404: If video not found or file cannot be accessed
254
+ """
255
+ if pk is None:
256
+ raise Http404("Video ID is required")
257
+
258
+ # Initialize variables in outer scope
259
+ video_id_int = None
260
+
261
+ try:
262
+ # Validate video_id is numeric
263
+ try:
264
+ video_id_int = int(pk)
265
+ except (ValueError, TypeError):
266
+ raise Http404("Invalid video ID format")
267
+
268
+ # Support both 'type' (frontend standard) and 'file_type' (legacy)
269
+ # Priority: type > file_type > default 'raw'
270
+ file_type = 'raw' # Default value
271
+ try:
272
+ file_type_param = request.query_params.get('type') or request.query_params.get('file_type')
273
+ if file_type_param:
274
+ file_type = file_type_param.lower()
275
+
276
+ if file_type not in ['raw', 'processed']:
277
+ logger.warning("Invalid file_type '%s', defaulting to 'raw'", file_type)
278
+ file_type = 'raw'
279
+
280
+ except Exception as e:
281
+ logger.warning("Error parsing file_type parameter: %s, defaulting to 'raw'", e)
282
+ file_type = 'raw'
283
+
284
+ # Fetch video from database
285
+ vf = VideoFile.objects.get(pk=video_id_int)
286
+
287
+ # Get frontend origin for CORS
288
+ frontend_origin = os.environ.get('FRONTEND_ORIGIN', 'http://localhost:8000')
289
+
290
+ # βœ… NEW: Extract Range header for HTTP 206 support
291
+ range_header = request.META.get('HTTP_RANGE')
292
+
293
+ # Stream the video file with optional range support
294
+ return _stream_video_file(vf, frontend_origin, file_type, range_header)
295
+
296
+ except VideoFile.DoesNotExist:
297
+ raise Http404(f"Video with ID {pk} not found")
298
+
299
+ except Http404:
300
+ # Re-raise Http404 exceptions as they should bubble up
301
+ raise
302
+
303
+ except Exception as e:
304
+ # Log unexpected errors and convert to Http404
305
+ logger.error("Unexpected error in VideoStreamView for video_id=%s: %s", pk, str(e))
306
+ raise Http404("Video streaming failed")
endoreg_db/views.py ADDED
File without changes
@@ -0,0 +1,383 @@
1
+ Metadata-Version: 2.4
2
+ Name: endoreg-db
3
+ Version: 0.8.6.4
4
+ Summary: EndoReg Db Django App
5
+ Project-URL: Homepage, https://info.coloreg.de
6
+ Project-URL: Repository, https://github.com/wg-lux/endoreg-db
7
+ Author: Max Hild, Hamza Zahid, Peter Kowalczyk
8
+ Author-email: "Thomas J. Lux" <lux_t1@ukw.de>
9
+ Maintainer: Max Hild, Hamza Zahid, Peter Kowalczyk
10
+ Maintainer-email: "Thomas J. Lux" <lux_t1@ukw.de>
11
+ License-File: LICENSE
12
+ Classifier: Development Status :: 3 - Alpha
13
+ Classifier: Programming Language :: Python :: 3.12
14
+ Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
15
+ Requires-Python: <3.13,>=3.12
16
+ Requires-Dist: agl-report-reader>=0.4.0
17
+ Requires-Dist: build>=1.3.0
18
+ Requires-Dist: celery>=5.5.3
19
+ Requires-Dist: cryptography>=44.0.2
20
+ Requires-Dist: decord==0.6.0
21
+ Requires-Dist: django-bootstrap5>=25.2
22
+ Requires-Dist: django-cors-headers>=4.8.0
23
+ Requires-Dist: django-extensions>=3.2.3
24
+ Requires-Dist: django-filter>=25.1
25
+ Requires-Dist: django-modeltranslation>=0.19.16
26
+ Requires-Dist: django>=5.2.4
27
+ Requires-Dist: djangorestframework>=3.16.1
28
+ Requires-Dist: dotenv>=0.9.9
29
+ Requires-Dist: faker>=37.6.0
30
+ Requires-Dist: flake8>=7.3.0
31
+ Requires-Dist: gunicorn>=23.0.0
32
+ Requires-Dist: icecream>=2.1.4
33
+ Requires-Dist: librosa==0.11.0
34
+ Requires-Dist: llvmlite>=0.44.0
35
+ Requires-Dist: lx-anonymizer[llm,ocr]>=0.8.8
36
+ Requires-Dist: moviepy==2.2.1
37
+ Requires-Dist: mypy>=1.16.0
38
+ Requires-Dist: numpy>=2.2.3
39
+ Requires-Dist: opencv-python>=4.11.0.86
40
+ Requires-Dist: openpyxl>=3.1.5
41
+ Requires-Dist: optimum==1.26.1
42
+ Requires-Dist: pandas>=2.2.3
43
+ Requires-Dist: pillow>=11.3.0
44
+ Requires-Dist: pip>=25.2
45
+ Requires-Dist: pydantic>=2.10.6
46
+ Requires-Dist: pyjwt>=2.10.1
47
+ Requires-Dist: pytesseract>=0.3.13
48
+ Requires-Dist: pytest-cov>=6.1.1
49
+ Requires-Dist: pytest-django>=4.11.1
50
+ Requires-Dist: pytest-env>=1.1.5
51
+ Requires-Dist: pytest>=8.3.5
52
+ Requires-Dist: python-ffmpeg>=2.0.12
53
+ Requires-Dist: python-magic>=0.4.27
54
+ Requires-Dist: pytorch-lightning>=2.5.1
55
+ Requires-Dist: pyyaml>=6.0.2
56
+ Requires-Dist: regex>=2024.11.6
57
+ Requires-Dist: requests>=2.32.3
58
+ Requires-Dist: ruff>=0.14.0
59
+ Requires-Dist: scikit-learn>=1.5.2
60
+ Requires-Dist: scipy>=1.16.1
61
+ Requires-Dist: soundfile==0.12.1
62
+ Requires-Dist: sphinx-rtd-theme>=3.0.2
63
+ Requires-Dist: sphinx>=8.2.3
64
+ Requires-Dist: tesseract>=0.1.3
65
+ Requires-Dist: torch==2.5.1
66
+ Requires-Dist: torchaudio==2.5.1
67
+ Requires-Dist: torchvision==0.20.1
68
+ Requires-Dist: tqdm>=4.67.0
69
+ Requires-Dist: transformers==4.44.2
70
+ Requires-Dist: twine>=6.2.0
71
+ Requires-Dist: urllib3>=2.2.3
72
+ Requires-Dist: vector-quantize-pytorch==1.18.5
73
+ Requires-Dist: vocos==0.1.0
74
+ Requires-Dist: whitenoise>=6.10.0
75
+ Provides-Extra: dev
76
+ Description-Content-Type: text/markdown
77
+
78
+ # EndoregDB - Professional Data Infrastructure for Clinical Research
79
+
80
+ EndoregDB is a comprehensive database framework designed to manage medical and research-related data for clinical trials. This repository focuses on efficient data processing, automated deployment, security, and reproducibility, offering a flexible setup for local development environments as well as distributed systems. It supports the integration of AI/ML tools and advanced image and report processing.
81
+
82
+ This infrastructure was originally designed for clinical research studies and is optimized for handling large data volumes, including:
83
+
84
+ - Medical reports,
85
+ - Patient imaging and video data,
86
+ - Clinical product and treatment data,
87
+ and more.
88
+
89
+ ## πŸš€ Key Features
90
+
91
+ ### System Architecture
92
+
93
+ - **Modular Design**: Built on scalable and reusable components to simplify integration into various environments.
94
+ - **Multi-System Support**: Manages configurations for local workstations and production servers.
95
+ - **Role-Specific Configuration**: Predefined roles for common use cases:
96
+ - Medical data processing systems
97
+ - AI/ML model deployment
98
+ - Research workstation configuration
99
+
100
+ ### Security & Data Management
101
+
102
+ - **Data Encryption**: All sensitive data is encrypted, and privacy policies are enforced.
103
+ - **Impermanence**: Stateless system configuration with persistence for critical data.
104
+ - **Access Control**: Role-based access and identity management integration.
105
+
106
+ ### Data and Processing Environment
107
+
108
+ - **Data Processing**: Optimized for processing medical datasets with preprocessing tools.
109
+ - **AI/ML Support**:
110
+ - Integration of machine learning tools for predictive analysis.
111
+ - TensorFlow, PyTorch, and other frameworks supported for model training.
112
+ - **Image/Video Processing**: Support for analyzing patient images and clinical videos.
113
+
114
+ ### Development Tools & Infrastructure
115
+
116
+ - **Data Science Toolchains**: Pre-configured environments for data processing, analysis, and visualization.
117
+ - **Monitoring & Logging**: Setup for continuous monitoring and logging to ensure system stability and performance.
118
+
119
+ ---
120
+
121
+ ## πŸ›  Getting Started
122
+
123
+ ### Prerequisites
124
+
125
+ - A Linux-based system (Ubuntu/Debian recommended) or NixOS
126
+ - Hardware with sufficient storage for data processing (at least 1 TB recommended)
127
+
128
+ ### Quick Start
129
+
130
+ 1. Clone the repository:
131
+
132
+ ```bash
133
+ git clone https://github.com/wg-lux/endoreg-db.git
134
+ cd endoreg-db
135
+ ```
136
+
137
+ 2. Set up your Python environment
138
+ We need to have a `devenv.nix` file.
139
+ This Nix `devenv.nix` configuration sets up a Python development environment for a Django-based project using `uv` for dependency management. It defines project directories, environment variables, runtime packages, and several development tasks and scripts.
140
+
141
+ **Some available Test Shortcuts**
142
+
143
+ - `runtests`: Runs all tests β€” `uv run python runtests.py`
144
+ - `runtests-dataloader`: Runs dataloader tests β€” `uv run python runtests.py 'dataloader'`
145
+ - `runtests-other`: Runs other miscellaneous tests β€” `uv run python runtests.py 'other'`
146
+ - `runtests-helpers`: Runs helper module tests β€” `uv run python runtests.py 'helpers'`
147
+ - `runtests-administration`: Runs admin module tests β€” `uv run python runtests.py 'administration'`
148
+ - `runtests-medical`: Runs medical module tests β€” `uv run python runtests.py 'medical'`
149
+
150
+ 3. Then run
151
+
152
+ ```bash
153
+ direnv allow
154
+ ```
155
+
156
+ 4. Run tests:
157
+ Call Devenv Script to run tests
158
+
159
+ ```bash
160
+ runtests
161
+ ```
162
+ Tests Overview
163
+ - These tests ensure the functionality of different models and scenarios.
164
+ - After running them, you can view the results as demonstrated in the image below:
165
+
166
+ ![Test Results](Images/testscreenshort.png)
167
+
168
+ 5. Run
169
+ ```python
170
+ python manage.py migrate
171
+
172
+ ```
173
+ - It applies database migrations and make tables.
174
+ - It updates your database schema to match the current state of your Django models.
175
+
176
+ 6. To load the database data run
177
+ ```
178
+ python manage.py load_base_db_data
179
+
180
+ ```
181
+ ![Data](Images/loadbasedata0.png)
182
+ ![Data](Images/loadbasedata1.png)
183
+ ![Data](Images/loadbasedata2.png)
184
+ ![Data](Images/loadbasedata3.png)
185
+ ![Data](Images/loadbasedata4.png)
186
+ ![Data](Images/loadbasedata4b.png)
187
+ ![Data](Images/loadbasedata5.png)
188
+
189
+ 7. Accessing the Django Shell
190
+ - To fetch or interact with data in the terminal, run the following command to run the Django shell:
191
+
192
+ ```bash
193
+ python manage.py shell
194
+ ```
195
+ - Using the Django shell, you can:
196
+ - Import database models
197
+ - Fetch data from the database
198
+ - Access related data through model relationships (e.g., foreign keys, one-to-many, many-to-many)
199
+ - Example is shown below
200
+
201
+ #### EXAMPLE # 1
202
+ ![Shell](Images/shell2.png)
203
+ - Explanation:
204
+ This script fetches a patient by ID and prints their related examination(s) using Django ORM. It retrieves the examination name linked to the patient from the PatientExamination table.
205
+
206
+ #### EXAMPLE # 2
207
+ ![Shell](Images/shell0.png)
208
+ - Explanation:
209
+ In the Django shell, a specific ExaminationIndication named "colonoscopy_screening" was fetched, and its related FindingIntervention records were accessed using the reverse relation expected_interventions. The first intervention (colon_lesion_polypectomy_cold_snare) was then queried to confirm it is also linked to multiple indications, demonstrating a many-to-many relationship between indications and interventions.
210
+
211
+ #### EXAMPLE # 3
212
+ ![Shell](Images/shell1.png)
213
+ - Explanation:
214
+ All required labels (polyp, instrument, digital_chromo_endoscopy, etc.) are confirmed to exist. The first available video (VideoFile) was loaded, with a valid frame_dir. Using the label "polyp", 8 labeled polyp segments were found in that video, with specific start and end frame numbers.
215
+
216
+ #### EXAMPLE # 4
217
+ ##### Image a
218
+
219
+ ![Shell](Images/shell3.png)
220
+
221
+ ##### Image b - All classifications with their choices together
222
+
223
+ ![Shell](Images/shell3b.png)
224
+
225
+ - Explanation: Using the Django shell to fetch all morphology classifications (e.g., NICE, Paris) and their related choices from the database.
226
+
227
+
228
+
229
+
230
+ ---
231
+ ## πŸ“¦ Database Backup and Restore
232
+
233
+ This project includes two shell scripts to **export** and **import** database data in JSON format using Django's management commands.
234
+
235
+ ### Setup
236
+
237
+ First, make the scripts executable:
238
+
239
+ ```bash
240
+ chmod +x import_db.sh
241
+ chmod +x export_db.sh
242
+ ```
243
+ ### Export (Backup) the Database
244
+
245
+ To export the current database into a JSON file:
246
+
247
+ ```bash
248
+ ./export_db.sh
249
+ ```
250
+ This will create a backup file such as `endoreg_db_backup.json`.
251
+
252
+ #### List of the comands in 'export_db.sh'
253
+ 1. `python manage.py dumpdata --indent 4 --output=endoreg_db_backup.json` (if migrate comand generates and stores data in database table then wee nee dto exclude those tables from dumping)
254
+
255
+ 2. `python manage.py shell < fix_endoreg_db_backup_json.py`
256
+
257
+ ### Import (Restore) the Database
258
+
259
+ To load the data back into the database
260
+ ```bash
261
+ ./import_db.sh
262
+ ```
263
+ #### List of the comands in 'import_db.sh'
264
+ 1. `rm dev_db.sqlite3`
265
+ 2. `python manage.py migrate`
266
+ 3. `python manage.py shell < fix_endoreg_db_backup_json.py`
267
+ 4. `python manage.py loaddata endoreg_db_backup_fixed.json`
268
+
269
+ ---
270
+
271
+
272
+
273
+ ## πŸ“ Repository Structure
274
+
275
+ ```
276
+ endoreg-db/
277
+ β”œβ”€β”€ endoreg_db/ # Main Django app for medical data
278
+ β”‚ β”œβ”€β”€ case_generator/ # Sample case generator
279
+ β”‚ β”œβ”€β”€ data/ # Medical knowledge base
280
+ β”‚ β”œβ”€β”€ management/ # Data wrangling operations
281
+ β”‚ β”œβ”€β”€ models/ # Data models
282
+ β”‚ β”œβ”€β”€ migrations/ # Database migrations
283
+ β”‚ └── serializers/ # Serializers for data
284
+ β”œβ”€β”€ .gitignore # Git ignore file for unnecessary files
285
+ └── README.md # Project description and setup instructions
286
+ ```
287
+
288
+ ---
289
+
290
+ ## πŸ”’ Security Features
291
+
292
+ - **Data Encryption**: All sensitive patient data is encrypted.
293
+ - **Role-Based Access Control**: Configurable roles for managing access to various parts of the system.
294
+ - **Logging & Auditing**: Comprehensive logging system that tracks user activities and data changes.
295
+
296
+ ---
297
+
298
+ ## πŸ–₯️ Supported Systems
299
+
300
+ - **Workstations**: Local development or research workstations with low data processing demands.
301
+ - **Servers**: Scalable server infrastructure for processing large data volumes, integrated with cloud services for scalability.
302
+
303
+ ---
304
+
305
+ ## πŸ›Ÿ Support
306
+
307
+ For issues and questions:
308
+
309
+ - Create an issue in the repository
310
+ - Review the Deployment Guide for common issues
311
+
312
+ ---
313
+
314
+ ## πŸ“œ License
315
+
316
+ MIT - see LICENSE
317
+
318
+ ---
319
+
320
+
321
+ ## πŸ“– Further Documentation
322
+
323
+ All extended documentation lives in the project **Wiki** β†’ **[Browse the Wiki Β»](https://github.com/wg-lux/endoreg-db/wiki)**
324
+
325
+ ### Optimization Documentation
326
+ - [Complete Optimization Project Report](https://github.com/wg-lux/endoreg-db/wiki/Complete-Optimiztion-Project-Report)
327
+ - [Test Performance Optimization Guide](https://github.com/wg-lux/endoreg-db/wiki/Test-Performance-Optimization-Guide)
328
+ - [Test Performance Optimization - Success Summary](https://github.com/wg-lux/endoreg-db/wiki/Test-Performance-Optimization-‐-Succes-Summary)
329
+ - [Test Performance Optimization: Complete Implementation Summary](https://github.com/wg-lux/endoreg-db/wiki/Test-Performance-Optimization:-Complete-Implementation-Summary)
330
+ - [Test Suite Optimization - Final Status Report](https://github.com/wg-lux/endoreg-db/wiki/Test-Suite-Optimization-‐-Final-Status-Report)
331
+ - [Test Suite Analysis & Optimization Plan](https://github.com/wg-lux/endoreg-db/wiki/Test-Suite-Analysis-&-Optimization-Plan)
332
+
333
+ ---
334
+
335
+ ### Models and Migration Documentation
336
+ - [Models Documentation](https://github.com/wg-lux/endoreg-db/wiki/Models-Documentation)
337
+ - [Test Migration & Optimization Report](https://github.com/wg-lux/endoreg-db/wiki/Test-Migration-&-Optimization-Report)
338
+ - [Test Migration Success Summary](https://github.com/wg-lux/endoreg-db/wiki/Test-Migration-Success-Summary)
339
+ - [Test Optimization Migration Guide](https://github.com/wg-lux/endoreg-db/wiki/Test-Optimization-Migration-Guide)
340
+
341
+ ---
342
+
343
+ ### API Documentation
344
+ - [Upload API Documentation](https://github.com/wg-lux/endoreg-db/wiki/Upload-API-Documentation)
345
+
346
+ ---
347
+
348
+ ### Frame-Anonymisierung
349
+ - [Frame-Anonymisierung](https://github.com/wg-lux/endoreg-db/wiki/Frame-Anonymisierung)
350
+
351
+ ---
352
+
353
+ ### Tutorials Documentation
354
+ - [Run Production Server](https://github.com/wg-lux/endoreg-db/wiki/Run-Production-Server)
355
+ - [Date and Time Standardization for Models](https://github.com/wg-lux/endoreg-db/wiki/Date-and-Time-Standardization-for-Models)
356
+
357
+ ---
358
+
359
+ ### Keycloak
360
+ - [How to Create a New Account for Keycloak + Nextcloud](https://github.com/wg-lux/endoreg-db/wiki/How-to-Create-a-New-Account-for-Keycloak-+-Nextcloud)
361
+ - [Integration with the frontend](https://github.com/wg-lux/endoreg-db/wiki/Integration-with-the-frontend)
362
+ - [Merging Multi-User Accounts in Nextcloud // current options](https://github.com/wg-lux/endoreg-db/wiki/Merging-Multi-User-Accounts-in-Nextcloud-//-current-options)
363
+ - [New user login steps for keycloak and nextcloud](https://github.com/wg-lux/endoreg-db/wiki/New-user-login-steps-for-keycloak-and-nextcloud)
364
+ - [keycloak integration with backend endpoint](https://github.com/wg-lux/endoreg-db/wiki/keycloak-integration-with-backend-endpoint)
365
+
366
+ ---
367
+
368
+ ### Coding Principles & Practices
369
+ - [Timestamp Naming Standard](https://github.com/wg-lux/endoreg-db/wiki/Timestamp-Naming-Standard)
370
+
371
+ ---
372
+
373
+ ### Figures
374
+ - [Coloreg](https://github.com/wg-lux/endoreg-db/wiki/Coloreg)
375
+ - [EndoReg Framework](https://github.com/wg-lux/endoreg-db/wiki/EndoReg-Framework)
376
+ - [EndoReg Data Collection Workflow](https://github.com/wg-lux/endoreg-db/wiki/EndoReg-Data-Collection-Workflow)
377
+ - [Eine gemeinsame Datenplattform fΓΌr Klinik & Forschung](https://github.com/wg-lux/endoreg-db/wiki/Eine-gemeinsame-Datenplattform-fΓΌr-Klinik-&-Forschung)
378
+
379
+ ---
380
+
381
+ ### Miscellaneous
382
+ - [Requirement System Guide](https://github.com/wg-lux/endoreg-db/wiki/Requirement-System-Guide)
383
+ - [Official Site Link](https://github.com/wg-lux/endoreg-db/wiki/Official-Site-Link)