megadetector 5.0.16__tar.gz → 5.0.18__tar.gz

{megadetector-5.0.16/megadetector.egg-info → megadetector-5.0.18}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: megadetector
-Version: 5.0.16
+Version: 5.0.18
 Summary: MegaDetector is an AI model that helps conservation folks spend less time doing boring things with camera trap images.
 Author-email: Your friendly neighborhood MegaDetector team <cameratraps@lila.science>
 Maintainer-email: Your friendly neighborhood MegaDetector team <cameratraps@lila.science>
@@ -57,22 +57,20 @@ This package is a pip-installable version of the support/inference code for [Meg
 If you aren't looking for the Python package specifically, and you just want to learn more about what MegaDetector is all about, head over to the [MegaDetector repo](https://github.com/agentmorris/MegaDetector/?tab=readme-ov-file#megadetector).
 
 
-## Reasons you probably aren't looking for this package
+## Reasons you might not be looking for this package
 
 ### If you are an ecologist...
 
-If you are an ecologist looking to use MegaDetector to help you get through your camera trap images, you probably don't want this package.  We recommend starting with our "[Getting started with MegaDetector](https://github.com/agentmorris/MegaDetector/blob/main/getting-started.md)" page, then digging in to the [MegaDetector User Guide](https://github.com/agentmorris/MegaDetector/blob/main/megadetector.md), which will walk you through the process of using MegaDetector.  That journey will <i>not</i> involve this Python package.
+If you are an ecologist looking to use MegaDetector to help you get through your camera trap images, you probably don't want this package, or at least you probably don't want to start at this page.  We recommend starting with our "[Getting started with MegaDetector](https://github.com/agentmorris/MegaDetector/blob/main/getting-started.md)" page, then digging in to the [MegaDetector User Guide](https://github.com/agentmorris/MegaDetector/blob/main/megadetector.md), which will walk you through the process of using MegaDetector.
 
 ### If you are a computer-vision-y type...
 
-If you are a computer-vision-y person looking to run or fine-tune MegaDetector programmatically, you still probably don't want this package.  MegaDetector is just a fine-tuned version of [YOLOv5](https://github.com/ultralytics/yolov5), and the [ultralytics](https://github.com/ultralytics/ultralytics/) package (from the developers of YOLOv5) has a zillion bells and whistles for both inference and fine-tuning that this package doesn't.
+If you are a computer-vision-y person looking to run or fine-tune MegaDetector programmatically, you probably don't want this package.  MegaDetector is just a fine-tuned version of [YOLOv5](https://github.com/ultralytics/yolov5), and the [ultralytics](https://github.com/ultralytics/ultralytics/) package (from the developers of YOLOv5) has a zillion bells and whistles for both inference and fine-tuning that this package doesn't.
 
 ## Reasons you might want to use this package
 
 If you want to programmatically interact with the postprocessing tools from the MegaDetector repo, or programmatically run MegaDetector in a way that produces [Timelapse](https://saul.cpsc.ucalgary.ca/timelapse)-friendly output (i.e., output in the standard [MegaDetector output format](https://github.com/agentmorris/MegaDetector/tree/main/megadetector/api/batch_processing#megadetector-batch-output-format)), this package might be for you.
 
-Although even if that describes you, you <i>still</i> might be better off cloning the MegaDetector repo.  Pip-installability requires that some dependencies be newer than what was available at the time MDv5 was trained, so results are <i>very slightly</i> different than results produced in the "official" environment.  These differences <i>probably</i> don't matter much, but they have not been formally characterized.
-
 ## If I haven't talked you out of using this package...
 
 To install:

{megadetector-5.0.16 → megadetector-5.0.18}/README-package.md

@@ -5,22 +5,20 @@ This package is a pip-installable version of the support/inference code for [Meg
 If you aren't looking for the Python package specifically, and you just want to learn more about what MegaDetector is all about, head over to the [MegaDetector repo](https://github.com/agentmorris/MegaDetector/?tab=readme-ov-file#megadetector).
 
 
-## Reasons you probably aren't looking for this package
+## Reasons you might not be looking for this package
 
 ### If you are an ecologist...
 
-If you are an ecologist looking to use MegaDetector to help you get through your camera trap images, you probably don't want this package.  We recommend starting with our "[Getting started with MegaDetector](https://github.com/agentmorris/MegaDetector/blob/main/getting-started.md)" page, then digging in to the [MegaDetector User Guide](https://github.com/agentmorris/MegaDetector/blob/main/megadetector.md), which will walk you through the process of using MegaDetector.  That journey will <i>not</i> involve this Python package.
+If you are an ecologist looking to use MegaDetector to help you get through your camera trap images, you probably don't want this package, or at least you probably don't want to start at this page.  We recommend starting with our "[Getting started with MegaDetector](https://github.com/agentmorris/MegaDetector/blob/main/getting-started.md)" page, then digging in to the [MegaDetector User Guide](https://github.com/agentmorris/MegaDetector/blob/main/megadetector.md), which will walk you through the process of using MegaDetector.
 
 ### If you are a computer-vision-y type...
 
-If you are a computer-vision-y person looking to run or fine-tune MegaDetector programmatically, you still probably don't want this package.  MegaDetector is just a fine-tuned version of [YOLOv5](https://github.com/ultralytics/yolov5), and the [ultralytics](https://github.com/ultralytics/ultralytics/) package (from the developers of YOLOv5) has a zillion bells and whistles for both inference and fine-tuning that this package doesn't.
+If you are a computer-vision-y person looking to run or fine-tune MegaDetector programmatically, you probably don't want this package.  MegaDetector is just a fine-tuned version of [YOLOv5](https://github.com/ultralytics/yolov5), and the [ultralytics](https://github.com/ultralytics/ultralytics/) package (from the developers of YOLOv5) has a zillion bells and whistles for both inference and fine-tuning that this package doesn't.
 
 ## Reasons you might want to use this package
 
 If you want to programmatically interact with the postprocessing tools from the MegaDetector repo, or programmatically run MegaDetector in a way that produces [Timelapse](https://saul.cpsc.ucalgary.ca/timelapse)-friendly output (i.e., output in the standard [MegaDetector output format](https://github.com/agentmorris/MegaDetector/tree/main/megadetector/api/batch_processing#megadetector-batch-output-format)), this package might be for you.
 
-Although even if that describes you, you <i>still</i> might be better off cloning the MegaDetector repo.  Pip-installability requires that some dependencies be newer than what was available at the time MDv5 was trained, so results are <i>very slightly</i> different than results produced in the "official" environment.  These differences <i>probably</i> don't matter much, but they have not been formally characterized.
-
 ## If I haven't talked you out of using this package...
 
 To install:

{megadetector-5.0.16 → megadetector-5.0.18}/megadetector/data_management/importers/snapshot_safari_importer_reprise.py

@@ -24,7 +24,7 @@ from collections import defaultdict
 
 from megadetector.utils import path_utils
 
-input_base = '/media/user/Elements'
+input_base = 'e:/'
 output_base = os.path.expanduser('~/data/snapshot-safari-metadata')
 file_list_cache_file = os.path.join(output_base,'file_list.json')
 
@@ -76,23 +76,16 @@ print('Found a total of {} files, {} of which are images'.format(
     len(all_files_relative),len(all_image_files)))
 
 
-#%% Copy all csv files to the annotation cache folder
+#%% Copy all .csv files to the annotation cache folder
 
 # fn = csv_files[0]
-for fn in csv_files:
+for fn in tqdm(csv_files):
+    
     target_file = os.path.join(annotation_cache_dir,os.path.basename(fn))
     source_file = os.path.join(input_base,fn)
     shutil.copyfile(source_file,target_file)    
 
-def read_cached_csv_file(fn):
-    """
-    Later cells will ask to read a .csv file from the original hard drive;
-    read from the annotation cache instead.
-    """
-    
-    cached_csv_file = os.path.join(annotation_cache_dir,os.path.basename(fn))
-    df = pd.read_csv(cached_csv_file)
-    return df
+print('Copied {} .csv files to cache folder'.format(len(csv_files)))
 
 
 #%% List project folders
@@ -123,6 +116,21 @@ project_folder_to_project_code = {v: k for k, v in project_code_to_project_folde
 project_codes = sorted(list(project_code_to_project_folder.keys()))
 project_folders = sorted(list(project_code_to_project_folder.values()))
 
+print('Eumerated {} project folders'.format(len(project_folders)))
+
+
+#%% Support functions
+
+def read_cached_csv_file(fn):
+    """
+    Later cells will ask to read a .csv file from the original hard drive;
+    read from the annotation cache instead.
+    """
+    
+    cached_csv_file = os.path.join(annotation_cache_dir,os.path.basename(fn))
+    df = pd.read_csv(cached_csv_file)
+    return df
+
 def file_to_project_folder(fn):
     """
     For a given filename relative to the drive root, return the corresponding
@@ -138,7 +146,6 @@ def file_to_project_folder(fn):
     assert project_folder in project_folders
     return project_folder
 
-
 def file_to_project_code(fn):
     """
     For a given filename relative to the drive root, return the corresponding
@@ -147,6 +154,9 @@ def file_to_project_code(fn):
     
     return project_folder_to_project_code[file_to_project_folder(fn)]
 
+
+#%% Consistency checking
+
 assert file_to_project_folder(
     'APN/APN_S2/DW/DW_R5/APN_S2_DW_R5_IMAG0003.JPG') == 'APN'
 assert file_to_project_folder(
@@ -163,9 +173,11 @@ assert file_to_project_code(
 #
 # E.g.:
 #
-# 'DHP': ['Snapshot South Africa/DHP/LILA_Reports/DHP_S1_report_lila.csv',
-# 'Snapshot South Africa/DHP/LILA_Reports/DHP_S2_report_lila.csv',
-# 'Snapshot South Africa/DHP/LILA_Reports/DHP_S3_report_lila.csv']
+# 'DHP': [
+#        'Snapshot South Africa/DHP/LILA_Reports/DHP_S1_report_lila.csv',
+#        'Snapshot South Africa/DHP/LILA_Reports/DHP_S2_report_lila.csv',
+#        'Snapshot South Africa/DHP/LILA_Reports/DHP_S3_report_lila.csv'
+#       ]
 #
 project_code_to_report_files = defaultdict(list)
 

{megadetector-5.0.16 → megadetector-5.0.18}/megadetector/detection/process_video.py

@@ -345,7 +345,8 @@ def process_video(options):
     caller_provided_rendering_output_folder = (options.frame_rendering_folder is not None)
     
     frame_output_folder = None
-        
+    frame_filenames = None
+    
     # If we should re-use existing results, and the output file exists, don't bother running MD
     if (options.reuse_results_if_available and os.path.isfile(options.output_json_file)):
             
@@ -383,7 +384,7 @@ def process_video(options):
                                                verbose=options.verbose, 
                                                frames_to_process=options.frames_to_extract)
         
-        _add_frame_numbers_to_results(frame_results['results'])
+        frame_results['results'] = _add_frame_numbers_to_results(frame_results['results'])
         
         run_detector_batch.write_results_to_file(
             frame_results['results'], 
@@ -444,7 +445,7 @@ def process_video(options):
                 augment=options.augment,
                 image_size=options.image_size)
         
-            _add_frame_numbers_to_results(results)
+            results = _add_frame_numbers_to_results(results)
         
             run_detector_batch.write_results_to_file(
                 results, 
@@ -545,6 +546,7 @@ def process_video_folder(options):
     
     frame_output_folder = None
     image_file_names = None
+    video_filename_to_fs = {}
     
     # Run MD in memory if we don't need to generate frames
     #
@@ -575,6 +577,10 @@ def process_video_folder(options):
         
         video_results = md_results['results']
         
+        for i_video,video_filename in enumerate(md_results['video_filenames']):
+            assert video_filename not in video_filename_to_fs
+            video_filename_to_fs[video_filename] = md_results['frame_rates'][i_video]
+        
         all_frame_results = []
         
         # r = video_results[0]
@@ -586,13 +592,15 @@ def process_video_folder(options):
             all_frame_results, 
             frames_json,
             relative_path_base=None,
-            detector_file=options.model_file,
-            custom_metadata={'video_frame_rate':md_results['frame_rates']})
+            detector_file=options.model_file)
     
     else:
         
         ## Split every video into frames
         
+        if options.verbose:
+            print('Extracting frames for folder {}'.format(options.input_video_file))
+            
         if caller_provided_frame_output_folder:
             frame_output_folder = options.frame_folder
         else:
@@ -600,8 +608,6 @@ def process_video_folder(options):
             
         os.makedirs(frame_output_folder, exist_ok=True)
     
-        print('Extracting frames')
-        
         frame_filenames, Fs, video_filenames = \
             video_folder_to_frames(input_folder=options.input_video_file,
                                    output_folder_base=frame_output_folder, 
@@ -615,6 +621,10 @@ def process_video_folder(options):
                                    frames_to_extract=options.frames_to_extract,
                                    allow_empty_videos=options.allow_empty_videos)
         
+        for i_video,video_filename in enumerate(video_filenames):
+            assert video_filename not in video_filename_to_fs
+            video_filename_to_fs[video_filename] = Fs[i_video]
+                
         print('Extracted frames for {} videos'.format(len(set(video_filenames))))
         image_file_names = list(itertools.chain.from_iterable(frame_filenames))
         
@@ -660,15 +670,15 @@ def process_video_folder(options):
                 results, 
                 frames_json,
                 relative_path_base=frame_output_folder,
-                detector_file=options.model_file,
-                custom_metadata={'video_frame_rate':Fs})
+                detector_file=options.model_file)
         
     # ...if we're running MD on in-memory frames vs. extracting frames to disk
     
     ## Convert frame-level results to video-level results
 
     print('Converting frame-level results to video-level results')
-    frame_results_to_video_results(frames_json,video_json)
+    frame_results_to_video_results(frames_json,video_json,
+                                   video_filename_to_frame_rate=video_filename_to_fs)
 
 
     ## (Optionally) render output videos

{megadetector-5.0.16 → megadetector-5.0.18}/megadetector/detection/run_detector_batch.py

@@ -846,7 +846,7 @@ def write_results_to_file(results,
     https://github.com/agentmorris/MegaDetector/tree/main/megadetector/api/batch_processing#batch-processing-api-output-format
 
     Args:
-        results (list):  list of dict, each dict represents detections on one image
+        results (list): list of dict, each dict represents detections on one image
         output_file (str): path to JSON output file, should end in '.json'
         relative_path_base (str, optional): path to a directory as the base for relative paths, can
             be None if the paths in [results] are absolute

{megadetector-5.0.16 → megadetector-5.0.18}/megadetector/detection/video_utils.py

@@ -22,6 +22,7 @@ from functools import partial
 from inspect import signature
 
 from megadetector.utils import path_utils    
+from megadetector.utils.ct_utils import sort_list_of_dicts_by_key
 from megadetector.visualization import visualization_utils as vis_utils
 
 default_fourcc = 'h264'
@@ -197,7 +198,7 @@ def _filename_to_frame_number(filename):
 def _add_frame_numbers_to_results(results):
     """
     Given the 'images' list from a set of MD results that was generated on video frames,
-    add a 'frame_number' field to each image.
+    add a 'frame_number' field to each image, and return the list, sorted by frame number.
     
     Args:
         results (list): list of image dicts        
@@ -208,6 +209,9 @@ def _add_frame_numbers_to_results(results):
         fn = im['file']
         frame_number = _filename_to_frame_number(fn)
         im['frame_number'] = frame_number
+        
+    results = sort_list_of_dicts_by_key(results,'frame_number')
+    return results
     
 
 def run_callback_on_frames(input_video_file, 
@@ -332,8 +336,8 @@ def run_callback_on_frames_for_folder(input_video_folder,
         recursive (bool, optional): recurse into [input_video_folder]
     
     Returns:
-        dict: dict with keys 'video_filenames' (list), 'frame_rates' (list of floats), 'results' (list).
-        video_filenames will be *relative* filenames.
+        dict: dict with keys 'video_filenames' (list of str), 'frame_rates' (list of floats),
+        'results' (list of list of dicts). 'video_filenames' will contain *relative* filenames.
     """
     
     to_return = {'video_filenames':[],'frame_rates':[],'results':[]}
@@ -777,7 +781,8 @@ class FrameToVideoOptions:
         self.non_video_behavior = 'error'
     
 
-def frame_results_to_video_results(input_file,output_file,options=None):
+def frame_results_to_video_results(input_file,output_file,options=None,
+                                   video_filename_to_frame_rate=None):
     """
     Given an MD results file produced at the *frame* level, corresponding to a directory 
     created with video_folder_to_frames, maps those frame-level results back to the 
@@ -790,6 +795,8 @@ def frame_results_to_video_results(input_file,output_file,options=None):
         output_file (str): the .json file to which we should write video-level results
         options (FrameToVideoOptions, optional): parameters for converting frame-level results
             to video-level results, see FrameToVideoOptions for details            
+        video_filename_to_frame_rate (dict): maps (relative) video path names to frame rates,
+            used only to populate the output file
     """
 
     if options is None:
@@ -878,6 +885,10 @@ def frame_results_to_video_results(input_file,output_file,options=None):
         im_out['file'] = video_name
         im_out['detections'] = canonical_detections
         
+        if (video_filename_to_frame_rate is not None) and \
+            (video_name in video_filename_to_frame_rate):
+            im_out['frame_rate'] = video_filename_to_frame_rate[video_name]
+        
         # 'max_detection_conf' is no longer included in output files by default
         if False:
             im_out['max_detection_conf'] = 0

{megadetector-5.0.16 → megadetector-5.0.18}/megadetector/postprocessing/postprocess_batch_results.py

@@ -770,7 +770,7 @@ def _render_image_no_gt(file_info,detection_categories_to_results_name,
             if det['conf'] > max_conf:
                 max_conf = det['conf']
 
-            if ('classifications' in det):
+            if ('classifications' in det) and (len(det['classifications']) > 0):
 
                 # This is a list of [class,confidence] pairs, sorted by confidence
                 classifications = det['classifications']
@@ -1203,13 +1203,13 @@ def process_batch_results(options):
         # Rows / first index is ground truth, columns / second index is predicted category
         classifier_cm = collections.defaultdict(lambda: collections.defaultdict(lambda: 0))
 
-        # iDetection = 0; fn = detector_files[iDetection]; print(fn)
+        # i_detection = 0; fn = detector_files[i_detection]; print(fn)
         assert len(detector_files) == len(detections_df)
-        for iDetection, fn in enumerate(detector_files):
+        for i_detection, fn in enumerate(detector_files):
 
             image_id = ground_truth_indexed_db.filename_to_id[fn]
             image = ground_truth_indexed_db.image_id_to_image[image_id]
-            detections = detections_df['detections'].iloc[iDetection]
+            detections = detections_df['detections'].iloc[i_detection]
             pred_class_ids = [det['classifications'][0][0] \
                 for det in detections if 'classifications' in det.keys()]
             pred_classnames = [classification_categories[pd] for pd in pred_class_ids]

{megadetector-5.0.16 → megadetector-5.0.18}/megadetector/postprocessing/repeat_detection_elimination/repeat_detections_core.py

@@ -210,9 +210,12 @@ class RepeatDetectionOptions:
         #: a/b/c/RECONYX100 and a/b/c/RECONYX101 may really be the same camera).
         #:
         #: See ct_utils for a common replacement function that handles most common
-        #: manufacturer folder names.
+        #: manufacturer folder names:
+        #:
+        #: from megadetector.utils import ct_utils
+        #: self.customDirNameFunction = ct_utils.image_file_to_camera_folder
         self.customDirNameFunction = None
-        
+                
         #: Include only specific folders, mutually exclusive with [excludeFolders]
         self.includeFolders = None
         

{megadetector-5.0.16 → megadetector-5.0.18}/megadetector/utils/ct_utils.py

@@ -16,6 +16,8 @@ import os
 import jsonpickle
 import numpy as np
 
+from operator import itemgetter
+
 # List of file extensions we'll consider images; comparisons will be case-insensitive
 # (i.e., no need to include both .jpg and .JPG on this list).
 image_extensions = ['.jpg', '.jpeg', '.gif', '.png']
@@ -294,6 +296,29 @@ def get_max_conf(im):
     return max_conf
 
 
+def sort_results_for_image(im):
+    """
+    Sort classification and detection results in descending order by confidence (in place).
+    
+    Args:
+        im (dict): image dictionary in the MD output format (with a 'detections' field)
+    """
+    if 'detections' not in im or im['detections'] is None:
+        return
+
+    # Sort detections in descending order by confidence
+    im['detections'] = sort_list_of_dicts_by_key(im['detections'],k='conf',reverse=True)
+    
+    for det in im['detections']:
+        
+        # Sort classifications (which are (class,conf) tuples) in descending order by confidence
+        if 'classifications' in det and \
+            (det['classifications'] is not None) and \
+            (len(det['classifications']) > 0):
+            L = det['classifications']
+            det['classifications'] = sorted(L,key=itemgetter(1),reverse=True)
+
+
 def point_dist(p1,p2):
     """
     Computes the distance between two points, represented as length-two tuples.
@@ -406,6 +431,21 @@ def split_list_into_n_chunks(L, n, chunk_strategy='greedy'):
         raise ValueError('Invalid chunk strategy: {}'.format(chunk_strategy))
 
 
+def sort_list_of_dicts_by_key(L,k,reverse=False):
+    """
+    Sorts the list of dictionaries [L] by the key [k].
+    
+    Args:
+        L (list): list of dictionaries to sort
+        k (object, typically str): the sort key
+        reverse (bool, optional): whether to sort in reverse (descending) order
+        
+    Returns:
+        dict: sorted copy of [d]
+    """
+    return sorted(L, key=lambda d: d[k], reverse=reverse)
+    
+    
 def sort_dictionary_by_key(d,reverse=False):
     """
     Sorts the dictionary [d] by key.
@@ -611,3 +651,11 @@ if False:
     r1 = [0.4,0.8,10,22]; r2 = [100, 101, 200, 210.4]; assert abs(rect_distance(r1,r2)-119.753) < 0.001
     r1 = [0.4,0.8,10,22]; r2 = [101, 101, 200, 210.4]; assert abs(rect_distance(r1,r2)-120.507) < 0.001    
     r1 = [0.4,0.8,10,22]; r2 = [120, 120, 200, 210.4]; assert abs(rect_distance(r1,r2)-147.323) < 0.001
+
+    
+    #%% Test dictionary sorting
+    
+    L = [{'a':5},{'a':0},{'a':10}]
+    k = 'a'
+    sort_list_of_dicts_by_key(L, k, reverse=True)
+