megadetector 5.0.24__tar.gz → 5.0.26__tar.gz

{megadetector-5.0.24/megadetector.egg-info → megadetector-5.0.26}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: megadetector
-Version: 5.0.24
+Version: 5.0.26
 Summary: MegaDetector is an AI model that helps conservation folks spend less time doing boring things with camera trap images.
 Author-email: Your friendly neighborhood MegaDetector team <cameratraps@lila.science>
 Maintainer-email: Your friendly neighborhood MegaDetector team <cameratraps@lila.science>
@@ -30,7 +30,7 @@ Project-URL: Bug Reports, https://github.com/agentmorris/MegaDetector/issues
 Project-URL: Source, https://github.com/agentmorris/MegaDetector
 Keywords: camera traps,conservation,wildlife,ai,megadetector
 Classifier: Programming Language :: Python :: 3
-Requires-Python: <=3.13,>=3.9
+Requires-Python: <3.14,>=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: mkl==2024.0; sys_platform != "darwin"
@@ -51,6 +51,7 @@ Requires-Dist: dill
 Requires-Dist: ultralytics-yolov5==0.1.1
 Requires-Dist: yolov9pip==0.0.4
 Requires-Dist: python-dateutil
+Dynamic: license-file
 
 # MegaDetector
 
@@ -58,6 +59,8 @@ This package is a pip-installable version of the support/inference code for [Meg
 
 If you aren't looking for the Python package specifically, and you just want to learn more about what MegaDetector is all about, head over to the [MegaDetector repo](https://github.com/agentmorris/MegaDetector/?tab=readme-ov-file#megadetector).
 
+If you don't want to run MegaDetector, and you just want to use the utilities in this package - postprocessing, manipulating large volumes of camera trap images, etc. - you may want to check out the [megadetector-utils](https://pypi.org/project/megadetector-utils/) package, which is identical to this one, but excludes all of the PyTorch/YOLO dependencies, and is thus approximately one zillion times smaller.
+
 ## Installation
 
 Install with:

{megadetector-5.0.24 → megadetector-5.0.26}/README-package.md

@@ -4,6 +4,8 @@ This package is a pip-installable version of the support/inference code for [Meg
 
 If you aren't looking for the Python package specifically, and you just want to learn more about what MegaDetector is all about, head over to the [MegaDetector repo](https://github.com/agentmorris/MegaDetector/?tab=readme-ov-file#megadetector).
 
+If you don't want to run MegaDetector, and you just want to use the utilities in this package - postprocessing, manipulating large volumes of camera trap images, etc. - you may want to check out the [megadetector-utils](https://pypi.org/project/megadetector-utils/) package, which is identical to this one, but excludes all of the PyTorch/YOLO dependencies, and is thus approximately one zillion times smaller.
+
 ## Installation
 
 Install with:

{megadetector-5.0.24 → megadetector-5.0.26}/README.md

@@ -10,12 +10,16 @@
 5. [Repo contents](#repo-contents)
 6. [Contact](#contact)
 7. [Gratuitous camera trap picture](#gratuitous-camera-trap-picture)
+8. [License](#license)
+9. [Contributing](#contributing)
 
 
 ## What's MegaDetector all about?
 
 [MegaDetector](megadetector.md) is an AI model that identifies animals, people, and vehicles in camera trap images (which also makes it useful for eliminating blank images).  This model is trained on several million images from a variety of ecosystems.
 
+MegaDetector only finds animals, it doesn't identify them to species level.  If you're looking for a species classifier, check out [SpeciesNet](https://github.com/google/cameratrapai), a species classifier that plays nicely with MegaDetector.
+
 Here's a “teaser” image of what MegaDetector output looks like:
 
 ![Red bounding box on fox](images/detector_example.jpg)<br/>Image credit University of Washington.
@@ -27,7 +31,7 @@ Here's a &ldquo;teaser&rdquo; image of what MegaDetector output looks like:
 * If you're just <i>considering</i> the use of AI in your workflow, and you aren't even sure yet whether MegaDetector would be useful to you, we recommend reading the "[getting started with MegaDetector](getting-started.md)" page.
 * If you're already familiar with MegaDetector and you're ready to run it on your data, see the [MegaDetector User Guide](megadetector.md) for instructions on running MegaDetector.
 * If you're a programmer-type looking to use tools from this repo, check out the [MegaDetector Python package](https://pypi.org/project/megadetector/) that provides access to everything in this repo (yes, you guessed it, "pip install megadetector").
-* If you have any questions, or you want to tell us that MegaDetector was amazing/terrible on your images, <a href="mailto:cameratraps@lila.science">email us</a>!
+* If you have any questions, or you want to tell us that MegaDetector was amazing/terrible on your images, or you have a zillion images and you want some help digging out of that backlog, <a href="mailto:cameratraps@lila.science">email us</a>!
 
 MegaDetector is just one of many tools that aims to make conservation biologists more efficient with AI.  If you want to learn about other ways to use AI to accelerate camera trap workflows, check out our of the field, affectionately titled &ldquo;[Everything I know about machine learning and camera traps](https://agentmorris.github.io/camera-trap-ml-survey/)&rdquo;.
 
@@ -45,9 +49,9 @@ Here are a few of the organizations that have used MegaDetector... we're only li
 * [Canadian Parks and Wilderness Society (CPAWS) Northern Alberta Chapter](https://cpawsnab.org/)
 * [Conservation X Labs](https://conservationxlabs.com/)
 * [Czech University of Life Sciences Prague](https://www.czu.cz/en)
-* [Dudek Camera Trap AI Image Toolkit (AIT)](https://dudek.com/services/wildlife-camera-trap-ai-image-processing-and-management/)
+* [Dudek Camera Trap AI Image Toolkit (AIT)](https://ait.dudek.com/)
 * [EcoLogic Consultants Ltd.](https://www.consult-ecologic.com/)
-* [Estación Biológica de Doñana](http://www.ebd.csic.es/inicio)
+* [Estación Biológica de Doñana](https://www.ebd.csic.es/)
 * [Indigenous Desert Alliance](https://www.indigenousdesertalliance.com/)
 * [Myall Lakes Dingo Project](https://carnivorecoexistence.info/myall-lakes-dingo-project/)
 * [Norwegian Institute for Nature Research](https://www.nina.no/english/Home)
@@ -84,7 +88,6 @@ Here are a few of the organizations that have used MegaDetector... we're only li
 * [Department of Fish and Wildlife Sciences](https://www.uidaho.edu/cnr/departments/fish-and-wildlife-sciences), University of Idaho
 * [Department of Society & Conservation](https://www.umt.edu/environment/about/departments/socon/), W.A. Franke College of Forestry & Conservation, University of Montana
 * [Department of Wildlife Ecology and Conservation](https://wec.ifas.ufl.edu/), University of Florida
-* [Ecology and Conservation of Amazonian Vertebrates Research Group](https://www.researchgate.net/lab/Fernanda-Michalski-Lab-4), Federal University of Amapá
 * [Gola Forest Programme](https://www.rspb.org.uk/our-work/conservation/projects/scientific-support-for-the-gola-forest-programme/), Royal Society for the Protection of Birds (RSPB)
 * [Graeme Shannon's Research Group](https://wildliferesearch.co.uk/group-1), Bangor University 
 * [Grizzly Bear Recovery Program](https://www.fws.gov/office/grizzly-bear-recovery-program), U.S. Fish & Wildlife Service
@@ -102,12 +105,13 @@ Here are a few of the organizations that have used MegaDetector... we're only li
 * [Santa Monica Mountains Recreation Area](https://www.nps.gov/samo/index.htm), National Park Service
 * [Seattle Urban Carnivore Project](https://www.zoo.org/seattlecarnivores), Woodland Park Zoo
 * [Serra dos Órgãos National Park](https://www.icmbio.gov.br/parnaserradosorgaos/), ICMBio
-* [Snapshot USA](https://emammal.si.edu/snapshot-usa), Smithsonian
+* [Snapshot USA](https://www.snapshot-usa.org/), Smithsonian
 * [TROPECOLNET project](https://www.anabenitezlopez.com/research/global-change-biology/tropecolnet/), Museo Nacional de Ciencias Naturales
 * [Wildlife Coexistence Lab](https://wildlife.forestry.ubc.ca/), University of British Columbia
 * [Wildlife Research](https://www.dfw.state.or.us/wildlife/research/index.asp), Oregon Department of Fish and Wildlife
 * [Wildlife Division](https://www.michigan.gov/dnr/about/contact/wildlife), Michigan Department of Natural Resources
 * [Kohl Wildlife Lab](https://kohlwildlifelab.com/), University of Georgia
+* Ecology and Conservation of Amazonian Vertebrates Research Group, Federal University of Amapá
 * Department of Ecology, TU Berlin
 * Ghost Cat Analytics
 * Protected Areas Unit, Canadian Wildlife Service
@@ -132,7 +136,7 @@ Here are a few of the organizations that have used MegaDetector... we're only li
 * [BirdLife Malta](https://birdlifemalta.org/) ([tweet](https://x.com/BirdLife_Malta/status/1817456839862173783?t=S-KRiZ5R1-CoW8-tbYNjqQ&s=03)) ([LI post](https://www.linkedin.com/posts/birdlifemalta_worldnatureconservationday-shearwater-colony-activity-7223220656589463553-X2Mc/?utm_source=share&utm_medium=member_android))
 * [Endangered Landscapes and Seascapes Programme](https://www.endangeredlandscapes.org/), Cambridge Conservation Initiative ([blog post](https://www.endangeredlandscapes.org/news/ai-for-wildlife-monitoring-a-real-time-alert-system-for-bears-and-wild-boars-in-romanias-carpathian-mountains/))
 
-* [Road Ecology Center](https://roadecology.ucdavis.edu/), University of California, Davis ([Wildlife Observer Network platform](https://wildlifeobserver.net/))
+* [Road Ecology Center](https://roadecology.ucdavis.edu/), University of California, Davis ([Wildlife Observer Network platform](https://roadecology.ucdavis.edu/research/projects/wildlife-observer-network))
 * [The Nature Conservancy in California](https://www.nature.org/en-us/about-us/where-we-work/united-states/california/) ([Animl platform](https://github.com/tnc-ca-geo/animl-frontend)) ([story](https://www.vision-systems.com/non-factory/environment-agriculture/article/14304433/the-nature-conservancy-brings-cameras-ai-to-invasive-species-prevention))
 * [San Diego Zoo Wildlife Alliance](https://science.sandiegozoo.org/)  ([Animl R package](https://github.com/conservationtechlab/animl))
 * [TerrOïko](https://www.terroiko.fr/) ([OCAPI platform](https://www.terroiko.fr/ocapi))
@@ -202,9 +206,9 @@ Code for hosting our models as an API, either for synchronous operation (i.e., f
 
 #### megadetector/classification
 
-Experimental code for training species classifiers on new data sets, generally trained on MegaDetector crops.  Currently the main pipeline described in this folder relies on a large database of labeled images that is not publicly available; therefore, this folder is not yet set up to facilitate training of your own classifiers.  However, it is useful for <i>users</i> of the classifiers that we train, and contains some useful starting points if you are going to take a "DIY" approach to training classifiers on cropped images.  
+This folder is largely deprecated thanks to the release of [SpeciesNet](https://github.com/google/cameratrapai), a species classifier that is better than any of the classifiers we ever trained with the stuff in this folder.  That said, this folder contains code for training species classifiers on new data sets, generally trained on MegaDetector crops.
 
-All that said, here's another "teaser image" of what you get at the end of training and running a classifier:
+Here's another "teaser image" of what you get at the end of training and running a classifier:
 
 <img src="images/warthog_classifications.jpg" width="700"><br/>Image credit University of Minnesota, from the Snapshot Safari program.
 

{megadetector-5.0.24 → megadetector-5.0.26}/megadetector/data_management/cct_json_utils.py

@@ -300,7 +300,10 @@ class SequenceOptions:
     def __init__(self):
         #: Images separated by <= this duration will be grouped into the same sequence.
         self.episode_interval_seconds = 60.0
-
+        
+        #: How to handle invalid datetimes: 'error' or 'none'
+        self.datetime_conversion_failure_behavior = 'none'
+        
     
 #%% Functions
 
@@ -445,7 +448,17 @@ def create_sequences(image_info,options=None):
         raise ValueError('Unrecognized type for [image_info]')
         
     # Modifies the images in place
-    _ = parse_datetimes_from_cct_image_list(image_info)
+    _ = parse_datetimes_from_cct_image_list(image_info, 
+            conversion_failure_behavior=options.datetime_conversion_failure_behavior)
+    
+    n_invalid_datetimes = 0
+    for im in image_info:
+        if not isinstance(im['datetime'],datetime.datetime):
+            assert im['datetime'] is None, 'At this point, datetimes should be valid or None'
+            n_invalid_datetimes += 1
+    if n_invalid_datetimes > 0:
+        print('Warning: {} of {} images have invalid datetimes'.format(
+            n_invalid_datetimes,len(image_info)))
     
     # Find all unique locations
     locations = set()

{megadetector-5.0.24 → megadetector-5.0.26}/megadetector/data_management/coco_to_yolo.py

@@ -47,6 +47,9 @@ def write_yolo_dataset_file(yolo_dataset_file,
         class_list (list or str): an ordered list of class names (the first item will be class 0, 
             etc.), or the name of a text file containing an ordered list of class names (one per 
             line, starting from class zero).
+        train_folder_relative (str, optional): train folder name, used only to populate dataset.yaml
+        val_folder_relative (str, optional): val folder name, used only to populate dataset.yaml
+        test_folder_relative (str, optional): test folder name, used only to populate dataset.yaml        
     """
     
     # Read class names
@@ -97,7 +100,7 @@ def coco_to_yolo(input_image_folder,
                  category_names_to_exclude=None,
                  category_names_to_include=None,
                  write_output=True,
-                 flatten_paths=True):
+                 flatten_paths=False):
     """
     Converts a COCO-formatted dataset to a YOLO-formatted dataset, optionally flattening the 
     dataset to a single folder in the process.
@@ -116,17 +119,21 @@ def coco_to_yolo(input_image_folder,
             images are left alone.
         source_format (str, optional): can be 'coco' (default) or 'coco_camera_traps'.  The only difference
             is that when source_format is 'coco_camera_traps', we treat an image with a non-bbox
-            annotation with a category id of 0 as a special case, i.e. that's how an empty image
-            is indicated.  The original COCO standard is a little ambiguous on this issue.  If
-            source_format is 'coco', we either treat images as empty or error, depending on the value
-            of [allow_empty_annotations].  [allow_empty_annotations] has no effect if source_format is
-            'coco_camera_traps'.
+            annotation as a special case, i.e. that's how an empty image is indicated.  The original 
+            COCO standard is a little ambiguous on this issue.  If source_format is 'coco', we 
+            either treat images as empty or error, depending on the value of [allow_empty_annotations].
+            [allow_empty_annotations] has no effect if source_format is 'coco_camera_traps'.
+        overwrite_images (bool, optional): over-write images in the output folder if they exist
         create_image_and_label_folder (bool, optional): whether to create separate folders called 'images' and
             'labels' in the YOLO output folder.  If create_image_and_label_folders is False, 
             a/b/c/image001.jpg will become a#b#c#image001.jpg, and the corresponding text file will 
             be a#b#c#image001.txt.  If create_image_and_label_folders is True, a/b/c/image001.jpg will become 
             images/a#b#c#image001.jpg, and the corresponding text file will be 
             labels/a#b#c#image001.txt.    
+        class_file_name (str, optional): .txt file (relative to the output folder) that we should 
+            populate with a list of classes (or None to omit)
+        allow_empty_annotations (bool, optional): if this is False and [source_format] is 'coco', 
+            we'll error on annotations that have no 'bbox' field
         clip_boxes (bool, optional): whether to clip bounding box coordinates to the range [0,1] before
             converting to YOLO xywh format
         image_id_to_output_image_json_file (str, optional): an optional *output* file, to which we will write
@@ -139,12 +146,14 @@ def coco_to_yolo(input_image_folder,
         category_names_to_exclude (str, optional): category names that should not be represented in the
             YOLO output; only impacts annotations, does not prevent copying images.  There's almost no reason
             you would want to specify this and [category_names_to_include]. 
-        category_names_to_include (str, optional): allow-list of category names that should be represented in the
-            YOLO output; only impacts annotations, does not prevent copying images.  There's almost no reason
-            you would want to specify this and [category_names_to_exclude]. 
+        category_names_to_include (str, optional): allow-list of category names that should be represented 
+            in the YOLO output; only impacts annotations, does not prevent copying images.  There's almost 
+            no reason you would want to specify this and [category_names_to_exclude]. 
         write_output (bool, optional): determines whether we actually copy images and write annotations;
             setting this to False mostly puts this function in "dry run" "mode.  The class list
             file is written regardless of the value of write_output.
+        flatten_paths (bool, optional): replace /'s in image filenames with [path_replacement_char],
+            which ensures that the output folder is a single flat folder.
     
     Returns:
         dict: information about the coco --> yolo mapping, containing at least the fields:
@@ -313,9 +322,9 @@ def coco_to_yolo(input_image_folder,
                     
                     elif source_format == 'coco_camera_traps':
                         
-                        # We allow empty bbox lists in COCO camera traps; this is typically a negative
-                        # example in a dataset that has bounding boxes, and 0 is typically the empty 
-                        # category.
+                        # We allow empty bbox lists in COCO camera traps files; this is typically a 
+                        # negative example in a dataset that has bounding boxes, and 0 is typically 
+                        # the empty category, which is typically 0.
                         if ann['category_id'] != 0:
                             if not printed_empty_annotation_warning:
                                 printed_empty_annotation_warning = True
@@ -429,13 +438,14 @@ def coco_to_yolo(input_image_folder,
     
     print('Generating class list')
     
-    class_list_filename = os.path.join(output_folder,class_file_name)
-    with open(class_list_filename, 'w') as f:
-        print('Writing class list to {}'.format(class_list_filename))
-        for i_class in range(0,len(yolo_id_to_name)):
-            # Category IDs should range from 0..N-1
-            assert i_class in yolo_id_to_name
-            f.write(yolo_id_to_name[i_class] + '\n')
+    if class_file_name is not None:
+        class_list_filename = os.path.join(output_folder,class_file_name)
+        with open(class_list_filename, 'w') as f:
+            print('Writing class list to {}'.format(class_list_filename))
+            for i_class in range(0,len(yolo_id_to_name)):
+                # Category IDs should range from 0..N-1
+                assert i_class in yolo_id_to_name
+                f.write(yolo_id_to_name[i_class] + '\n')
     
     if image_id_to_output_image_json_file is not None:
         print('Writing image ID mapping to {}'.format(image_id_to_output_image_json_file))
@@ -457,6 +467,9 @@ def coco_to_yolo(input_image_folder,
         
     source_image_to_dest_image = {}
     
+    label_files_written = []
+    n_boxes_written = 0
+    
     # TODO: parallelize this loop
     #
     # output_info = images_to_copy[0]
@@ -471,6 +484,7 @@ def coco_to_yolo(input_image_folder,
         
         source_image_to_dest_image[source_image] = dest_image
         
+        # Copy the image if necessary
         if write_output:
             
             os.makedirs(os.path.dirname(dest_image),exist_ok=True)        
@@ -482,17 +496,24 @@ def coco_to_yolo(input_image_folder,
             if (not os.path.isfile(dest_image)) or (overwrite_images):
                 shutil.copyfile(source_image,dest_image)
         
-            bboxes = output_info['bboxes']        
+        bboxes = output_info['bboxes']        
+        
+        # Write the annotation file if necessary
+        #
+        # Only write an annotation file if there are bounding boxes.  Images with 
+        # no .txt files are treated as hard negatives, at least by YOLOv5:
+        #
+        # https://github.com/ultralytics/yolov5/issues/3218
+        #
+        # I think this is also true for images with empty .txt files, but 
+        # I'm using the convention suggested on that issue, i.e. hard 
+        # negatives are expressed as images without .txt files.
+        if len(bboxes) > 0:
             
-            # Only write an annotation file if there are bounding boxes.  Images with 
-            # no .txt files are treated as hard negatives, at least by YOLOv5:
-            #
-            # https://github.com/ultralytics/yolov5/issues/3218
-            #
-            # I think this is also true for images with empty .txt files, but 
-            # I'm using the convention suggested on that issue, i.e. hard 
-            # negatives are expressed as images without .txt files.
-            if len(bboxes) > 0:
+            n_boxes_written += len(bboxes)
+            label_files_written.append(dest_txt)
+            
+            if write_output:
                 
                 with open(dest_txt,'w') as f:
                     
@@ -501,8 +522,7 @@ def coco_to_yolo(input_image_folder,
                         assert len(bbox) == 5
                         s = '{} {} {} {} {}'.format(bbox[0],bbox[1],bbox[2],bbox[3],bbox[4])
                         f.write(s + '\n')
-                        
-        # ...if we're actually writing output
+                            
         
     # ...for each image
     
@@ -510,6 +530,8 @@ def coco_to_yolo(input_image_folder,
     coco_to_yolo_info['class_list_filename'] = class_list_filename
     coco_to_yolo_info['source_image_to_dest_image'] = source_image_to_dest_image
     coco_to_yolo_info['coco_id_to_yolo_id'] = coco_id_to_yolo_id
+    coco_to_yolo_info['label_files_written'] = label_files_written
+    coco_to_yolo_info['n_boxes_written'] = n_boxes_written
     
     return coco_to_yolo_info
 

{megadetector-5.0.24 → megadetector-5.0.26}/megadetector/data_management/databases/combine_coco_camera_traps_files.py

@@ -24,8 +24,10 @@ import sys
 
 #%% Merge functions
 
-def combine_cct_files(input_files, output_file=None, require_uniqueness=True,
-                      filename_prefixes=None):                             
+def combine_cct_files(input_files, 
+                      output_file=None, 
+                      require_uniqueness=True,
+                      filename_prefixes=None):
     """
     Merges the list of COCO Camera Traps files [input_files] into a single
     dictionary, optionally writing the result to [output_file].
@@ -33,8 +35,10 @@ def combine_cct_files(input_files, output_file=None, require_uniqueness=True,
     Args:
         input_files (list): paths to CCT .json files
         output_file (str, optional): path to write merged .json file
-        require_uniqueness (bool): whether to require that the images in
+        require_uniqueness (bool, optional): whether to require that the images in
             each input_dict be unique
+        filename_prefixes (dict, optional): dict mapping input filenames to strings
+            that should be prepended to image filenames from that source            
     
     Returns:
         dict: the merged COCO-formatted .json dict

{megadetector-5.0.24 → megadetector-5.0.26}/megadetector/data_management/databases/integrity_check_json_db.py

@@ -327,7 +327,7 @@ def integrity_check_json_db(jsonFile, options=None):
                 
         for i_image,result in enumerate(results):
             if result is not None:
-                validation_errors.append(images[i_image]['file_name'],result)
+                validation_errors.append((images[i_image]['file_name'],result))
                             
     # ...for each image
     
@@ -393,7 +393,7 @@ def integrity_check_json_db(jsonFile, options=None):
             elif image['_count'] > 1:
                 nMultiAnnotated += 1
                 
-        print('Found {} unannotated images, {} images with multiple annotations'.format(
+        print('\nFound {} unannotated images, {} images with multiple annotations'.format(
                 nUnannotated,nMultiAnnotated))
         
         if (len(base_dir) > 0) and options.bFindUnusedImages:

megadetector-5.0.26/megadetector/data_management/lila/add_locations_to_island_camera_traps.py

@@ -0,0 +1,101 @@
+"""
+
+add_locations_to_island_camera_traps.py
+
+The Island Conservation Camera Traps dataset had unique camera identifiers embedded
+in filenames, but not in the proper metadata fields.  This script copies that information
+to metadata.
+
+"""
+
+#%% Imports and constants
+
+import os
+import json
+from tqdm import tqdm
+
+input_fn = os.path.expanduser('~/lila/metadata/island_conservation.json')
+output_fn = os.path.expanduser('~/tmp/island_conservation.json')
+preview_folder = os.path.expanduser('~/tmp/island_conservation_preview')
+image_directory = os.path.expanduser('~/data/icct/public/')
+
+
+#%% Prevent imports during testing
+
+if False:
+
+    #%% Read input file
+    
+    with open(input_fn,'r') as f:
+        d = json.load(f)
+        
+    d['info']
+    d['info']['version'] = '1.01'
+    
+    
+    #%% Find locations
+    
+    images = d['images']
+    
+    locations = set()
+    
+    for i_image,im in tqdm(enumerate(images),total=len(images)):
+        tokens_fn = im['file_name'].split('/')
+        tokens_id = im['id'].split('_')
+        assert tokens_fn[0] == tokens_id[0]
+        assert tokens_fn[1] == tokens_id[1]
+        location = tokens_fn[0] + '_' + tokens_fn[1]
+        im['location'] = location
+        locations.add(location)
+    
+    locations = sorted(list(locations))
+        
+    for s in locations:
+        print(s)
+        
+        
+    #%% Write output file
+    
+    with open(output_fn,'w') as f:
+        json.dump(d,f,indent=1)
+        
+    
+    #%% Validate .json files
+    
+    from megadetector.data_management.databases import integrity_check_json_db
+    
+    options = integrity_check_json_db.IntegrityCheckOptions()
+    options.baseDir = image_directory
+    options.bCheckImageSizes = False
+    options.bCheckImageExistence = True
+    options.bFindUnusedImages = True
+    
+    sorted_categories, data, error_info = integrity_check_json_db.integrity_check_json_db(output_fn, options)
+    
+    
+    #%% Preview labels
+    
+    from megadetector.visualization import visualize_db
+    
+    viz_options = visualize_db.DbVizOptions()
+    viz_options.num_to_visualize = 2000
+    viz_options.trim_to_images_with_bboxes = False
+    viz_options.add_search_links = False
+    viz_options.sort_by_filename = False
+    viz_options.parallelize_rendering = True
+    viz_options.classes_to_exclude = ['test']
+    html_output_file, image_db = visualize_db.visualize_db(db_path=output_fn,
+                                                             output_dir=preview_folder,
+                                                             image_base_dir=image_directory,
+                                                             options=viz_options)
+    
+    from megadetector.utils import path_utils
+    path_utils.open_file(html_output_file)
+    
+    
+    #%% Zip output file
+    
+    from megadetector.utils.path_utils import zip_file
+    
+    zip_file(output_fn, verbose=True)
+    assert os.path.isfile(output_fn + '.zip')

megadetector-5.0.26/megadetector/data_management/lila/add_locations_to_nacti.py

@@ -0,0 +1,151 @@
+"""
+
+add_locations_to_nacti.py
+
+As of 10.2023, NACTI metadata only has very coarse location information (e.g. "Florida"),
+but camera IDs are embedded in filenames.  This script pulls that information from filenames
+and adds it to metadata.
+
+"""
+
+#%% Imports and constants
+
+import os
+import json
+import shutil
+
+from tqdm import tqdm
+from collections import defaultdict
+
+input_file = r'd:\lila\nacti\nacti_metadata.json.1.13\nacti_metadata.json'
+output_file = r'g:\temp\nacti_metadata.1.14.json'
+
+
+#%% Prevent execution during testing
+
+if False:
+
+    #%% Read metadata
+    
+    with open(input_file,'r') as f:
+        d = json.load(f)
+        
+    assert d['info']['version'] == 1.13
+    
+    
+    #%% Map images to locations (according to the metadata)
+    
+    file_name_to_original_location = {}
+    
+    # im = dataset_labels['images'][0]
+    for im in tqdm(d['images']):
+        file_name_to_original_location[im['file_name']] = im['location']
+    
+    original_locations = set(file_name_to_original_location.values())
+    
+    print('Found {} locations in the original metadata:'.format(len(original_locations)))
+    for loc in original_locations:
+        print('[{}]'.format(loc))
+    
+    
+    #%% Map images to new locations
+    
+    def path_to_location(relative_path):
+        
+        relative_path = relative_path.replace('\\','/')
+        if relative_path in file_name_to_original_location:
+            location_name = file_name_to_original_location[relative_path]
+            if location_name == 'San Juan Mntns, Colorado':
+                # "part0/sub000/2010_Unit150_Ivan097_img0003.jpg"
+                tokens = relative_path.split('/')[-1].split('_')
+                assert tokens[1].startswith('Unit')
+                location_name = 'sanjuan_{}_{}_{}'.format(tokens[0],tokens[1],tokens[2])
+            elif location_name == 'Lebec, California':
+                # "part0/sub035/CA-03_08_13_2015_CA-03_0009738.jpg"
+                tokens = relative_path.split('/')[-1].split('_')
+                assert tokens[0].startswith('CA-') or tokens[0].startswith('TAG-')
+                location_name = 'lebec_{}'.format(tokens[0])   
+            elif location_name == 'Archbold, FL':
+                # "part1/sub110/FL-01_01_25_2016_FL-01_0040421.jpg"
+                tokens = relative_path.split('/')[-1].split('_')
+                assert tokens[0].startswith('FL-')
+                location_name = 'archbold_{}'.format(tokens[0])   
+            else:
+                assert location_name == ''
+                tokens = relative_path.split('/')[-1].split('_')
+                if tokens[0].startswith('CA-') or tokens[0].startswith('TAG-') or tokens[0].startswith('FL-'):
+                    location_name = '{}'.format(tokens[0])
+                
+        else:
+            
+            location_name = 'unknown'    
+        
+        # print('Returning location {} for file {}'.format(location_name,relative_path))          
+    
+        return location_name
+    
+    file_name_to_updated_location = {}
+    updated_location_to_count = defaultdict(int)
+    for im in tqdm(d['images']):
+        
+        updated_location = path_to_location(im['file_name'])
+        file_name_to_updated_location[im['file_name']] = updated_location
+        updated_location_to_count[updated_location] += 1
+    
+    updated_location_to_count = {k: v for k, v in sorted(updated_location_to_count.items(), 
+                             key=lambda item: item[1],
+                             reverse=True)}
+    
+    updated_locations = set(file_name_to_updated_location.values())
+    
+    print('Found {} updated locations in the original metadata:'.format(len(updated_locations)))
+    for loc in updated_location_to_count:
+        print('{}: {}'.format(loc,updated_location_to_count[loc]))
+    
+    
+    #%% Re-write metadata
+    
+    for im in d['images']:
+        im['location'] = file_name_to_updated_location[im['file_name']]
+    d['info']['version'] = 1.14
+    
+    with open(output_file,'w') as f:
+        json.dump(d,f,indent=1)
+        
+    
+    #%% For each location, sample some random images to make sure they look consistent
+    
+    input_base = r'd:\lila\nacti-unzipped'
+    assert os.path.isdir(input_base)
+    
+    location_to_images = defaultdict(list)
+    
+    for im in d['images']:
+        location_to_images[im['location']].append(im)
+        
+    n_to_sample = 10
+    import random
+    random.seed(0)
+    sampling_folder_base = r'g:\temp\nacti_samples'
+    
+    for location in tqdm(location_to_images):
+        
+        images_this_location = location_to_images[location]
+        if len(images_this_location) > n_to_sample:
+            images_this_location = random.sample(images_this_location,n_to_sample)
+            
+        for i_image,im in enumerate(images_this_location):
+            
+            fn_relative = im['file_name']
+            source_fn_abs = os.path.join(input_base,fn_relative)
+            assert os.path.isfile(source_fn_abs)
+            ext = os.path.splitext(fn_relative)[1]            
+            target_fn_abs = os.path.join(sampling_folder_base,'{}/{}'.format(
+                location,'image_{}{}'.format(str(i_image).zfill(2),ext)))
+            os.makedirs(os.path.dirname(target_fn_abs),exist_ok=True)
+            shutil.copyfile(source_fn_abs,target_fn_abs)
+            
+        # ...for each image
+        
+    # ...for each location
+                

{megadetector-5.0.24 → megadetector-5.0.26}/megadetector/data_management/lila/generate_lila_per_image_labels.py

@@ -349,7 +349,7 @@ with open(output_file,'w',encoding='utf-8',newline='') as f:
 
 # ...with open()
 
-print('Processed {} datasets'.format(len(metadata_table)))
+print('\nProcessed {} datasets'.format(len(metadata_table)))
 
 
 #%% Read the .csv back
@@ -393,7 +393,7 @@ def check_row(row):
     dataset_name_to_locations[ds_name].add(row['location_id'])
     
 # Faster, but more annoying to debug
-if False:
+if True:
     
     df.progress_apply(check_row, axis=1)