megadetector 5.0.23__tar.gz → 5.0.25__tar.gz

{megadetector-5.0.23/megadetector.egg-info → megadetector-5.0.25}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: megadetector
-Version: 5.0.23
+Version: 5.0.25
 Summary: MegaDetector is an AI model that helps conservation folks spend less time doing boring things with camera trap images.
 Author-email: Your friendly neighborhood MegaDetector team <cameratraps@lila.science>
 Maintainer-email: Your friendly neighborhood MegaDetector team <cameratraps@lila.science>
@@ -29,26 +29,27 @@ Project-URL: Documentation, https://megadetector.readthedocs.io
 Project-URL: Bug Reports, https://github.com/agentmorris/MegaDetector/issues
 Project-URL: Source, https://github.com/agentmorris/MegaDetector
 Keywords: camera traps,conservation,wildlife,ai,megadetector
-Classifier: Development Status :: 3 - Alpha
-Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
 Requires-Python: <=3.13,>=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: mkl==2024.0; sys_platform != "darwin"
+Requires-Dist: numpy<2.0,>=1.26.4
 Requires-Dist: Pillow>=9.5
 Requires-Dist: tqdm>=4.64.0
 Requires-Dist: jsonpickle>=3.0.2
 Requires-Dist: humanfriendly>=10.0
-Requires-Dist: numpy<2.0,>=1.26.4
 Requires-Dist: matplotlib>=3.8.0
 Requires-Dist: opencv-python>=4.8.0
 Requires-Dist: requests>=2.31.0
 Requires-Dist: pyqtree>=1.0.0
-Requires-Dist: seaborn>=0.12.2
 Requires-Dist: scikit-learn>=1.3.1
 Requires-Dist: pandas>=2.1.1
-Requires-Dist: PyYAML>=6.0.1
+Requires-Dist: python-dateutil
+Requires-Dist: send2trash
+Requires-Dist: dill
 Requires-Dist: ultralytics-yolov5==0.1.1
+Requires-Dist: yolov9pip==0.0.4
 Requires-Dist: python-dateutil
 
 # MegaDetector
@@ -57,28 +58,15 @@ This package is a pip-installable version of the support/inference code for [Meg
 
 If you aren't looking for the Python package specifically, and you just want to learn more about what MegaDetector is all about, head over to the [MegaDetector repo](https://github.com/agentmorris/MegaDetector/?tab=readme-ov-file#megadetector).
 
+If you don't want to run MegaDetector, and you just want to use the utilities in this package - postprocessing, manipulating large volumes of camera trap images, etc. - you may want to check out the [megadetector-utils](https://pypi.org/project/megadetector-utils/) package, which is identical to this one, but excludes all of the PyTorch/YOLO dependencies, and is thus approximately one zillion times smaller.
 
-## Reasons you might not be looking for this package
-
-### If you are an ecologist...
-
-If you are an ecologist looking to use MegaDetector to help you get through your camera trap images, you probably don't want this package, or at least you probably don't want to start at this page.  We recommend starting with our "[Getting started with MegaDetector](https://github.com/agentmorris/MegaDetector/blob/main/getting-started.md)" page, then digging in to the [MegaDetector User Guide](https://github.com/agentmorris/MegaDetector/blob/main/megadetector.md), which will walk you through the process of using MegaDetector.
-
-### If you are a computer-vision-y type...
-
-If you are a computer-vision-y person looking to run or fine-tune MegaDetector programmatically, you probably don't want this package.  MegaDetector is just a fine-tuned version of [YOLOv5](https://github.com/ultralytics/yolov5), and the [ultralytics](https://github.com/ultralytics/ultralytics/) package (from the developers of YOLOv5) has a zillion bells and whistles for both inference and fine-tuning that this package doesn't.
-
-## Reasons you might want to use this package
-
-If you want to programmatically interact with the postprocessing tools from the MegaDetector repo, or programmatically run MegaDetector in a way that produces [Timelapse](https://saul.cpsc.ucalgary.ca/timelapse)-friendly output (i.e., output in the standard [MegaDetector output format](https://github.com/agentmorris/MegaDetector/tree/main/megadetector/api/batch_processing#megadetector-batch-output-format)), this package might be for you.
-
-## If I haven't talked you out of using this package...
+## Installation
 
-To install:
+Install with:
 
 `pip install megadetector`
 
-MegaDetector model weights aren't downloaded at pip-install time, but they will be (optionally) automatically downloaded the first time you run the model.
+MegaDetector model weights aren't downloaded at the time you install the package, but they will be (optionally) automatically downloaded the first time you run the model.
 
 ## Package reference
 

{megadetector-5.0.23 → megadetector-5.0.25}/README-package.md

@@ -4,28 +4,15 @@ This package is a pip-installable version of the support/inference code for [Meg
 
 If you aren't looking for the Python package specifically, and you just want to learn more about what MegaDetector is all about, head over to the [MegaDetector repo](https://github.com/agentmorris/MegaDetector/?tab=readme-ov-file#megadetector).
 
+If you don't want to run MegaDetector, and you just want to use the utilities in this package - postprocessing, manipulating large volumes of camera trap images, etc. - you may want to check out the [megadetector-utils](https://pypi.org/project/megadetector-utils/) package, which is identical to this one, but excludes all of the PyTorch/YOLO dependencies, and is thus approximately one zillion times smaller.
 
-## Reasons you might not be looking for this package
+## Installation
 
-### If you are an ecologist...
-
-If you are an ecologist looking to use MegaDetector to help you get through your camera trap images, you probably don't want this package, or at least you probably don't want to start at this page.  We recommend starting with our "[Getting started with MegaDetector](https://github.com/agentmorris/MegaDetector/blob/main/getting-started.md)" page, then digging in to the [MegaDetector User Guide](https://github.com/agentmorris/MegaDetector/blob/main/megadetector.md), which will walk you through the process of using MegaDetector.
-
-### If you are a computer-vision-y type...
-
-If you are a computer-vision-y person looking to run or fine-tune MegaDetector programmatically, you probably don't want this package.  MegaDetector is just a fine-tuned version of [YOLOv5](https://github.com/ultralytics/yolov5), and the [ultralytics](https://github.com/ultralytics/ultralytics/) package (from the developers of YOLOv5) has a zillion bells and whistles for both inference and fine-tuning that this package doesn't.
-
-## Reasons you might want to use this package
-
-If you want to programmatically interact with the postprocessing tools from the MegaDetector repo, or programmatically run MegaDetector in a way that produces [Timelapse](https://saul.cpsc.ucalgary.ca/timelapse)-friendly output (i.e., output in the standard [MegaDetector output format](https://github.com/agentmorris/MegaDetector/tree/main/megadetector/api/batch_processing#megadetector-batch-output-format)), this package might be for you.
-
-## If I haven't talked you out of using this package...
-
-To install:
+Install with:
 
 `pip install megadetector`
 
-MegaDetector model weights aren't downloaded at pip-install time, but they will be (optionally) automatically downloaded the first time you run the model.
+MegaDetector model weights aren't downloaded at the time you install the package, but they will be (optionally) automatically downloaded the first time you run the model.
 
 ## Package reference
 

{megadetector-5.0.23 → megadetector-5.0.25}/README.md

@@ -10,12 +10,16 @@
 5. [Repo contents](#repo-contents)
 6. [Contact](#contact)
 7. [Gratuitous camera trap picture](#gratuitous-camera-trap-picture)
+8. [License](#license)
+9. [Contributing](#contributing)
 
 
 ## What's MegaDetector all about?
 
 [MegaDetector](megadetector.md) is an AI model that identifies animals, people, and vehicles in camera trap images (which also makes it useful for eliminating blank images).  This model is trained on several million images from a variety of ecosystems.
 
+MegaDetector only finds animals, it doesn't identify them to species level.  If you're looking for a species classifier, check out [SpeciesNet](https://github.com/google/cameratrapai), a species classifier that plays nicely with MegaDetector.
+
 Here's a “teaser” image of what MegaDetector output looks like:
 
 ![Red bounding box on fox](images/detector_example.jpg)<br/>Image credit University of Washington.
@@ -23,11 +27,11 @@ Here's a &ldquo;teaser&rdquo; image of what MegaDetector output looks like:
 
 ## How do I get started with MegaDetector?
 
-* If you are looking for a convenient tool to run MegaDetector, you don't need anything from this repository: check out [EcoAssist](https://github.com/PetervanLunteren/EcoAssist?tab=readme-ov-file).
+* If you are looking for a convenient tool to run MegaDetector, you don't need anything from this repository: check out [AddaxAI](https://addaxdatascience.com/addaxai/) (formerly EcoAssist), a GUI-based tool for running AI models (including MegaDetector) on camera trap images.
 * If you're just <i>considering</i> the use of AI in your workflow, and you aren't even sure yet whether MegaDetector would be useful to you, we recommend reading the "[getting started with MegaDetector](getting-started.md)" page.
 * If you're already familiar with MegaDetector and you're ready to run it on your data, see the [MegaDetector User Guide](megadetector.md) for instructions on running MegaDetector.
 * If you're a programmer-type looking to use tools from this repo, check out the [MegaDetector Python package](https://pypi.org/project/megadetector/) that provides access to everything in this repo (yes, you guessed it, "pip install megadetector").
-* If you have any questions, or you want to tell us that MegaDetector was amazing/terrible on your images, <a href="mailto:cameratraps@lila.science">email us</a>!
+* If you have any questions, or you want to tell us that MegaDetector was amazing/terrible on your images, or you have a zillion images and you want some help digging out of that backlog, <a href="mailto:cameratraps@lila.science">email us</a>!
 
 MegaDetector is just one of many tools that aims to make conservation biologists more efficient with AI.  If you want to learn about other ways to use AI to accelerate camera trap workflows, check out our of the field, affectionately titled &ldquo;[Everything I know about machine learning and camera traps](https://agentmorris.github.io/camera-trap-ml-survey/)&rdquo;.
 
@@ -45,9 +49,9 @@ Here are a few of the organizations that have used MegaDetector... we're only li
 * [Canadian Parks and Wilderness Society (CPAWS) Northern Alberta Chapter](https://cpawsnab.org/)
 * [Conservation X Labs](https://conservationxlabs.com/)
 * [Czech University of Life Sciences Prague](https://www.czu.cz/en)
-* [Dudek Camera Trap AI Image Toolkit (AIT)](https://dudek.com/services/wildlife-camera-trap-ai-image-processing-and-management/)
+* [Dudek Camera Trap AI Image Toolkit (AIT)](https://ait.dudek.com/)
 * [EcoLogic Consultants Ltd.](https://www.consult-ecologic.com/)
-* [Estación Biológica de Doñana](http://www.ebd.csic.es/inicio)
+* [Estación Biológica de Doñana](https://www.ebd.csic.es/)
 * [Indigenous Desert Alliance](https://www.indigenousdesertalliance.com/)
 * [Myall Lakes Dingo Project](https://carnivorecoexistence.info/myall-lakes-dingo-project/)
 * [Norwegian Institute for Nature Research](https://www.nina.no/english/Home)
@@ -84,7 +88,6 @@ Here are a few of the organizations that have used MegaDetector... we're only li
 * [Department of Fish and Wildlife Sciences](https://www.uidaho.edu/cnr/departments/fish-and-wildlife-sciences), University of Idaho
 * [Department of Society & Conservation](https://www.umt.edu/environment/about/departments/socon/), W.A. Franke College of Forestry & Conservation, University of Montana
 * [Department of Wildlife Ecology and Conservation](https://wec.ifas.ufl.edu/), University of Florida
-* [Ecology and Conservation of Amazonian Vertebrates Research Group](https://www.researchgate.net/lab/Fernanda-Michalski-Lab-4), Federal University of Amapá
 * [Gola Forest Programme](https://www.rspb.org.uk/our-work/conservation/projects/scientific-support-for-the-gola-forest-programme/), Royal Society for the Protection of Birds (RSPB)
 * [Graeme Shannon's Research Group](https://wildliferesearch.co.uk/group-1), Bangor University 
 * [Grizzly Bear Recovery Program](https://www.fws.gov/office/grizzly-bear-recovery-program), U.S. Fish & Wildlife Service
@@ -102,12 +105,13 @@ Here are a few of the organizations that have used MegaDetector... we're only li
 * [Santa Monica Mountains Recreation Area](https://www.nps.gov/samo/index.htm), National Park Service
 * [Seattle Urban Carnivore Project](https://www.zoo.org/seattlecarnivores), Woodland Park Zoo
 * [Serra dos Órgãos National Park](https://www.icmbio.gov.br/parnaserradosorgaos/), ICMBio
-* [Snapshot USA](https://emammal.si.edu/snapshot-usa), Smithsonian
+* [Snapshot USA](https://www.snapshot-usa.org/), Smithsonian
 * [TROPECOLNET project](https://www.anabenitezlopez.com/research/global-change-biology/tropecolnet/), Museo Nacional de Ciencias Naturales
 * [Wildlife Coexistence Lab](https://wildlife.forestry.ubc.ca/), University of British Columbia
 * [Wildlife Research](https://www.dfw.state.or.us/wildlife/research/index.asp), Oregon Department of Fish and Wildlife
 * [Wildlife Division](https://www.michigan.gov/dnr/about/contact/wildlife), Michigan Department of Natural Resources
 * [Kohl Wildlife Lab](https://kohlwildlifelab.com/), University of Georgia
+* Ecology and Conservation of Amazonian Vertebrates Research Group, Federal University of Amapá
 * Department of Ecology, TU Berlin
 * Ghost Cat Analytics
 * Protected Areas Unit, Canadian Wildlife Service
@@ -132,7 +136,7 @@ Here are a few of the organizations that have used MegaDetector... we're only li
 * [BirdLife Malta](https://birdlifemalta.org/) ([tweet](https://x.com/BirdLife_Malta/status/1817456839862173783?t=S-KRiZ5R1-CoW8-tbYNjqQ&s=03)) ([LI post](https://www.linkedin.com/posts/birdlifemalta_worldnatureconservationday-shearwater-colony-activity-7223220656589463553-X2Mc/?utm_source=share&utm_medium=member_android))
 * [Endangered Landscapes and Seascapes Programme](https://www.endangeredlandscapes.org/), Cambridge Conservation Initiative ([blog post](https://www.endangeredlandscapes.org/news/ai-for-wildlife-monitoring-a-real-time-alert-system-for-bears-and-wild-boars-in-romanias-carpathian-mountains/))
 
-* [Road Ecology Center](https://roadecology.ucdavis.edu/), University of California, Davis ([Wildlife Observer Network platform](https://wildlifeobserver.net/))
+* [Road Ecology Center](https://roadecology.ucdavis.edu/), University of California, Davis ([Wildlife Observer Network platform](https://roadecology.ucdavis.edu/research/projects/wildlife-observer-network))
 * [The Nature Conservancy in California](https://www.nature.org/en-us/about-us/where-we-work/united-states/california/) ([Animl platform](https://github.com/tnc-ca-geo/animl-frontend)) ([story](https://www.vision-systems.com/non-factory/environment-agriculture/article/14304433/the-nature-conservancy-brings-cameras-ai-to-invasive-species-prevention))
 * [San Diego Zoo Wildlife Alliance](https://science.sandiegozoo.org/)  ([Animl R package](https://github.com/conservationtechlab/animl))
 * [TerrOïko](https://www.terroiko.fr/) ([OCAPI platform](https://www.terroiko.fr/ocapi))
@@ -141,7 +145,7 @@ Also see:
 
 * The [list of MD-related GUIs, platforms, and GitHub repos](https://github.com/agentmorris/MegaDetector/blob/main/megadetector.md#is-there-a-gui) within the MegaDetector User Guide
 
-* [Peter's map of EcoAssist users](https://addaxdatascience.com/ecoassist) (who are also MegaDetector users!)
+* [Peter's map of AddaxAI (formerly EcoAssist) users](https://addaxdatascience.com/addaxai/) (who are also MegaDetector users!)
 
 * The list of papers tagged "MegaDetector" on our [list of papers about ML and camera traps](https://agentmorris.github.io/camera-trap-ml-survey/#camera-trap-ml-papers)
 
@@ -202,9 +206,9 @@ Code for hosting our models as an API, either for synchronous operation (i.e., f
 
 #### megadetector/classification
 
-Experimental code for training species classifiers on new data sets, generally trained on MegaDetector crops.  Currently the main pipeline described in this folder relies on a large database of labeled images that is not publicly available; therefore, this folder is not yet set up to facilitate training of your own classifiers.  However, it is useful for <i>users</i> of the classifiers that we train, and contains some useful starting points if you are going to take a "DIY" approach to training classifiers on cropped images.  
+This folder is largely deprecated thanks to the release of [SpeciesNet](https://github.com/google/cameratrapai), a species classifier that is better than any of the classifiers we ever trained with the stuff in this folder.  That said, this folder contains code for training species classifiers on new data sets, generally trained on MegaDetector crops.
 
-All that said, here's another "teaser image" of what you get at the end of training and running a classifier:
+Here's another "teaser image" of what you get at the end of training and running a classifier:
 
 <img src="images/warthog_classifications.jpg" width="700"><br/>Image credit University of Minnesota, from the Snapshot Safari program.
 
@@ -262,6 +266,6 @@ Code written on or before April 28, 2023 is [copyright Microsoft](https://github
 
 ## Contributing
 
-This project welcomes contributions, as pull requests, issues, or suggestions by [email](mailto:cameratraps@lila.science).  We have a [list](https://github.com/agentmorris/MegaDetector/issues/84) of issues that we're hoping to address, many of which would be good starting points for new contributors.  We also depend on other open-source tools that help users run MegaDetector (e.g. [EcoAssist](https://github.com/PetervanLunteren/EcoAssist) and [CamTrap Detector](https://github.com/bencevans/camtrap-detector)) and work with MegaDetector results (e.g. [Timelapse](https://github.com/saulgreenberg/Timelapse)); if you are looking to get involved in GUI development, reach out to the developers of those tools as well!
+This project welcomes contributions, as pull requests, issues, or suggestions by [email](mailto:cameratraps@lila.science).  We have a [list](https://github.com/agentmorris/MegaDetector/issues/84) of issues that we're hoping to address, many of which would be good starting points for new contributors.  We also depend on other open-source tools that help users run MegaDetector (particularly [AddaxAI](https://github.com/PetervanLunteren/AddaxAI) (formerly EcoAssist), and open-source tools that help users work with MegaDetector results (particularly [Timelapse](https://github.com/saulgreenberg/Timelapse)).  If you are looking to get involved in GUI development, reach out to the developers of those tools as well!
 
 If you are interesting in getting involved in the conservation technology space, and MegaDetector just happens to be the first page you landed on, and none of our open issues are getting you fired up, don't fret!  Head over to the [WILDLABS discussion forums](https://wildlabs.net/discussions) and let the community know you're a developer looking to get involved.  Someone needs your help!

{megadetector-5.0.23 → megadetector-5.0.25}/megadetector/api/synchronous/api_core/animal_detection_api/api_backend.py

@@ -14,9 +14,8 @@ import redis
 import argparse
 import PIL
 
-from io import BytesIO
-
-from detection.run_detector import load_detector, convert_to_tf_coords
+from detection.run_detector import load_detector
+from utils.ct_utils import convert_xywh_to_xyxy as convert_to_tf_coords
 import config
 import visualization.visualization_utils as vis_utils 
 

{megadetector-5.0.23 → megadetector-5.0.25}/megadetector/classification/merge_classification_detection_output.py

@@ -70,7 +70,7 @@ from typing import Any
 import pandas as pd
 from tqdm import tqdm
 
-from megadetector.utils.ct_utils import truncate_float
+from megadetector.utils.ct_utils import round_float
 
 
 #%% Example usage
@@ -124,7 +124,7 @@ def row_to_classification_list(row: Mapping[str, Any],
 
         # filter out confidences below the threshold, and set precision to 4
         result = [
-            (k, truncate_float(conf, precision=4))
+            (k, round_float(conf, precision=4))
             for k, conf in result if conf >= threshold
         ]
 

{megadetector-5.0.23 → megadetector-5.0.25}/megadetector/data_management/coco_to_labelme.py

@@ -18,6 +18,7 @@ from tqdm import tqdm
 from collections import defaultdict
 
 from megadetector.visualization.visualization_utils import open_image
+from megadetector.detection.run_detector import FAILURE_IMAGE_OPEN
 
 
 #%% Functions
@@ -145,7 +146,7 @@ def coco_to_labelme(coco_data,image_base,overwrite=False,bypass_image_size_check
                 except Exception:
                     print('Warning: cannot open image {}'.format(im_full_path))
                     if 'failure' not in im:
-                        im['failure'] = 'Failure image access'
+                        im['failure'] = FAILURE_IMAGE_OPEN
     
             # ...if we need to read w/h information
             

{megadetector-5.0.23 → megadetector-5.0.25}/megadetector/data_management/databases/integrity_check_json_db.py

@@ -86,7 +86,7 @@ def _check_image_existence_and_size(image,options=None):
         options (IntegrityCheckOptions): parameters impacting validation
     
     Returns:
-        bool: whether this image passes validation
+        str: None if this image passes validation, otherwise an error string
     """
 
     if options is None:        
@@ -96,23 +96,23 @@ def _check_image_existence_and_size(image,options=None):
     
     filePath = os.path.join(options.baseDir,image['file_name'])
     if not os.path.isfile(filePath):
-        # print('Image path {} does not exist'.format(filePath))
-        return False
+        s = 'Image path {} does not exist'.format(filePath)
+        return s
     
     if options.bCheckImageSizes:
         if not ('height' in image and 'width' in image):
-            print('Missing image size in {}'.format(filePath))
-            return False
+            s = 'Missing image size in {}'.format(filePath)
+            return s
 
         # width, height = Image.open(filePath).size
         pil_im = open_image(filePath)
         width,height = pil_im.size
         if (not (width == image['width'] and height == image['height'])):
-            print('Size mismatch for image {}: {} (reported {},{}, actual {},{})'.format(
-                    image['id'], filePath, image['width'], image['height'], width, height))
-            return False
+            s = 'Size mismatch for image {}: {} (reported {},{}, actual {},{})'.format(
+                    image['id'], filePath, image['width'], image['height'], width, height)
+            return s
         
-    return True
+    return None
 
   
 def integrity_check_json_db(jsonFile, options=None):
@@ -287,6 +287,7 @@ def integrity_check_json_db(jsonFile, options=None):
             if fn_relative not in image_paths_in_json:
                 unused_files.append(fn_relative)
                 
+    # List of (filename,error_string) tuples
     validation_errors = []
     
     # If we're checking image existence but not image size, we don't need to read the images
@@ -298,8 +299,8 @@ def integrity_check_json_db(jsonFile, options=None):
         image_paths_relative_set = set(image_paths_relative)
         
         for im in images:
-            if im['file_name'] not in image_paths_relative_set:
-                validation_errors.append(im['file_name'])
+            if im['file_name'] not in image_paths_relative_set:                
+                validation_errors.append((im['file_name'],'not found in relative path list'))
             
     # If we're checking image size, we need to read the images
     if options.bCheckImageSizes:
@@ -321,12 +322,12 @@ def integrity_check_json_db(jsonFile, options=None):
             results = tqdm(pool.imap(_check_image_existence_and_size, images), total=len(images))
         else:
             results = []
-            for im in tqdm(images):
+            for im in tqdm(images):                
                 results.append(_check_image_existence_and_size(im,options))
                 
         for i_image,result in enumerate(results):
-            if result is not None:            
-                validation_errors.append(images[i_image]['file_name'])
+            if result is not None:
+                validation_errors.append(images[i_image]['file_name'],result)
                             
     # ...for each image
     

{megadetector-5.0.23 → megadetector-5.0.25}/megadetector/data_management/databases/subset_json_db.py

@@ -12,16 +12,18 @@ subset_json_detector_output.py.
     
 #%% Constants and imports
 
+import os
 import sys
 import json
 import argparse
 
 from tqdm import tqdm
+from copy import copy
 
 
 #%% Functions
 
-def subset_json_db(input_json, query, output_json=None, ignore_case=False):
+def subset_json_db(input_json, query, output_json=None, ignore_case=False, verbose=False):
     """
     Given a json file (or dictionary already loaded from a json file), produce a new 
     database containing only the images whose filenames contain the string 'query', 
@@ -29,54 +31,80 @@ def subset_json_db(input_json, query, output_json=None, ignore_case=False):
     
     Args:
         input_json (str): COCO Camera Traps .json file to load, or an already-loaded dict
-        query (str): string to query for, only include images in the output whose filenames 
-            contain this string.
+        query (str or list): string to query for, only include images in the output whose filenames 
+            contain this string.  If this is a list, test for exact matches.
         output_json (str, optional): file to write the resulting .json file to
         ignore_case (bool, optional): whether to perform a case-insensitive search for [query]
+        verbose (bool, optional): enable additional debug output
         
     Returns:
-        dict: possibly-modified CCT dictionary
+        dict: CCT dictionary containing a subset of the images and annotations in the input dict
     """
-    
-    if ignore_case:
-        query = query.lower()
         
     # Load the input file if necessary
     if isinstance(input_json,str):
         print('Loading input .json...')
         with open(input_json, 'r') as f:
-            data = json.load(f)
+            input_data = json.load(f)
     else:
-        data = input_json
+        input_data = input_json
 
     # Find images matching the query
     images = []
-    image_ids = set()
     
-    for im in tqdm(data['images']):
-        fn = im['file_name']
+    if isinstance(query,str):
+        
+        if ignore_case:
+            query = query.lower()
+        
+        for im in tqdm(input_data['images']):
+            fn = im['file_name']
+            if ignore_case:
+                fn = fn.lower()
+            if query in fn:
+                images.append(im)
+                
+    else:
+        
+        query = set(query)
+        
         if ignore_case:
-            fn = fn.lower()
-        if query in fn:
-            images.append(im)
-            image_ids.add(im['id'])        
+            query = set([s.lower() for s in query])
+            
+        for im in input_data['images']:
+            fn = im['file_name']
+            if ignore_case:
+                fn = fn.lower()
+            if fn in query:
+                images.append(im)
+
+    image_ids = set([im['id'] for im in images])
     
     # Find annotations referring to those images
     annotations = []
     
-    for ann in tqdm(data['annotations']):
+    for ann in input_data['annotations']:
         if ann['image_id'] in image_ids:
             annotations.append(ann)
     
-    output_data = data
+    output_data = copy(input_data)
     output_data['images'] = images
     output_data['annotations'] = annotations
     
     # Write the output file if requested
     if output_json is not None:
-        print('Writing output .json...')
-        json.dump(output_data,open(output_json,'w'),indent=1)
-        
+        if verbose:
+            print('Writing output .json to {}'.format(output_json))
+        output_dir = os.path.dirname(output_json)
+        os.makedirs(output_dir,exist_ok=True)
+        with open(output_json,'w') as f:
+            json.dump(output_data,f,indent=1)
+    
+    if verbose:
+        print('Keeping {} of {} images, {} of {} annotations'.format(
+            len(output_data['images']),len(input_data['images']),
+            len(output_data['annotations']),len(input_data['annotations'])))
+
     return output_data
 
 

megadetector-5.0.25/megadetector/data_management/lila/add_locations_to_island_camera_traps.py

@@ -0,0 +1,101 @@
+"""
+
+add_locations_to_island_camera_traps.py
+
+The Island Conservation Camera Traps dataset had unique camera identifiers embedded
+in filenames, but not in the proper metadata fields.  This script copies that information
+to metadata.
+
+"""
+
+#%% Imports and constants
+
+import os
+import json
+from tqdm import tqdm
+
+input_fn = os.path.expanduser('~/lila/metadata/island_conservation.json')
+output_fn = os.path.expanduser('~/tmp/island_conservation.json')
+preview_folder = os.path.expanduser('~/tmp/island_conservation_preview')
+image_directory = os.path.expanduser('~/data/icct/public/')
+
+
+#%% Prevent imports during testing
+
+if False:
+
+    #%% Read input file
+    
+    with open(input_fn,'r') as f:
+        d = json.load(f)
+        
+    d['info']
+    d['info']['version'] = '1.01'
+    
+    
+    #%% Find locations
+    
+    images = d['images']
+    
+    locations = set()
+    
+    for i_image,im in tqdm(enumerate(images),total=len(images)):
+        tokens_fn = im['file_name'].split('/')
+        tokens_id = im['id'].split('_')
+        assert tokens_fn[0] == tokens_id[0]
+        assert tokens_fn[1] == tokens_id[1]
+        location = tokens_fn[0] + '_' + tokens_fn[1]
+        im['location'] = location
+        locations.add(location)
+    
+    locations = sorted(list(locations))
+        
+    for s in locations:
+        print(s)
+        
+        
+    #%% Write output file
+    
+    with open(output_fn,'w') as f:
+        json.dump(d,f,indent=1)
+        
+    
+    #%% Validate .json files
+    
+    from megadetector.data_management.databases import integrity_check_json_db
+    
+    options = integrity_check_json_db.IntegrityCheckOptions()
+    options.baseDir = image_directory
+    options.bCheckImageSizes = False
+    options.bCheckImageExistence = True
+    options.bFindUnusedImages = True
+    
+    sorted_categories, data, error_info = integrity_check_json_db.integrity_check_json_db(output_fn, options)
+    
+    
+    #%% Preview labels
+    
+    from megadetector.visualization import visualize_db
+    
+    viz_options = visualize_db.DbVizOptions()
+    viz_options.num_to_visualize = 2000
+    viz_options.trim_to_images_with_bboxes = False
+    viz_options.add_search_links = False
+    viz_options.sort_by_filename = False
+    viz_options.parallelize_rendering = True
+    viz_options.classes_to_exclude = ['test']
+    html_output_file, image_db = visualize_db.visualize_db(db_path=output_fn,
+                                                             output_dir=preview_folder,
+                                                             image_base_dir=image_directory,
+                                                             options=viz_options)
+    
+    from megadetector.utils import path_utils
+    path_utils.open_file(html_output_file)
+    
+    
+    #%% Zip output file
+    
+    from megadetector.utils.path_utils import zip_file
+    
+    zip_file(output_fn, verbose=True)
+    assert os.path.isfile(output_fn + '.zip')