megadetector 5.0.12__tar.gz → 5.0.14__tar.gz

{megadetector-5.0.12/megadetector.egg-info → megadetector-5.0.14}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: megadetector
-Version: 5.0.12
+Version: 5.0.14
 Summary: MegaDetector is an AI model that helps conservation folks spend less time doing boring things with camera trap images.
 Author-email: Your friendly neighborhood MegaDetector team <cameratraps@lila.science>
 Maintainer-email: Your friendly neighborhood MegaDetector team <cameratraps@lila.science>
@@ -114,7 +114,7 @@ print('Found {} detections above threshold'.format(len(detections_above_threshol
 
 ```
 from megadetector.detection.run_detector_batch import \
-    load_and_run_detector_batch,write_results_to_file
+    load_and_run_detector_batch, write_results_to_file
 from megadetector.utils import path_utils
 import os
 

{megadetector-5.0.12 → megadetector-5.0.14}/README-package.md

@@ -62,7 +62,7 @@ print('Found {} detections above threshold'.format(len(detections_above_threshol
 
 ```
 from megadetector.detection.run_detector_batch import \
-    load_and_run_detector_batch,write_results_to_file
+    load_and_run_detector_batch, write_results_to_file
 from megadetector.utils import path_utils
 import os
 

{megadetector-5.0.12 → megadetector-5.0.14}/README.md

@@ -11,6 +11,7 @@
 6. [Contact](#contact)
 7. [Gratuitous camera trap picture](#gratuitous-camera-trap-picture)
 
+
 ## What's MegaDetector all about?
 
 [MegaDetector](megadetector.md) is an AI model that identifies animals, people, and vehicles in camera trap images (which also makes it useful for eliminating blank images).  This model is trained on several million images from a variety of ecosystems.
@@ -25,7 +26,7 @@ Here's a “teaser” image of what MegaDetector output looks like:
 * If you are looking for a convenient tool to run MegaDetector, you don't need anything from this repository: check out [EcoAssist](https://github.com/PetervanLunteren/EcoAssist?tab=readme-ov-file).
 * If you're just <i>considering</i> the use of AI in your workflow, and you aren't even sure yet whether MegaDetector would be useful to you, we recommend reading the "[getting started with MegaDetector](getting-started.md)" page.
 * If you're already familiar with MegaDetector and you're ready to run it on your data, see the [MegaDetector User Guide](megadetector.md) for instructions on running MegaDetector.
-* If you're a programmer-type looking to use tools from this repo, check out the [Python package](https://pypi.org/project/megadetector/) that provides access to everything in this repo (yes, you guessed it, "pip install megadetector").
+* If you're a programmer-type looking to use tools from this repo, check out the [MegaDetector Python package](https://pypi.org/project/megadetector/) that provides access to everything in this repo (yes, you guessed it, "pip install megadetector").
 * If you have any questions, or you want to tell us that MegaDetector was amazing/terrible on your images, <a href="mailto:cameratraps@lila.science">email us</a>!
 
 MegaDetector is just one of many tools that aims to make conservation biologists more efficient with AI.  If you want to learn about other ways to use AI to accelerate camera trap workflows, check out our of the field, affectionately titled &ldquo;[Everything I know about machine learning and camera traps](https://agentmorris.github.io/camera-trap-ml-survey/)&rdquo;.
@@ -65,12 +66,14 @@ Here are a few of the organizations that have used MegaDetector... we're only li
 * [Parc national du Mont-Tremblant](https://www.sepaq.com/pq/mot/index.dot?language_id=1)
 * [The Land Banking Group](https://thelandbankinggroup.com/)
 * [New Zealand Department of Conservation](https://www.doc.govt.nz)
+* [Habitat NZ](https://habitatnz.co.nz/)
 
 * [Applied Conservation Macro Ecology Lab](http://www.acmelab.ca/), University of Victoria
 * [Banff National Park Resource Conservation](https://www.pc.gc.ca/en/pn-np/ab/banff/nature/conservation), Parks Canada
 * [Blumstein Lab](https://blumsteinlab.eeb.ucla.edu/), UCLA
 * [Borderlands Research Institute](https://bri.sulross.edu/), Sul Ross State University
 * [Capitol Reef National Park](https://www.nps.gov/care/index.htm) / Utah Valley University
+* [Canyon Critters Project](https://www.zooniverse.org/projects/arw36/canyon-critters), University of Georgia
 * [Center for Biodiversity and Conservation](https://www.amnh.org/research/center-for-biodiversity-conservation), American Museum of Natural History
 * [Centre for Ecosystem Science](https://www.unsw.edu.au/research/), UNSW Sydney
 * [Cross-Cultural Ecology Lab](https://crossculturalecology.net/), Macquarie University
@@ -79,7 +82,7 @@ Here are a few of the organizations that have used MegaDetector... we're only li
 * [Department of Society & Conservation](https://www.umt.edu/environment/about/departments/socon/), W.A. Franke College of Forestry & Conservation, University of Montana
 * [Department of Wildlife Ecology and Conservation](https://wec.ifas.ufl.edu/), University of Florida
 * [Ecology and Conservation of Amazonian Vertebrates Research Group](https://www.researchgate.net/lab/Fernanda-Michalski-Lab-4), Federal University of Amapá
-* [Gola Forest Programma](https://www.rspb.org.uk/our-work/conservation/projects/scientific-support-for-the-gola-forest-programme/), Royal Society for the Protection of Birds (RSPB)
+* [Gola Forest Programme](https://www.rspb.org.uk/our-work/conservation/projects/scientific-support-for-the-gola-forest-programme/), Royal Society for the Protection of Birds (RSPB)
 * [Graeme Shannon's Research Group](https://wildliferesearch.co.uk/group-1), Bangor University 
 * [Grizzly Bear Recovery Program](https://www.fws.gov/office/grizzly-bear-recovery-program), U.S. Fish & Wildlife Service
 * [Hamaarag](https://hamaarag.org.il/), The Steinhardt Museum of Natural History, Tel Aviv University
@@ -138,49 +141,59 @@ Also see:
 
 ## Repo contents
 
-This repo is organized into the following folders...
+### Repo history
+
+MegaDetector was initially developed by the [Microsoft AI for Earth program](https://www.microsoft.com/en-us/ai/ai-for-earth); this repo was forked from the microsoft/cameratraps repo and is maintained by the original MegaDetector developers (who are no longer at Microsoft, but are absolutely fantastically eternally grateful to Microsoft for the investment and commitment that made MegaDetector happen).  If you're interested in MD's history, see the [downloading the model](https://github.com/agentmorris/MegaDetector/blob/main/megadetector.md#downloading-the-model) section in the MegaDetector User Guide to learn about the history of MegaDetector releases, and the [can you share the training data?](https://github.com/agentmorris/MegaDetector/blob/main/megadetector.md#can-you-share-the-training-data) section to learn about the training data used in each of those releases.
 
-This repo contains the tools for training and running [MegaDetector](megadetector.md), an object detection model that does a pretty good job finding animals, people, and vehicles (and therefore is pretty good at finding empty images) in camera trap images in a variety of terrestrial ecosystems.
+### What the code in this repo does
 
-The core functionality provided is:
+The core functionality provided in this repo is:
 
-- Training and running [MegaDetector](megadetector.md).
-- Training and running some [species classifiers](classification) that are used in conjunction with MegaDetector.
-- Tools to [convert](data_management) frequently-used camera trap metadata formats into a common format.
-- A [batch processing API](https://github.com/agentmorris/MegaDetector/tree/main/megadetector/api/batch_processing) that runs MegaDetector on large image collections, to accelerate population surveys.
-- A [real-time API](https://github.com/agentmorris/MegaDetector/tree/main/megadetector/api/synchronous) that runs MegaDetector (and some species classifiers) synchronously, primarily to support biosecurity applications.
+- Tools for training and running [MegaDetector](megadetector.md).
+- Tools for working with MegaDetector output, e.g. for reviewing the results of a large processing batch.
+- Tools to convert among frequently-used camera trap metadata formats.
 
 This repo does not host the data used to train MegaDetector, but we work with our collaborators to make data and annotations available whenever possible on [lila.science](http://lila.science).  See the [MegaDetector training data](megadetector.md#can-you-share-the-training-data) section to learn more about the data used to train MegaDetector.
 
-MegaDetector was initially developed by the [Microsoft AI for Earth program](https://www.microsoft.com/en-us/ai/ai-for-earth); this repo was forked from the microsoft/cameratraps repo and is maintained by the original MegaDetector developers (who are no longer at Microsoft, but are absolutely fantastically eternally grateful to Microsoft for the investment and commitment that made MegaDetector happen).
+### Repo organization
 
+This repo is organized into the following folders...
 
-### megadetector/detection
+#### megadetector/detection
 
 Code for running models, especially MegaDetector.
 
 
-### megadetector/postprocessing
+##### megadetector/postprocessing
 
 Code for common operations one might do after running MegaDetector, e.g. [generating preview pages to summarize your results](https://github.com/agentmorris/MegaDetector/blob/main/megadetector/postprocessing/postprocess_batch_results.py), [separating images into different folders based on AI results](https://github.com/agentmorris/MegaDetector/blob/main/megadetector/postprocessing/separate_detections_into_folders.py), or [converting results to a different format](https://github.com/agentmorris/MegaDetector/blob/main/megadetector/postprocessing/convert_output_format.py).
 
 
-### megadetector/utils
+#### megadetector/utils
+
+Small utility functions for string manipulation, filename manipulation, downloading files from URLs, etc.
 
-Small utility functions for string manipulation, filename manipulation, downloading files from URLs, etc.  Mostly adapted from the [ai4eutils](https://github.com/microsoft/ai4eutils) repo.
 
+#### megadetector/visualization
 
-### megadetector/visualization
+Tools for visualizing images with ground truth and/or predicted bounding boxes.
 
-Tools for visualizing images with ground truth and/or predicted annotations.
+
+#### megadetector/data_management
+
+Code for:
+
+* Converting frequently-used metadata formats to [COCO Camera Traps](https://github.com/agentmorris/MegaDetector/blob/main/megadetector/data_management/README.md#coco-cameratraps-format) format
+* Converting the output of AI models (especially YOLOv5) to the format used for AI results throughout this repo
+* Creating, visualizing, and  editing COCO Camera Traps .json databases
 
 
-### megadetector/api
+#### megadetector/api
 
 Code for hosting our models as an API, either for synchronous operation (i.e., for real-time inference) or as a batch process (for large biodiversity surveys).
 
 
-### megadetector/classification
+#### megadetector/classification
 
 Experimental code for training species classifiers on new data sets, generally trained on MegaDetector crops.  Currently the main pipeline described in this folder relies on a large database of labeled images that is not publicly available; therefore, this folder is not yet set up to facilitate training of your own classifiers.  However, it is useful for <i>users</i> of the classifiers that we train, and contains some useful starting points if you are going to take a "DIY" approach to training classifiers on cropped images.  
 
@@ -189,41 +202,32 @@ All that said, here's another "teaser image" of what you get at the end of train
 <img src="images/warthog_classifications.jpg" width="700"><br/>Image credit University of Minnesota, from the Snapshot Safari program.
 
 
-### megadetector/taxonomy_mapping
+#### megadetector/taxonomy_mapping
 
 Code to facilitate mapping data-set-specific category names (e.g. "lion", which means very different things in Idaho vs. South Africa) to a standard taxonomy.
 
 
-### megadetector/data_management
-
-Code for:
-
-* Converting frequently-used metadata formats to [COCO Camera Traps](https://github.com/agentmorris/MegaDetector/blob/main/megadetector/data_management/README.md#coco-cameratraps-format) format
-* Converting the output of AI models (especially YOLOv5) to the format used for AI results throughout this repo
-* Creating, visualizing, and  editing COCO Camera Traps .json databases
-
-
-### envs
+#### envs
 
 Environment files... specifically .yml files for mamba/conda environments (these are what we recommend in our [MegaDetector User Guide](megadetector.md)), and a requirements.txt for the pip-inclined.
 
 
-### images
+#### images
 
 Media used in documentation.
 
 
-### archive
+#### archive
 
 Old code that we didn't <i>quite</i> want to delete, but is basically obsolete.
 
 
-### sandbox
+#### sandbox
 
 Random things that don't fit in any other directory, but aren't quite deprecated.  Mostly postprocessing scripts that were built for a single use case but could potentially be useful in the future.
 
 
-### test_images
+#### test_images
 
 A handful of images from [LILA](https://lila.science) that facilitate testing and debugging.
 

{megadetector-5.0.12 → megadetector-5.0.14}/megadetector/api/batch_processing/api_core/server.py

@@ -105,7 +105,7 @@ def request_detections():
     model_version = post_body.get('model_version', '')
     if model_version != '':
         model_version = str(model_version)  # in case user used an int
-        if model_version not in api_config.MD_VERSIONS_TO_REL_PATH:  # TODO use AppConfig to store model version info
+        if model_version not in api_config.MD_VERSIONS_TO_REL_PATH:
             return make_error(400, f'model_version {model_version} is not supported.')
 
     # check request_name has only allowed characters

{megadetector-5.0.12 → megadetector-5.0.14}/megadetector/api/batch_processing/api_core/server_api_config.py

@@ -47,7 +47,6 @@ MAX_BATCH_ACCOUNT_ACTIVE_JOBS = 300
 DETECTION_CONF_THRESHOLD = 0.1
 
 # relative to the `megadetector_copies` folder in the container `models`
-# TODO add MD versions info to AppConfig
 MD_VERSIONS_TO_REL_PATH = {
     '4.1': 'megadetector_v4_1/md_v4.1.0.pb',
     '3': 'megadetector_v3/megadetector_v3_tf19.pb',

{megadetector-5.0.12 → megadetector-5.0.14}/megadetector/api/batch_processing/api_core/server_job_status_table.py

@@ -67,9 +67,6 @@ class JobStatusTable:
         assert 'request_status' in status and 'message' in status
         assert status['request_status'] in JobStatusTable.allowed_statuses
 
-        # TODO do not read the entry first to get the call_params when the Cosmos SDK add a
-        # patching functionality:
-        # https://feedback.azure.com/forums/263030-azure-cosmos-db/suggestions/6693091-be-able-to-do-partial-updates-on-document
         item_old = self.read_job_status(job_id)
         if item_old is None:
             raise ValueError

{megadetector-5.0.12 → megadetector-5.0.14}/megadetector/api/batch_processing/api_core/server_utils.py

@@ -19,7 +19,6 @@ log = logging.getLogger(os.environ['FLASK_APP'])
 #%% helper classes and functions
 
 def make_error(error_code: int, error_message: str) -> Tuple[dict, int]:
-    # TODO log exception when we have more telemetry
     log.error(f'Error {error_code} - {error_message}')
     return {'error': error_message}, error_code
 
@@ -28,8 +27,6 @@ def check_data_container_sas(input_container_sas: str) -> Optional[Tuple[int, st
     """
     Returns a tuple (error_code, msg) if not a usable SAS URL, else returns None
     """
-    # TODO check that the expiry date of input_container_sas is at least a month
-    # into the future
     permissions = sas_blob_utils.get_permissions_from_uri(input_container_sas)
     data = sas_blob_utils.get_all_query_parts(input_container_sas)
 
@@ -38,7 +35,6 @@ def check_data_container_sas(input_container_sas: str) -> Optional[Tuple[int, st
     if 'read' not in permissions or 'list' not in permissions:
         if 'si' in data:
             # if no permission specified explicitly but has an access policy, assumes okay
-            # TODO - check based on access policy as well
             return None
 
         return 400, msg

{megadetector-5.0.12 → megadetector-5.0.14}/megadetector/api/batch_processing/integration/eMammal/test_scripts/push_annotations_to_emammal.py

@@ -72,7 +72,6 @@ def main():
    
     print(deployment_id)
 
-    # TODO: check project ID ?
     sql = ''' SELECT emammal_project_taxa_id FROM wild_id.emammal_project_taxa
             where species in ("No Animal", "Unknown Animal", "Homo sapiens", "Vehicle") '''
 

{megadetector-5.0.12 → megadetector-5.0.14}/megadetector/api/synchronous/api_core/animal_detection_api/api_frontend.py

@@ -147,8 +147,6 @@ def detect_sync():
         
         try:
             # Write images to temporary files
-            #
-            # TODO: read from memory rather than using intermediate files
             os.makedirs(temp_direc,exist_ok=True)
             for name, file in request.files.items():
                 if file.content_type in config.IMAGE_CONTENT_TYPES:
@@ -166,7 +164,6 @@ def detect_sync():
         
         while True:
             
-            # TODO: convert to a blocking read and eliminate the sleep() statement in this loop
             result = db.get(redis_id)
             
             if result:

{megadetector-5.0.12 → megadetector-5.0.14}/megadetector/classification/efficientnet/utils.py

@@ -90,9 +90,6 @@ def round_filters(filters, global_params):
     multiplier = global_params.width_coefficient
     if not multiplier:
         return filters
-    # TODO: modify the params names.
-    #       maybe the names (width_divisor,min_width)
-    #       are more suitable than (depth_divisor,min_depth).
     divisor = global_params.depth_divisor
     min_depth = global_params.min_depth
     filters *= multiplier

{megadetector-5.0.12 → megadetector-5.0.14}/megadetector/data_management/camtrap_dp_to_coco.py

@@ -235,5 +235,3 @@ if False:
 #%% Command-line driver
 
 # TODO
-
-    

{megadetector-5.0.12 → megadetector-5.0.14}/megadetector/data_management/cct_json_utils.py

@@ -295,8 +295,9 @@ class SequenceOptions:
     Options parameterizing the grouping of images into sequences by time.
     """
     
-    #: Images separated by <= this duration will be grouped into the same sequence.
-    episode_interval_seconds = 60.0
+    def __init__(self):
+        #: Images separated by <= this duration will be grouped into the same sequence.
+        self.episode_interval_seconds = 60.0
 
     
 #%% Functions
@@ -305,16 +306,24 @@ def create_sequences(image_info,options=None):
     """
     Synthesizes episodes/sequences/bursts for the images in [image_info].
     
-    Modifies [image_info], populating the 'seq_id', 'seq_num_frames', and 'frame_num' fields
-    for each image.
+    Modifies [image_info] in place, populating the 'seq_id', 'seq_num_frames', and 'frame_num' 
+    fields for each image.
     
     Args:
-        image_info (dict): a list of dicts in CCT format, i.e. with fields 'file_name' (str),
-            'datetime' (datetime), and 'location' (str).    
+        image_info (str, dict, or list): a dict in CCT format, a CCT .json file, or just the 'images' component
+            of a CCT dataset (a list of dicts with  fields 'file_name' (str), 'datetime' (datetime), and
+            'location' (str)).
     """
     
     if options is None:
         options = SequenceOptions()
+    
+    if isinstance(image_info,str):
+        with open(image_info,'r') as f:
+            image_info = json.load(f)
+        
+    if isinstance(image_info,dict):
+        image_info = image_info['images']
         
     # Find all unique locations
     locations = set()

{megadetector-5.0.12 → megadetector-5.0.14}/megadetector/data_management/coco_to_labelme.py

@@ -95,7 +95,18 @@ def get_labelme_dict_for_image_from_coco_record(im,annotations,categories,info=N
 def coco_to_labelme(coco_data,image_base,overwrite=False,bypass_image_size_check=False,verbose=False):
     """
     For all the images in [coco_data] (a dict or a filename), write a .json file in 
-    labelme format alongside the corresponding relative path within image_base.    
+    labelme format alongside the corresponding relative path within image_base.
+    
+    Args:
+        coco_data (dict or str): path to a COCO-formatted .json file, or an already-loaded
+            COCO-formatted dict
+        image_base (str): path where images live (filenames in [coco_data] should be relative to
+            [image_base]); this is also where labelme files will be written
+        overwrite (bool, optional): overwrite existing .json files
+        bypass_image_size_check (bool, optional): if you're sure that the COCO data already has
+            correct 'width' and 'height' fields, this bypasses the somewhat-slow loading of 
+            each image to fetch image sizes
+        verbose (bool, optional): enable additional debug output
     """
     
     # Load COCO data if necessary

{megadetector-5.0.12 → megadetector-5.0.14}/megadetector/data_management/databases/integrity_check_json_db.py

@@ -37,30 +37,34 @@ class IntegrityCheckOptions:
     Options for integrity_check_json_db()
     """
     
-    #: Image path; the filenames in the .json file should be relative to this folder
-    baseDir = ''
-    
-    #: Should we validate the image sizes?
-    bCheckImageSizes = False
-    
-    #: Should we check that all the images in the .json file exist on disk?
-    bCheckImageExistence = False
-    
-    #: Should we search [baseDir] for images that are not used in the .json file?
-    bFindUnusedImages = False
-    
-    #: Should we require that all images in the .json file have a 'location' field?
-    bRequireLocation = True
-    
-    #: For debugging, limit the number of images we'll process
-    iMaxNumImages = -1
-    
-    #: Number of threads to use for parallelization, set to <= 1 to disable parallelization
-    nThreads = 10
-    
-    #: Enable additional debug output
-    verbose = True
-    
+    def __init__(self):
+        
+        #: Image path; the filenames in the .json file should be relative to this folder
+        self.baseDir = ''
+        
+        #: Should we validate the image sizes?
+        self.bCheckImageSizes = False
+        
+        #: Should we check that all the images in the .json file exist on disk?
+        self.bCheckImageExistence = False
+        
+        #: Should we search [baseDir] for images that are not used in the .json file?
+        self.bFindUnusedImages = False
+        
+        #: Should we require that all images in the .json file have a 'location' field?
+        self.bRequireLocation = True
+        
+        #: For debugging, limit the number of images we'll process
+        self.iMaxNumImages = -1
+        
+        #: Number of threads to use for parallelization, set to <= 1 to disable parallelization
+        self.nThreads = 10
+        
+        #: Enable additional debug output
+        self.verbose = True
+        
+        #: Allow integer-valued image and annotation IDs (COCO uses this, CCT files use strings)
+        self.allowIntIDs = False
     
 # This is used in a medium-hacky way to share modified options across threads
 defaultOptions = IntegrityCheckOptions()
@@ -231,7 +235,12 @@ def integrity_check_json_db(jsonFile, options=None):
         imagePathsInJson.add(image['file_name'])
         
         assert isinstance(image['file_name'],str), 'Illegal image filename type'
-        assert isinstance(image['id'],str), 'Illegal image ID type'
+        
+        if options.allowIntIDs:
+            assert isinstance(image['id'],str) or isinstance(image['id'],int), \
+                'Illegal image ID type'
+        else:
+            assert isinstance(image['id'],str), 'Illegal image ID type'
         
         imageId = image['id']        
         
@@ -329,9 +338,16 @@ def integrity_check_json_db(jsonFile, options=None):
         assert 'id' in ann
         assert 'category_id' in ann
         
-        assert isinstance(ann['id'],str), 'Illegal annotation ID type'
+        if options.allowIntIDs:
+            assert isinstance(ann['id'],str) or isinstance(ann['id'],int), \
+                'Illegal annotation ID type'
+            assert isinstance(ann['image_id'],str) or isinstance(ann['image_id'],int), \
+                'Illegal annotation image ID type'
+        else:
+            assert isinstance(ann['id'],str), 'Illegal annotation ID type'
+            assert isinstance(ann['image_id'],str), 'Illegal annotation image ID type'
+        
         assert isinstance(ann['category_id'],int), 'Illegal annotation category ID type'
-        assert isinstance(ann['image_id'],str), 'Illegal annotation image ID type'
         
         if 'bbox' in ann:
             nBoxes += 1