mbari-aidata 1.41.9__tar.gz

mbari_aidata-1.41.9/PKG-INFO

@@ -0,0 +1,151 @@
+Metadata-Version: 2.3
+Name: mbari-aidata
+Version: 1.41.9
+Summary: Command line tool to do extract, transform, load and download operations on AI data for a number of projects at MBARI that require detection, clustering or classification workflows.
+License: Apache
+Author: Danelle Cline
+Author-email: dcline@mbari.org
+Requires-Python: >=3.10,<3.12
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Requires-Dist: albucore (==0.0.16)
+Requires-Dist: albumentations (==1.4.15)
+Requires-Dist: click (==8.1.7)
+Requires-Dist: ephem (==4.1.6)
+Requires-Dist: moviepy (==2.1.1)
+Requires-Dist: opencv-contrib-python (==4.11.0.86)
+Requires-Dist: pandas (==2.2.2)
+Requires-Dist: pascal-voc-writer (>=0.1.4)
+Requires-Dist: piexif (==1.1.3)
+Requires-Dist: pytz (==2024.2)
+Requires-Dist: redis (==5.0.7)
+Requires-Dist: requests (==2.32.3)
+Requires-Dist: tator (==1.2.3)
+Requires-Dist: torchvision (==0.18.1)
+Requires-Dist: tqdm (==4.67.1)
+Requires-Dist: transformers (==4.48.0)
+Description-Content-Type: text/markdown
+
+[![MBARI](https://www.mbari.org/wp-content/uploads/2014/11/logo-mbari-3b.png)](http://www.mbari.org)
+[![semantic-release](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg)](https://github.com/semantic-release/semantic-release)
+[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![Python](https://img.shields.io/badge/language-Python-blue.svg)](https://www.python.org/downloads/)
+
+*mbari-aidata* is a command line tool to do extract, transform, load and download operations
+on AI data for a number of projects at MBARI that require detection, clustering or classification
+workflows.
+
+Full documentation is available on commands at [https://docs.mbari.org/internal/ai/data](https://docs.mbari.org/internal/ai/data/).
+ 
+This supports loading [sdcat](https://github.com/mbari-org/sdcat) formatted output and downloads from [Tator](https://www.tatorapp.com/) and 
+[Redis](https://redis.io) databases, although support for other data sources is also possible, e.g. [FathomNet](https://fathomnet.org/).
+so we decided to keep the name generic.
+
+This also supports loading media from a directory or URL, and transforming data into various 
+formats for machine learning, e.g. COCO, CIFAR, or PASCAL VOC format.
+
+## Requirements
+- Python 3.10 or higher
+- A Tator API token and Redis password for the .env file. Contact the MBARI AI team for access.
+- Docker for development and testing only
+
+## Installation 
+Install from PyPi
+
+```shell
+pip install mbari-aidata
+```
+ 
+Create the .env file with the following contents in the root directory of the project:
+```shell
+TATOR_TOKEN=your_api_token
+REDIS_PASSWORD=your_redis_password
+ENVIRONMENT=testing or production
+```
+
+Create a configuration file in the root directory of the project:
+```shell
+touch config_cfe.yaml
+```
+
+This file will be used to configure the project data, such as mounts, plugins, and database connections.
+```shell
+aidata download --version Baseline --labels "Diatoms, Copepods" --config config_cfe.yml
+```
+
+Example configuration file:
+```yaml
+# config_cfe.yml
+# Config file for CFE project production
+mounts:
+  - name: "image"
+    path: "/mnt/CFElab"
+    host: "mantis.shore.mbari.org"
+    nginx_root: "/CFElab"
+
+  - name: "video"
+    path: "/mnt/CFElab"
+    host: "mantis.shore.mbari.org"
+    nginx_root: "/CFElab"
+
+
+plugins:
+  - name: "extractor"
+    module: "mbari_aidata.plugins.extractors.tap_cfe_media"
+    function: "extract_media"
+
+redis:
+  host: "doris.shore.mbari.org"
+  port: 6382
+
+vss:
+  project: "902111-CFE"
+  model: "google/vit-base-patch16-224"
+
+tator:
+  project: "902111-CFE"
+  host: "mantis.shore.mbari.org"
+  image:
+    attributes:
+      iso_datetime:
+        type: datetime
+      depth:
+        type: float
+  video:
+    attributes:
+      iso_start_datetime:
+        type: datetime
+  box:
+    attributes:
+      Label:
+        type: string
+      score:
+        type: float
+      cluster:
+        type: string
+      saliency:
+        type: float
+      area:
+        type: int
+      exemplar:
+        type: bool
+```
+
+A docker version is also available at `mbari/aidata:latest` or `mbari/aidata:latest:cuda-124`.
+
+## Commands
+
+* `aidata download --help` -  Download data, such as images, boxes, into various formats for machine learning e,g, COCO, CIFAR, or PASCAL VOC format
+* `aidata load --help` -  Load data, such as images, and boxes into either a Postgres or REDIS database
+* `aidata db --help` -  Commands related to database management
+* `aidata transform --help` - Commands related to transforming downloaded data
+* `aidata  -h` - Print help message and exit.
+ 
+Source code is available at [github.com/mbari-org/aidata](https://github.com/mbari-org/aidata/). 
+
+## Development
+See the [Development Guide](DEVELOPMENT.md) for more information on how to set up the development environment.
+
+**updated: 2025-01-28**

mbari_aidata-1.41.9/README.md

@@ -0,0 +1,121 @@
+[![MBARI](https://www.mbari.org/wp-content/uploads/2014/11/logo-mbari-3b.png)](http://www.mbari.org)
+[![semantic-release](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg)](https://github.com/semantic-release/semantic-release)
+[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![Python](https://img.shields.io/badge/language-Python-blue.svg)](https://www.python.org/downloads/)
+
+*mbari-aidata* is a command line tool to do extract, transform, load and download operations
+on AI data for a number of projects at MBARI that require detection, clustering or classification
+workflows.
+
+Full documentation is available on commands at [https://docs.mbari.org/internal/ai/data](https://docs.mbari.org/internal/ai/data/).
+ 
+This supports loading [sdcat](https://github.com/mbari-org/sdcat) formatted output and downloads from [Tator](https://www.tatorapp.com/) and 
+[Redis](https://redis.io) databases, although support for other data sources is also possible, e.g. [FathomNet](https://fathomnet.org/).
+so we decided to keep the name generic.
+
+This also supports loading media from a directory or URL, and transforming data into various 
+formats for machine learning, e.g. COCO, CIFAR, or PASCAL VOC format.
+
+## Requirements
+- Python 3.10 or higher
+- A Tator API token and Redis password for the .env file. Contact the MBARI AI team for access.
+- Docker for development and testing only
+
+## Installation 
+Install from PyPi
+
+```shell
+pip install mbari-aidata
+```
+ 
+Create the .env file with the following contents in the root directory of the project:
+```shell
+TATOR_TOKEN=your_api_token
+REDIS_PASSWORD=your_redis_password
+ENVIRONMENT=testing or production
+```
+
+Create a configuration file in the root directory of the project:
+```shell
+touch config_cfe.yaml
+```
+
+This file will be used to configure the project data, such as mounts, plugins, and database connections.
+```shell
+aidata download --version Baseline --labels "Diatoms, Copepods" --config config_cfe.yml
+```
+
+Example configuration file:
+```yaml
+# config_cfe.yml
+# Config file for CFE project production
+mounts:
+  - name: "image"
+    path: "/mnt/CFElab"
+    host: "mantis.shore.mbari.org"
+    nginx_root: "/CFElab"
+
+  - name: "video"
+    path: "/mnt/CFElab"
+    host: "mantis.shore.mbari.org"
+    nginx_root: "/CFElab"
+
+
+plugins:
+  - name: "extractor"
+    module: "mbari_aidata.plugins.extractors.tap_cfe_media"
+    function: "extract_media"
+
+redis:
+  host: "doris.shore.mbari.org"
+  port: 6382
+
+vss:
+  project: "902111-CFE"
+  model: "google/vit-base-patch16-224"
+
+tator:
+  project: "902111-CFE"
+  host: "mantis.shore.mbari.org"
+  image:
+    attributes:
+      iso_datetime:
+        type: datetime
+      depth:
+        type: float
+  video:
+    attributes:
+      iso_start_datetime:
+        type: datetime
+  box:
+    attributes:
+      Label:
+        type: string
+      score:
+        type: float
+      cluster:
+        type: string
+      saliency:
+        type: float
+      area:
+        type: int
+      exemplar:
+        type: bool
+```
+
+A docker version is also available at `mbari/aidata:latest` or `mbari/aidata:latest:cuda-124`.
+
+## Commands
+
+* `aidata download --help` -  Download data, such as images, boxes, into various formats for machine learning e,g, COCO, CIFAR, or PASCAL VOC format
+* `aidata load --help` -  Load data, such as images, and boxes into either a Postgres or REDIS database
+* `aidata db --help` -  Commands related to database management
+* `aidata transform --help` - Commands related to transforming downloaded data
+* `aidata  -h` - Print help message and exit.
+ 
+Source code is available at [github.com/mbari-org/aidata](https://github.com/mbari-org/aidata/). 
+
+## Development
+See the [Development Guide](DEVELOPMENT.md) for more information on how to set up the development environment.
+
+**updated: 2025-01-28**

mbari_aidata-1.41.9/mbari_aidata/.gitignore

@@ -0,0 +1,45 @@
+# database files
+*.db
+
+# test data
+data/
+*.png
+*.tar.gz
+
+# Pycharm
+.idea/
+
+# Node artifact files
+node_modules/
+dist/
+
+# Compiled Python bytecode
+*.py[cod]
+
+# JetBrains IDE
+.idea/
+
+# Unit test reports
+TEST*.xml
+
+# Generated by MacOS
+.DS_Store
+
+# Generated by Windows
+Thumbs.db
+
+# Applications
+*.app
+*.exe
+*.war
+
+# Large media files
+*.mp4
+*.tiff
+*.avi
+*.flv
+*.mov
+*.wmv
+
+#Mkdocs
+docs/site

mbari_aidata-1.41.9/mbari_aidata/__init__.py

@@ -0,0 +1,2 @@
+#mbari_aidata/__init__.py
+__version__ = "1.41.9"

mbari_aidata-1.41.9/mbari_aidata/__main__.py

@@ -0,0 +1,100 @@
+# mbari_aidata, Apache-2.0 license
+# Filename: __main__.py
+# Description: Main entry point for the mbari_aidata command line interface
+from datetime import datetime
+from pathlib import Path
+
+import pytz
+import click
+import sys
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from pathlib import Path
+
+from mbari_aidata.commands.download import download
+from mbari_aidata.commands.load_images import load_images
+from mbari_aidata.commands.load_video import load_video
+from mbari_aidata.commands.load_exemplars import load_exemplars
+from mbari_aidata.commands.db_utils import reset_redis
+from mbari_aidata.commands.transform import transform, voc_to_yolo
+from mbari_aidata.logger import err, info
+
+from mbari_aidata import __version__
+from mbari_aidata.commands.load_queue import load_queue
+from mbari_aidata.commands.load_boxes import load_boxes
+
+if "LOG_PATH" not in locals():
+    LOG_PATH = Path.home().as_posix()
+
+
+@click.group(context_settings={"help_option_names": ["-h", "--help"]})
+@click.version_option(__version__, "-V", "--version", message="%(prog)s, version %(version)s")
+def cli():
+    """
+    Load data to tator database from a command line.
+    """
+    pass
+
+
+@click.group(name="load")
+def cli_load():
+    """
+    Load data, such as images, boxes, and exemplars into either a Postgres or REDIS database
+    """
+    pass
+
+
+cli.add_command(cli_load)
+cli_load.add_command(load_images)
+cli_load.add_command(load_video)
+cli_load.add_command(load_boxes)
+cli_load.add_command(load_queue)
+cli_load.add_command(load_exemplars)
+
+
+@click.group(name="download")
+def cli_download():
+    """
+    Download data, such as images, boxes, into various formats for machine learning e,g, COCO, CIFAR, or PASCAL VOC format
+    """
+    pass
+
+
+cli.add_command(cli_download)
+cli_download.add_command(download)
+
+
+@click.group(name="db")
+def cli_db():
+    """
+    Commands related to database management
+    """
+    pass
+
+
+cli.add_command(cli_db)
+cli_db.add_command(reset_redis)
+
+
+@click.group(name="transform")
+def cli_transform():
+    """
+    Commands related to transforming downloaded data
+    """
+    pass
+
+
+cli.add_command(cli_transform)
+cli_transform.add_command(transform)
+cli_transform.add_command(voc_to_yolo)
+
+if __name__ == "__main__":
+    try:
+        start = datetime.now(pytz.utc)
+        cli()
+        end = datetime.now(pytz.utc)
+        info(f"Done. Elapsed time: {end - start} seconds")
+    except Exception as e:
+        err(f"Exiting. Error: {e}")
+        exit(-1)

mbari_aidata-1.41.9/mbari_aidata/commands/db_utils.py

@@ -0,0 +1,35 @@
+# mbari_aidata, Apache-2.0 license
+# Filename: commands/db_utils.py
+# Description: Miscellaneous functions for working with the database
+
+
+import click
+import redis
+
+from mbari_aidata import common_args
+from mbari_aidata.logger import create_logger_file, info
+from mbari_aidata.plugins.loaders.tator.common import init_yaml_config
+from mbari_aidata.predictors.process_vits import ViTWrapper
+
+
+@click.command("reset", help="Reset the REDIS server")
+@common_args.yaml_config
+@click.option("--redis-password", type=str, required=True, help="Password for the REDIS server")
+def reset_redis(redis_password: str, config: str) -> bool:
+    """Reset the REDIS database."""
+    try:
+        # Load the configuration file
+        # Each project needs a separate redis server for exemplar embeddings - this
+        # is done through separate ports
+        config_dict = init_yaml_config(config)
+        redis_host = config_dict["redis"]["host"]
+        redis_port = config_dict["redis"]["port"]
+        vss_model = config_dict["vss"]["model"]
+        info(f"Connecting to REDIS server at {redis_host}:{redis_port}")
+        r = redis.Redis(host=redis_host, port=redis_port, password=redis_password)
+        ViTWrapper(r, model_name=vss_model, reset=True)
+        info("Redis server reset")
+        return True
+    except Exception as e:
+        info(f"Error resetting REDIS server: {e}")
+        return False

mbari_aidata-1.41.9/mbari_aidata/commands/download.py

@@ -0,0 +1,194 @@
+# mbari_aidata, Apache-2.0 license
+# Filename: commands/download.py
+# Description: Download a dataset for training detection or classification models
+
+from pathlib import Path
+
+import click
+
+from mbari_aidata import common_args
+from mbari_aidata.logger import create_logger_file, info, exception
+from mbari_aidata.generators.coco_voc import download as download_full
+
+from mbari_aidata.plugins.loaders.tator.common import init_yaml_config, init_api_project, find_project
+
+# Default values
+# The base directory is the same directory as this file
+DEFAULT_BASE_DIR = Path.home() / "mbari_aidata" / "datasets"
+
+
+@click.command(name="dataset", help="Download a dataset for training detection or classification models")
+@common_args.token
+@common_args.yaml_config
+@common_args.version
+@click.option(
+    "--base-path",
+    default=DEFAULT_BASE_DIR,
+    type=Path,
+    help=f"Path to the base directory to save all data to. Defaults to {DEFAULT_BASE_DIR}",
+)
+@click.option("--group", help="Group name, e.g. VB250")
+@click.option("--depth", type=int, help="Depth, e.g. 200")
+@click.option("--section", help="Media section name, e.g. 25000_depth_v1")
+@click.option("--min-saliency", type=int, help="Minimum saliency score")
+@click.option("--max-saliency", type=int, help="Maximum saliency score")
+@click.option("--min-score", type=float, help="Minimum model score")
+@click.option("--generator", help="Generator name, e.g. vars-labelbot or vars-annotation")
+@click.option("--labels", default="all", help='Comma separated list of labels to download, or "all" for all labels.')
+@click.option(
+    "--concepts",
+    default="all",
+    help='Comma separated list of concepts to download, or "all" for all concepts. For legacy projects only',
+)
+@click.option("--crop-roi", is_flag=True, help="True to download the rois cropped from the original images/video.")
+@click.option("--resize", type=int, help="Resize images to this size after cropping them.")
+@click.option("--voc", is_flag=True, help="True if export as VOC dataset, False if not.")
+@click.option("--coco", is_flag=True, help="True if export as COCO dataset, False if not.")
+@click.option("--cifar", is_flag=True, help="True if export as CIFAR dataset, False if not.")
+@click.option("--cifar-size", default=32, help="Size of CIFAR images.")
+@click.option("--save-score", is_flag=True, help="True to save score in YOLO output, False if not.")
+@click.option("--verified", is_flag=True, help="True if only download verified annotations.")
+@click.option("--unverified", is_flag=True, help="True if only download not verified annotations.")
+@click.option("--single-class", type=str, help="Set to collapse all classes into a single class, e.g. 'marine organism'")
+@click.option(
+    "--skip-image-download", is_flag=True, help="Skip image download, only download annotations. CIFAR requires images."
+)
+def download(
+    token: str,
+    config: str,
+    base_path: Path,
+    group: str,
+    depth: int,
+    section: str,
+    min_saliency: int,
+    max_saliency: int,
+    min_score: float,
+    version: str,
+    generator: str,
+    labels: str,
+    concepts: str,
+    crop_roi: bool,
+    resize: int,
+    voc: bool,
+    cifar: bool,
+    coco: bool,
+    cifar_size: int,
+    save_score: bool,
+    single_class: str,
+    skip_image_download: bool,
+    verified: bool,
+    unverified: bool,
+) -> bool:
+    create_logger_file("download")
+    try:
+        base_path.mkdir(exist_ok=True, parents=True)
+        # Load the configuration file
+        config_dict = init_yaml_config(config)
+        project = config_dict["tator"]["project"]
+        host = config_dict["tator"]["host"]
+
+        # Initialize the Tator API
+        api, tator_project = init_api_project(host, token, project)
+
+        # Find the project
+        project = find_project(api, project)
+        info(f"Found project id: {project.name} for project {project}")
+
+        # Download a dataset by its version if it exists
+        if version:
+            data_path = base_path / version
+        else:
+            data_path = base_path
+        data_path.mkdir(exist_ok=True)
+        info(f"Downloading data to {data_path}")
+
+        # Convert comma separated list of concepts to a list
+        if labels == "all":
+            labels_list = []
+        else:
+            labels_list = labels.split(",")
+            labels_list = [l.strip() for l in labels_list]
+            # Check if this is empty
+            if len(labels_list) == 1 and labels_list[0] == "":
+                labels_list = []
+            # Strip off any zero length strings
+            labels_list = [l for l in labels_list if len(l) > 0]
+        if concepts == "all":
+            concepts_list = []
+        else:
+            concepts_list = concepts.split(",")
+            concepts_list = [l.strip() for l in concepts_list]
+            # Check if this is empty
+            if len(concepts_list) == 1 and concepts_list[0] == "":
+                concepts_list = []
+            # Strip off any zero length strings
+            concepts_list = [c for c in concepts_list if len(c) > 0]
+
+        # Convert comma separated list of versions to a list
+        if version:
+            version_list = version.split(",")
+            version_list = [l.strip() for l in version_list]
+        else:
+            # If no version is specified, download all versions
+            versions = api.get_version_list(project.id)
+            version_list = [v.name for v in versions]
+
+        success = download_full(
+            api,
+            project_id=project.id,
+            group=group,
+            depth=depth,
+            section=section,
+            min_saliency=min_saliency,
+            max_saliency=max_saliency,
+            min_score=min_score,
+            version_list=version_list,
+            verified=verified,
+            unverified=unverified,
+            generator=generator,
+            output_path=data_path,
+            labels_list=labels_list,
+            concepts_list=concepts_list,
+            single_class=single_class,
+            skip_image_download=skip_image_download,
+            save_score=save_score,
+            cifar_size=cifar_size,
+            voc=voc,
+            coco=coco,
+            cifar=cifar,
+            crop_roi=crop_roi,
+            resize=resize
+        )
+        return success
+    except Exception as e:
+        exception(f"Error: {e}")
+        return False
+
+
+if __name__ == "__main__":
+    import os
+
+    # To run this script, you need to have the TATOR_TOKEN environment variable set and uncomment all @click decorators above
+    # TODO: move this to pytest
+    os.environ["ENVIRONMENT"] = "TESTING"
+    test_path = Path(__file__).parent.parent.parent / "tests" / "data" / "i2map"
+    yaml_path = Path(__file__).parent.parent.parent / "tests" / "config" / "config_i2map.yml"
+    base_path = Path(__file__).parent.parent.parent / "tests" / "data" / "download"
+    tator_token = os.getenv("TATOR_TOKEN")
+    download(
+        token=tator_token,
+        config=yaml_path.as_posix(),
+        version="dino_vits8_20240205_225539,dino_vits8_20240207_022529,dinov2_vits14_hdbscan_",
+        base_path=base_path,
+        voc=True,
+        labels="Acanthamunnopsis milleri,Euphausiacea1,Pyrosoma1,Pyrosoma2",
+        concepts="",
+        cifar=True,
+        coco=True,
+        save_score=False,
+        skip_image_download=False,
+        group="",
+        depth="",
+        generator="",
+        cifar_size=32,
+    )