hirundo 0.1.7__tar.gz → 0.1.8__tar.gz

hirundo-0.1.8/PKG-INFO

@@ -0,0 +1,176 @@
+Metadata-Version: 2.1
+Name: hirundo
+Version: 0.1.8
+Summary: This package is used to interface with Hirundo's platform. It provides a simple API to optimize your ML datasets.
+Author-email: Hirundo <dev@hirundo.io>
+License: MIT License
+        
+        Copyright (c) 2024, Hirundo
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/Hirundo-io/hirundo-client
+Keywords: dataset,machine learning,data science,data engineering
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml>=6.0.1
+Requires-Dist: types-PyYAML>=6.0.12
+Requires-Dist: pydantic>=2.7.1
+Requires-Dist: twine>=5.0.0
+Requires-Dist: python-dotenv>=1.0.1
+Requires-Dist: types-requests>=2.31.0
+Requires-Dist: typer>=0.12.3
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: stamina>=24.2.0
+Requires-Dist: httpx-sse>=0.4.0
+Requires-Dist: pandas>=2.2.2
+Requires-Dist: tqdm>=4.66.5
+Provides-Extra: dev
+Requires-Dist: pyyaml>=6.0.1; extra == "dev"
+Requires-Dist: types-PyYAML>=6.0.12; extra == "dev"
+Requires-Dist: pydantic>=2.7.1; extra == "dev"
+Requires-Dist: twine>=5.0.0; extra == "dev"
+Requires-Dist: python-dotenv>=1.0.1; extra == "dev"
+Requires-Dist: types-requests>=2.31.0; extra == "dev"
+Requires-Dist: types-setuptools>=69.5.0; extra == "dev"
+Requires-Dist: typer>=0.12.3; extra == "dev"
+Requires-Dist: httpx>=0.27.0; extra == "dev"
+Requires-Dist: stamina>=24.2.0; extra == "dev"
+Requires-Dist: httpx-sse>=0.4.0; extra == "dev"
+Requires-Dist: pytest>=8.2.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23.6; extra == "dev"
+Requires-Dist: uv; extra == "dev"
+Requires-Dist: pre-commit>=3.7.1; extra == "dev"
+Requires-Dist: ruff==0.6.5; extra == "dev"
+Requires-Dist: bumpver; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: sphinx>=7.4.7; extra == "docs"
+Requires-Dist: sphinx-autobuild>=2024.4.16; extra == "docs"
+Requires-Dist: sphinx-click>=5.0.1; extra == "docs"
+Requires-Dist: autodoc_pydantic>=2.2.0; extra == "docs"
+Requires-Dist: furo; extra == "docs"
+Requires-Dist: sphinx-multiversion; extra == "docs"
+
+# Hirundo
+
+This package exposes access to Hirundo APIs for dataset optimization for Machine Learning.
+
+Dataset optimization is currently available for datasets labelled for classification and object detection.
+
+
+Support dataset storage integrations include:
+   - Google Cloud (GCP) Storage
+   - Amazon Web Services (AWS) S3
+   - Git LFS (Large File Storage) repositories (e.g. GitHub or HuggingFace)
+
+Optimizing a classification dataset
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Currently ``hirundo`` requires a CSV file with the following columns (all columns are required):
+   - ``image_path``: The location of the image within the dataset ``root``
+   - ``label``: The label of the image, i.e. which the class that was annotated for this image
+
+And outputs a CSV with the same columns and:
+   - ``suspect_level``: mislabel suspect level
+   - ``suggested_label``: suggested label
+   - ``suggested_label_conf``: suggested label confidence
+
+Optimizing an object detection (OD) dataset
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Currently ``hirundo`` requires a CSV file with the following columns (all columns are required):
+   - ``image_path``: The location of the image within the dataset ``root``
+   - ``bbox_id``: The index of the bounding box within the dataset. Used to indicate label suspects
+   - ``label``: The label of the image, i.e. which the class that was annotated for this image
+   - ``x1``, ``y1``, ``x2``, ``y2``: The bounding box coordinates of the object within the image
+
+And outputs a CSV with the same columns and:
+   - ``suspect_level``: object mislabel suspect level
+   - ``suggested_label``: suggested object label
+   - ``suggested_label_conf``: suggested object label confidence
+
+Note: This Python package must be used alongside a Hirundo server, either the SaaS platform, a custom VPC deployment or an on-premises installation.
+
+
+## Installation
+
+You can install the codebase with a simple `pip install hirundo` to install the latest version of this package. If you prefer to install from the Git repository and/or need a specific version or branch, you can simply clone the repository, check out the relevant commit and then run `pip install .` to install that version. A full list of dependencies can be found in `requirements.txt`, but these will be installed automatically by either of these commands.
+
+## Usage
+
+Classification example:
+```
+from hirundo.dataset_optimization import OptimizationDataset
+from hirundo.enum import LabellingType
+from hirundo.storage import StorageIntegration, StorageLink, StorageTypes
+
+test_dataset = OptimizationDataset(
+    name="TEST-GCP cifar 100 classification dataset",
+    labelling_type=LabellingType.SingleLabelClassification,
+    dataset_storage=StorageLink(
+        storage_integration=StorageIntegration(
+            name="cifar100bucket",
+            type=StorageTypes.GCP,
+            gcp=StorageGCP(
+                bucket_name="cifar100bucket",
+                project="Hirundo-global",
+                credentials_json=json.loads(os.environ["GCP_CREDENTIALS"]),
+            ),
+        ),
+        path="/pytorch-cifar/data",
+    ),
+    dataset_metadata_path="cifar100.csv",
+    classes=cifar100_classes,
+)
+
+test_dataset.run_optimization()
+results = test_dataset.check_run()
+print(results)
+```
+
+
+Object detection example:
+
+```
+from hirundo.dataset_optimization import OptimizationDataset
+from hirundo.enum import LabellingType
+from hirundo.storage import StorageIntegration, StorageLink, StorageTypes
+
+test_dataset = OptimizationDataset(
+    name=f"TEST-HuggingFace-BDD-100k-validation-OD-validation-dataset{unique_id}",
+    labelling_type=LabellingType.ObjectDetection,
+    dataset_storage=StorageLink(
+        storage_integration=StorageIntegration(
+            name=f"BDD-100k-validation-dataset{unique_id}",
+            type=StorageTypes.GIT,
+            git=StorageGit(
+                repo=GitRepo(
+                    name=f"BDD-100k-validation-dataset{unique_id}",
+                    repository_url="https://git@hf.co/datasets/hirundo-io/bdd100k-validation-only",
+                ),
+                branch="main",
+            ),
+        ),
+        path="/BDD100K Val from Hirundo.zip/bdd100k",
+    ),
+    dataset_metadata_path="bdd100k.csv",
+)
+
+test_dataset.run_optimization()
+results = test_dataset.check_run()
+print(results)
+```
+
+Note: Currently we only support the main CPython release 3.9, 3.10 and 3.11. PyPy support may be introduced in the future.
+
+## Further documentation
+
+To learn about mroe how to use this library, please visit the [http://docs.hirundo.io/](documentation) or see the Google Colab examples.

hirundo-0.1.8/README.md

@@ -0,0 +1,115 @@
+# Hirundo
+
+This package exposes access to Hirundo APIs for dataset optimization for Machine Learning.
+
+Dataset optimization is currently available for datasets labelled for classification and object detection.
+
+
+Support dataset storage integrations include:
+   - Google Cloud (GCP) Storage
+   - Amazon Web Services (AWS) S3
+   - Git LFS (Large File Storage) repositories (e.g. GitHub or HuggingFace)
+
+Optimizing a classification dataset
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Currently ``hirundo`` requires a CSV file with the following columns (all columns are required):
+   - ``image_path``: The location of the image within the dataset ``root``
+   - ``label``: The label of the image, i.e. which the class that was annotated for this image
+
+And outputs a CSV with the same columns and:
+   - ``suspect_level``: mislabel suspect level
+   - ``suggested_label``: suggested label
+   - ``suggested_label_conf``: suggested label confidence
+
+Optimizing an object detection (OD) dataset
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Currently ``hirundo`` requires a CSV file with the following columns (all columns are required):
+   - ``image_path``: The location of the image within the dataset ``root``
+   - ``bbox_id``: The index of the bounding box within the dataset. Used to indicate label suspects
+   - ``label``: The label of the image, i.e. which the class that was annotated for this image
+   - ``x1``, ``y1``, ``x2``, ``y2``: The bounding box coordinates of the object within the image
+
+And outputs a CSV with the same columns and:
+   - ``suspect_level``: object mislabel suspect level
+   - ``suggested_label``: suggested object label
+   - ``suggested_label_conf``: suggested object label confidence
+
+Note: This Python package must be used alongside a Hirundo server, either the SaaS platform, a custom VPC deployment or an on-premises installation.
+
+
+## Installation
+
+You can install the codebase with a simple `pip install hirundo` to install the latest version of this package. If you prefer to install from the Git repository and/or need a specific version or branch, you can simply clone the repository, check out the relevant commit and then run `pip install .` to install that version. A full list of dependencies can be found in `requirements.txt`, but these will be installed automatically by either of these commands.
+
+## Usage
+
+Classification example:
+```
+from hirundo.dataset_optimization import OptimizationDataset
+from hirundo.enum import LabellingType
+from hirundo.storage import StorageIntegration, StorageLink, StorageTypes
+
+test_dataset = OptimizationDataset(
+    name="TEST-GCP cifar 100 classification dataset",
+    labelling_type=LabellingType.SingleLabelClassification,
+    dataset_storage=StorageLink(
+        storage_integration=StorageIntegration(
+            name="cifar100bucket",
+            type=StorageTypes.GCP,
+            gcp=StorageGCP(
+                bucket_name="cifar100bucket",
+                project="Hirundo-global",
+                credentials_json=json.loads(os.environ["GCP_CREDENTIALS"]),
+            ),
+        ),
+        path="/pytorch-cifar/data",
+    ),
+    dataset_metadata_path="cifar100.csv",
+    classes=cifar100_classes,
+)
+
+test_dataset.run_optimization()
+results = test_dataset.check_run()
+print(results)
+```
+
+
+Object detection example:
+
+```
+from hirundo.dataset_optimization import OptimizationDataset
+from hirundo.enum import LabellingType
+from hirundo.storage import StorageIntegration, StorageLink, StorageTypes
+
+test_dataset = OptimizationDataset(
+    name=f"TEST-HuggingFace-BDD-100k-validation-OD-validation-dataset{unique_id}",
+    labelling_type=LabellingType.ObjectDetection,
+    dataset_storage=StorageLink(
+        storage_integration=StorageIntegration(
+            name=f"BDD-100k-validation-dataset{unique_id}",
+            type=StorageTypes.GIT,
+            git=StorageGit(
+                repo=GitRepo(
+                    name=f"BDD-100k-validation-dataset{unique_id}",
+                    repository_url="https://git@hf.co/datasets/hirundo-io/bdd100k-validation-only",
+                ),
+                branch="main",
+            ),
+        ),
+        path="/BDD100K Val from Hirundo.zip/bdd100k",
+    ),
+    dataset_metadata_path="bdd100k.csv",
+)
+
+test_dataset.run_optimization()
+results = test_dataset.check_run()
+print(results)
+```
+
+Note: Currently we only support the main CPython release 3.9, 3.10 and 3.11. PyPy support may be introduced in the future.
+
+## Further documentation
+
+To learn about mroe how to use this library, please visit the [http://docs.hirundo.io/](documentation) or see the Google Colab examples.

{hirundo-0.1.7 → hirundo-0.1.8}/hirundo/__init__.py

@@ -32,4 +32,4 @@ __all__ = [
     "StorageIntegration",
 ]
 
-__version__ = "0.1.7"
+__version__ = "0.1.8"

hirundo-0.1.8/hirundo/_env.py

@@ -0,0 +1,26 @@
+import enum
+import os
+from pathlib import Path
+
+from dotenv import load_dotenv
+
+
+class EnvLocation(enum.Enum):
+    DOTENV = Path.cwd() / ".env"
+    HOME = Path.home() / ".hirundo.conf"
+
+
+if os.path.exists(EnvLocation.DOTENV.value):
+    load_dotenv(EnvLocation.DOTENV.value)
+elif os.path.exists(EnvLocation.HOME.value):
+    load_dotenv(EnvLocation.HOME.value)
+
+API_HOST = os.getenv("API_HOST", "https://api.hirundo.io")
+API_KEY = os.getenv("API_KEY")
+
+
+def check_api_key():
+    if not API_KEY:
+        raise ValueError(
+            "API_KEY is not set. Please run `hirundo setup` to set the API key"
+        )

hirundo-0.1.8/hirundo/_http.py

@@ -0,0 +1,14 @@
+from requests import Response
+
+import hirundo.logger
+
+logger = hirundo.logger.get_logger(__name__)
+
+
+def raise_for_status_with_reason(response: Response):
+    try:
+        response.reason = response.json().get("reason", None)
+    except Exception as e:
+        logger.debug("Failed to parse response as JSON: %s", e)
+
+    response.raise_for_status()

{hirundo-0.1.7 → hirundo-0.1.8}/hirundo/_iter_sse_retrying.py

@@ -1,7 +1,7 @@
 import asyncio
 import time
+import typing
 from collections.abc import AsyncGenerator, Generator
-from typing import Union
 
 import httpx
 from httpx_sse import ServerSentEvent, aconnect_sse, connect_sse
@@ -13,7 +13,7 @@ def iter_sse_retrying(
     client: httpx.Client,
     method: str,
     url: str,
-    headers: Union[dict[str, str], None] = None,
+    headers: typing.Optional[dict[str, str]] = None,
 ) -> Generator[ServerSentEvent, None, None]:
     if headers is None:
         headers = {}

{hirundo-0.1.7 → hirundo-0.1.8}/hirundo/cli.py

@@ -1,11 +1,14 @@
+import os
 import re
 import sys
+import typing
+from pathlib import Path
 from typing import Annotated
 from urllib.parse import urlparse
 
 import typer
 
-from hirundo._env import API_HOST
+from hirundo._env import API_HOST, EnvLocation
 
 docs = "sphinx" in sys.modules
 hirundo_epilog = (
@@ -23,7 +26,9 @@ app = typer.Typer(
 )
 
 
-def upsert_env(var_name: str, var_value: str):
+def _upsert_env(
+    dotenv_filepath: typing.Union[str, Path], var_name: str, var_value: str
+):
     """
     Change an environment variable in the .env file.
     If the variable does not exist, it will be added.
@@ -32,18 +37,30 @@ def upsert_env(var_name: str, var_value: str):
         var_name: The name of the environment variable to change.
         var_value: The new value of the environment variable.
     """
-    dotenv = "./.env"
     regex = re.compile(rf"^{var_name}=.*$")
-    with open(dotenv) as f:
-        lines = f.readlines()
+    lines = []
+    if os.path.exists(dotenv_filepath):
+        with open(dotenv_filepath) as f:
+            lines = f.readlines()
 
-    with open(dotenv, "w") as f:
+    with open(dotenv_filepath, "w") as f:
         f.writelines(line for line in lines if not regex.search(line) and line != "\n")
 
-    with open(dotenv, "a") as f:
+    with open(dotenv_filepath, "a") as f:
         f.writelines(f"\n{var_name}={var_value}")
 
 
+def upsert_env(var_name: str, var_value: str):
+    if os.path.exists(EnvLocation.DOTENV.value):
+        # If a `.env` file exists, re-use it
+        _upsert_env(EnvLocation.DOTENV.value, var_name, var_value)
+        return EnvLocation.DOTENV.name
+    else:
+        # Create a `.hirundo.conf` file with environment variables in the home directory
+        _upsert_env(EnvLocation.HOME.value, var_name, var_value)
+        return EnvLocation.HOME.name
+
+
 def fix_api_host(api_host: str):
     if not api_host.startswith("http") and not api_host.startswith("https"):
         api_host = f"https://{api_host}"
@@ -72,8 +89,15 @@ def setup_api_key(
     Setup the API key for the Hirundo client library.
     Values are saved to a .env file in the current directory for use by the library in requests.
     """
-    upsert_env("API_KEY", api_key)
-    print("API key saved to .env for future use. Please do not share the .env file")
+    saved_to = upsert_env("API_KEY", api_key)
+    if saved_to == EnvLocation.HOME.name:
+        print(
+            "API key saved to ~/.hirundo.conf for future use. Please do not share the ~/.hirundo.conf file since it contains your secret API key."
+        )
+    elif saved_to == EnvLocation.DOTENV.name:
+        print(
+            "API key saved to local .env file for future use. Please do not share the .env file since it contains your secret API key."
+        )
 
 
 @app.command("change-remote", epilog=hirundo_epilog)
@@ -94,8 +118,13 @@ def change_api_remote(
     """
     api_host = fix_api_host(api_host)
 
-    upsert_env("API_HOST", api_host)
-    print("API host saved to .env for future use. Please do not share this file")
+    saved_to = upsert_env("API_HOST", api_host)
+    if saved_to == EnvLocation.HOME.name:
+        print(
+            "API host saved to ~/.hirundo.conf for future use. Please do not share the ~/.hirundo.conf file"
+        )
+    elif saved_to == EnvLocation.DOTENV.name:
+        print("API host saved to .env for future use. Please do not share this file")
 
 
 @app.command("setup", epilog=hirundo_epilog)
@@ -123,11 +152,41 @@ def setup(
     Setup the Hirundo client library.
     """
     api_host = fix_api_host(api_host)
-    upsert_env("API_HOST", api_host)
-    upsert_env("API_KEY", api_key)
-    print(
-        "API host and API key saved to .env for future use. Please do not share this file"
-    )
+    api_host_saved_to = upsert_env("API_HOST", api_host)
+    api_key_saved_to = upsert_env("API_KEY", api_key)
+    if api_host_saved_to != api_key_saved_to:
+        print(
+            "API host and API key saved to different locations. This should not happen. Please report this issue."
+        )
+        if (
+            api_host_saved_to == EnvLocation.HOME.name
+            and api_key_saved_to == EnvLocation.DOTENV.name
+        ):
+            print(
+                "API host saved to ~/.hirundo.conf for future use. Please do not share the ~/.hirundo.conf file"
+            )
+            print(
+                "API key saved to local .env file for future use. Please do not share the .env file since it contains your secret API key."
+            )
+        elif (
+            api_host_saved_to == EnvLocation.DOTENV.name
+            and api_key_saved_to == EnvLocation.HOME.name
+        ):
+            print(
+                "API host saved to .env for future use. Please do not share this file"
+            )
+            print(
+                "API key saved to ~/.hirundo.conf for future use. Please do not share the ~/.hirundo.conf file since it contains your secret API key."
+            )
+        return
+    if api_host_saved_to == EnvLocation.HOME.name:
+        print(
+            "API host and API key saved to ~/.hirundo.conf for future use. Please do not share the ~/.hirundo.conf file since it contains your secret API key."
+        )
+    elif api_host_saved_to == EnvLocation.DOTENV.name:
+        print(
+            "API host and API key saved to .env for future use. Please do not share this file since it contains your secret API key."
+        )
 
 
 typer_click_object = typer.main.get_command(app)