hirundo 0.1.3__tar.gz

hirundo-0.1.3/LICENSE

@@ -0,0 +1,9 @@
+MIT License
+
+Copyright (c) 2024, Hirundo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

hirundo-0.1.3/PKG-INFO

@@ -0,0 +1,114 @@
+Metadata-Version: 2.1
+Name: hirundo
+Version: 0.1.3
+Summary: This package is used to interface with Hirundo's platform. It provides a simple API to optimize your ML datasets.
+Author-email: Hirundo <dev@hirundo.io>
+License: MIT License
+        
+        Copyright (c) 2024, Hirundo
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/Hirundo-io/hirundo-client
+Keywords: dataset,machine learning,data science,data engineering
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml>=6.0.1
+Requires-Dist: types-PyYAML>=6.0.12
+Requires-Dist: pydantic>=2.7.1
+Requires-Dist: twine>=5.0.0
+Requires-Dist: python-dotenv>=1.0.1
+Requires-Dist: types-requests>=2.31.0
+Requires-Dist: typer>=0.12.3
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: stamina>=24.2.0
+Requires-Dist: httpx-sse>=0.4.0
+Provides-Extra: dev
+Requires-Dist: pyyaml>=6.0.1; extra == "dev"
+Requires-Dist: types-PyYAML>=6.0.12; extra == "dev"
+Requires-Dist: pydantic>=2.7.1; extra == "dev"
+Requires-Dist: twine>=5.0.0; extra == "dev"
+Requires-Dist: python-dotenv>=1.0.1; extra == "dev"
+Requires-Dist: types-requests>=2.31.0; extra == "dev"
+Requires-Dist: types-setuptools>=69.5.0; extra == "dev"
+Requires-Dist: typer>=0.12.3; extra == "dev"
+Requires-Dist: httpx>=0.27.0; extra == "dev"
+Requires-Dist: stamina>=24.2.0; extra == "dev"
+Requires-Dist: httpx-sse>=0.4.0; extra == "dev"
+Requires-Dist: pytest>=8.2.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23.6; extra == "dev"
+Requires-Dist: uv; extra == "dev"
+Requires-Dist: pre-commit>=3.7.1; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: bumpver; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: sphinx>=7.4.7; extra == "docs"
+Requires-Dist: sphinx-autobuild>=2024.4.16; extra == "docs"
+Requires-Dist: sphinx-click>=5.0.1; extra == "docs"
+Requires-Dist: autodoc_pydantic>=2.2.0; extra == "docs"
+Requires-Dist: furo; extra == "docs"
+Requires-Dist: sphinx-multiversion; extra == "docs"
+
+# Hirundo client
+
+This repo contains the source code for the Hirundo client library
+
+## Usage:
+
+To learn about how to use this library, please visit the [http://docs.hirundo.io/](documentation) or see the Google Colab examples.
+
+## Development:
+
+### Install dev dependencies
+
+```bash
+pip install -r dev-requirements.txt
+```
+
+Note: You can install and use `uv` as a faster drop-in replacement for `pip`. We have it as part of our dev dependencies for this reason.
+
+### Install `git` hooks (optional)
+
+```bash
+pre-commit install
+```
+
+### Check lint and apply formatting with Ruff (optional; pre-commit hooks run this automatically)
+
+```bash
+ruff check
+ruff format
+```
+
+### Change packages
+
+#### Update `requirements.txt` files
+
+```bash
+uv pip compile pyproject.toml
+uv pip compile --extra dev -o dev-requirements.txt -c requirements.txt pyproject.toml
+uv pip compile --extra docs -o docs-requirements.txt -c requirements.txt pyproject.toml
+```
+
+#### Sync installed packages
+
+```bash
+uv pip sync dev-requirements.txt
+```
+
+### Build process
+
+To build the package, run:
+`python -m build`
+
+### Publish documentation & releases
+
+Documentation & releases are published via GitHub Actions on merges to `main`.

hirundo-0.1.3/README.md

@@ -0,0 +1,55 @@
+# Hirundo client
+
+This repo contains the source code for the Hirundo client library
+
+## Usage:
+
+To learn about how to use this library, please visit the [http://docs.hirundo.io/](documentation) or see the Google Colab examples.
+
+## Development:
+
+### Install dev dependencies
+
+```bash
+pip install -r dev-requirements.txt
+```
+
+Note: You can install and use `uv` as a faster drop-in replacement for `pip`. We have it as part of our dev dependencies for this reason.
+
+### Install `git` hooks (optional)
+
+```bash
+pre-commit install
+```
+
+### Check lint and apply formatting with Ruff (optional; pre-commit hooks run this automatically)
+
+```bash
+ruff check
+ruff format
+```
+
+### Change packages
+
+#### Update `requirements.txt` files
+
+```bash
+uv pip compile pyproject.toml
+uv pip compile --extra dev -o dev-requirements.txt -c requirements.txt pyproject.toml
+uv pip compile --extra docs -o docs-requirements.txt -c requirements.txt pyproject.toml
+```
+
+#### Sync installed packages
+
+```bash
+uv pip sync dev-requirements.txt
+```
+
+### Build process
+
+To build the package, run:
+`python -m build`
+
+### Publish documentation & releases
+
+Documentation & releases are published via GitHub Actions on merges to `main`.

hirundo-0.1.3/hirundo/__init__.py

@@ -0,0 +1,35 @@
+from .dataset_optimization import (
+    HirundoError,
+    OptimizationDataset,
+)
+from .enum import (
+    DatasetMetadataType,
+    LabellingType,
+)
+from .git import GitRepo
+from .storage import (
+    StorageGCP,
+    # StorageAzure,  TODO: Azure storage integration is coming soon
+    StorageGit,
+    StorageIntegration,
+    StorageLink,
+    StorageS3,
+    StorageTypes,
+)
+
+__all__ = [
+    "HirundoError",
+    "OptimizationDataset",
+    "LabellingType",
+    "DatasetMetadataType",
+    "GitRepo",
+    "StorageLink",
+    "StorageTypes",
+    "StorageS3",
+    "StorageGCP",
+    # "StorageAzure",  TODO: Azure storage integration is coming soon
+    "StorageGit",
+    "StorageIntegration",
+]
+
+__version__ = "0.1.3"

hirundo-0.1.3/hirundo/__main__.py

@@ -0,0 +1,3 @@
+from .cli import app
+
+app()

hirundo-0.1.3/hirundo/_constraints.py

@@ -0,0 +1,21 @@
+from typing import Annotated
+
+from pydantic import StringConstraints
+
+S3BucketUrl = Annotated[
+    str,
+    StringConstraints(
+        min_length=8,
+        max_length=1023,
+        pattern=r"s3?://[a-z0-9.-]{3,64}[/]?",  # Only allow real S3 bucket URLs
+    ),
+]
+
+StorageIntegrationName = Annotated[
+    str,
+    StringConstraints(
+        min_length=1,
+        max_length=255,
+        pattern=r"^[a-zA-Z0-9-_]+$",
+    ),
+]

hirundo-0.1.3/hirundo/_env.py

@@ -0,0 +1,12 @@
+import os
+
+from dotenv import load_dotenv
+
+load_dotenv()
+
+API_HOST = os.getenv("API_HOST", "https://api.hirundo.io")
+API_KEY = os.getenv("API_KEY")
+if not API_KEY:
+    raise ValueError(
+        "API_KEY is not set. Please run `hirundo setup` to set the API key"
+    )

hirundo-0.1.3/hirundo/_headers.py

@@ -0,0 +1,9 @@
+from hirundo._env import API_KEY
+
+json_headers = {
+    "Content-Type": "application/json",
+    "Accept": "application/json",
+}
+auth_headers = {
+    "Authorization": f"Bearer {API_KEY}",
+}

hirundo-0.1.3/hirundo/_iter_sse_retrying.py

@@ -0,0 +1,97 @@
+import asyncio
+import time
+from collections.abc import AsyncGenerator, Generator
+from typing import Union
+
+import httpx
+from httpx_sse import ServerSentEvent, aconnect_sse, connect_sse
+from stamina import retry
+
+
+# Credit: https://github.com/florimondmanca/httpx-sse/blob/master/README.md#handling-reconnections
+def iter_sse_retrying(
+    client: httpx.Client,
+    method: str,
+    url: str,
+    headers: Union[dict[str, str], None] = None,
+) -> Generator[ServerSentEvent, None, None]:
+    if headers is None:
+        headers = {}
+    last_event_id = ""
+    reconnection_delay = 0.0
+
+    # `stamina` will apply jitter and exponential backoff on top of
+    # the `retry` reconnection delay sent by the server.
+    # httpx.ReadError is thrown when there is a network error.
+    #   Some network errors may be temporary, hence the retries.
+    # httpx.RemoteProtocolError is thrown when the server closes the connection.
+    #  This may happen when the server is overloaded and closes the connection or
+    #  when Kubernetes restarts / replaces a pod.
+    #  Likewise, this will likely be temporary, hence the retries.
+    @retry(on=(httpx.ReadError, httpx.RemoteProtocolError))
+    def _iter_sse():
+        nonlocal last_event_id, reconnection_delay
+
+        time.sleep(reconnection_delay)
+
+        connect_headers = {
+            **headers,
+            "Accept": "text/event-stream",
+            "X-Accel-Buffering": "no",
+        }
+
+        if last_event_id:
+            connect_headers["Last-Event-ID"] = last_event_id
+
+        with connect_sse(client, method, url, headers=connect_headers) as event_source:
+            for sse in event_source.iter_sse():
+                last_event_id = sse.id
+
+                if sse.retry is not None:
+                    reconnection_delay = sse.retry / 1000
+
+                yield sse
+
+    return _iter_sse()
+
+
+async def aiter_sse_retrying(
+    client: httpx.AsyncClient,
+    method: str,
+    url: str,
+    headers: dict[str, str],
+) -> AsyncGenerator[ServerSentEvent, None]:
+    last_event_id = ""
+    reconnection_delay = 0.0
+
+    # `stamina` will apply jitter and exponential backoff on top of
+    # the `retry` reconnection delay sent by the server.
+    # httpx.ReadError is thrown when there is a network error.
+    #   Some network errors may be temporary, hence the retries.
+    # httpx.RemoteProtocolError is thrown when the server closes the connection.
+    #  This may happen when the server is overloaded and closes the connection or
+    #  when Kubernetes restarts / replaces a pod.
+    #  Likewise, this will likely be temporary, hence the retries.
+    @retry(on=(httpx.ReadError, httpx.RemoteProtocolError))
+    async def _iter_sse() -> AsyncGenerator[ServerSentEvent, None]:
+        nonlocal last_event_id, reconnection_delay
+
+        await asyncio.sleep(reconnection_delay)
+
+        connect_headers = {**headers, "Accept": "text/event-stream"}
+
+        if last_event_id:
+            connect_headers["Last-Event-ID"] = last_event_id
+
+        async with aconnect_sse(
+            client, method, url, headers=connect_headers
+        ) as event_source:
+            async for sse in event_source.aiter_sse():
+                last_event_id = sse.id
+
+                if sse.retry is not None:
+                    reconnection_delay = sse.retry / 1000
+
+                yield sse
+
+    return _iter_sse()

hirundo-0.1.3/hirundo/_timeouts.py

@@ -0,0 +1,2 @@
+READ_TIMEOUT = 30.0
+MODIFY_TIMEOUT = 60.0

hirundo-0.1.3/hirundo/cli.py

@@ -0,0 +1,132 @@
+import re
+import sys
+from typing import Annotated
+from urllib.parse import urlparse
+
+import typer
+
+from hirundo._env import API_HOST
+
+docs = "sphinx" in sys.modules
+hirundo_epilog = (
+    None
+    if docs
+    else "Made with ❤️ by Hirundo. Visit https://www.hirundo.io for more information."
+)
+
+app = typer.Typer(
+    name="hirundo", no_args_is_help=True, rich_markup_mode="rich", epilog=hirundo_epilog
+)
+
+
+def upsert_env(var_name: str, var_value: str):
+    """
+    Change an environment variable in the .env file.
+    If the variable does not exist, it will be added.
+
+    Args:
+        var_name: The name of the environment variable to change.
+        var_value: The new value of the environment variable.
+    """
+    dotenv = "./.env"
+    regex = re.compile(rf"^{var_name}=.*$")
+    with open(dotenv) as f:
+        lines = f.readlines()
+
+    with open(dotenv, "w") as f:
+        f.writelines(line for line in lines if not regex.search(line) and line != "\n")
+
+    with open(dotenv, "a") as f:
+        f.writelines(f"\n{var_name}={var_value}")
+
+
+def fix_api_host(api_host: str):
+    if not api_host.startswith("http") and not api_host.startswith("https"):
+        api_host = f"https://{api_host}"
+        print(
+            "API host must start with 'http://' or 'https://'. Automatically added 'https://'."
+        )
+    if (url := urlparse(api_host)) and url.path != "":
+        print("API host should not contain a path. Removing path.")
+        api_host = f"{url.scheme}://{url.hostname}"
+    return api_host
+
+
+@app.command("set-api-key", epilog=hirundo_epilog)
+def setup_api_key(
+    api_key: Annotated[
+        str,
+        typer.Option(
+            prompt="Please enter the API key value",
+            help=""
+            if docs
+            else f"Visit '{API_HOST}/api-key' to generate your API key.",
+        ),
+    ],
+):
+    """
+    Setup the API key for the Hirundo client library.
+    Values are saved to a .env file in the current directory for use by the library in requests.
+    """
+    upsert_env("API_KEY", api_key)
+    print("API key saved to .env for future use. Please do not share the .env file")
+
+
+@app.command("change-remote", epilog=hirundo_epilog)
+def change_api_remote(
+    api_host: Annotated[
+        str,  # TODO: Change to HttpUrl when https://github.com/tiangolo/typer/pull/723 is merged
+        typer.Option(
+            prompt="Please enter the API server address",
+            help=""
+            if docs
+            else f"Current API server address: '{API_HOST}'. This is the same address where you access the Hirundo web interface.",
+        ),
+    ],
+):
+    """
+    Change the API server address for the Hirundo client library.
+    This is the same address where you access the Hirundo web interface.
+    """
+    api_host = fix_api_host(api_host)
+
+    upsert_env("API_HOST", api_host)
+    print("API host saved to .env for future use. Please do not share this file")
+
+
+@app.command("setup", epilog=hirundo_epilog)
+def setup(
+    api_key: Annotated[
+        str,
+        typer.Option(
+            prompt="Please enter the API key value",
+            help=""
+            if docs
+            else f"Visit '{API_HOST}/api-key' to generate your API key.",
+        ),
+    ],
+    api_host: Annotated[
+        str,  # TODO: Change to HttpUrl as above
+        typer.Option(
+            prompt="Please enter the API server address",
+            help=""
+            if docs
+            else f"Current API server address: '{API_HOST}'. This is the same address where you access the Hirundo web interface.",
+        ),
+    ],
+):
+    """
+    Setup the Hirundo client library.
+    """
+    api_host = fix_api_host(api_host)
+    upsert_env("API_HOST", api_host)
+    upsert_env("API_KEY", api_key)
+    print(
+        "API host and API key saved to .env for future use. Please do not share this file"
+    )
+
+
+typer_click_object = typer.main.get_command(app)
+
+if __name__ == "__main__":
+    app()