labelr 0.7.0__tar.gz → 0.9.0__tar.gz

{labelr-0.7.0/src/labelr.egg-info → labelr-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: labelr
-Version: 0.7.0
+Version: 0.9.0
 Summary: A command-line tool to manage labeling tasks with Label Studio.
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
@@ -13,10 +13,19 @@ Requires-Dist: openfoodfacts>=2.9.0
 Requires-Dist: typer>=0.15.1
 Requires-Dist: google-cloud-batch==0.18.0
 Requires-Dist: huggingface-hub
+Requires-Dist: deepdiff>=8.6.1
+Requires-Dist: rapidfuzz>=3.14.3
+Requires-Dist: aiohttp
+Requires-Dist: aiofiles
+Requires-Dist: orjson
 Provides-Extra: ultralytics
 Requires-Dist: ultralytics==8.3.223; extra == "ultralytics"
 Provides-Extra: fiftyone
 Requires-Dist: fiftyone~=1.10.0; extra == "fiftyone"
+Provides-Extra: google
+Requires-Dist: google-genai>=1.56.0; extra == "google"
+Requires-Dist: gcloud-aio-storage; extra == "google"
+Requires-Dist: google-cloud-storage; extra == "google"
 Dynamic: license-file
 
 # Labelr
@@ -73,7 +82,7 @@ Once you have a Label Studio instance running, you can create a project easily.
 For an object detection task, a command allows you to create the configuration file automatically:
 
 ```bash
-labelr projects create-config --labels 'label1' --labels 'label2' --output-file label_config.xml
+labelr ls create-config --labels 'label1' --labels 'label2' --output-file label_config.xml
 ```
 
 where `label1` and `label2` are the labels you want to use for the object detection task, and `label_config.xml` is the output file that will contain the configuration.
@@ -81,17 +90,19 @@ where `label1` and `label2` are the labels you want to use for the object detect
 Then, you can create a project on Label Studio with the following command:
 
 ```bash
-labelr projects create --title my_project --api-key API_KEY --config-file label_config.xml
+labelr ls create --title my_project --api-key API_KEY --config-file label_config.xml
 ```
 
 where `API_KEY` is the API key of the Label Studio instance (API key is available at Account page), and `label_config.xml` is the configuration file of the project.
 
+`ls` stands for Label Studio in the CLI.
+
 #### Create a dataset file
 
 If you have a list of images, for an object detection task, you can quickly create a dataset file with the following command:
 
 ```bash
-labelr projects create-dataset-file --input-file image_urls.txt --output-file dataset.json
+labelr ls create-dataset-file --input-file image_urls.txt --output-file dataset.json
 ```
 
 where `image_urls.txt` is a file containing the URLs of the images, one per line, and `dataset.json` is the output file.
@@ -101,7 +112,7 @@ where `image_urls.txt` is a file containing the URLs of the images, one per line
 Next, import the generated data to a project with the following command:
 
 ```bash
-labelr projects import-data --project-id PROJECT_ID --dataset-path dataset.json
+labelr ls import-data --project-id PROJECT_ID --dataset-path dataset.json
 ```
 
 where `PROJECT_ID` is the ID of the project you created.
@@ -117,7 +128,7 @@ To accelerate annotation, you can pre-annotate the images with an object detecti
 To pre-annotate the data with Triton, use the following command:
 
 ```bash
-labelr projects add-prediction --project-id PROJECT_ID --backend ultralytics --labels 'product' --labels 'price tag' --label-mapping '{"price tag": "price-tag"}'
+labelr ls add-prediction --project-id PROJECT_ID --backend ultralytics --labels 'product' --labels 'price tag' --label-mapping '{"price tag": "price-tag"}'
 ```
 
 where `labels` is the list of labels to use for the object detection task (you can add as many labels as you want).

{labelr-0.7.0 → labelr-0.9.0}/README.md

@@ -52,7 +52,7 @@ Once you have a Label Studio instance running, you can create a project easily.
 For an object detection task, a command allows you to create the configuration file automatically:
 
 ```bash
-labelr projects create-config --labels 'label1' --labels 'label2' --output-file label_config.xml
+labelr ls create-config --labels 'label1' --labels 'label2' --output-file label_config.xml
 ```
 
 where `label1` and `label2` are the labels you want to use for the object detection task, and `label_config.xml` is the output file that will contain the configuration.
@@ -60,17 +60,19 @@ where `label1` and `label2` are the labels you want to use for the object detect
 Then, you can create a project on Label Studio with the following command:
 
 ```bash
-labelr projects create --title my_project --api-key API_KEY --config-file label_config.xml
+labelr ls create --title my_project --api-key API_KEY --config-file label_config.xml
 ```
 
 where `API_KEY` is the API key of the Label Studio instance (API key is available at Account page), and `label_config.xml` is the configuration file of the project.
 
+`ls` stands for Label Studio in the CLI.
+
 #### Create a dataset file
 
 If you have a list of images, for an object detection task, you can quickly create a dataset file with the following command:
 
 ```bash
-labelr projects create-dataset-file --input-file image_urls.txt --output-file dataset.json
+labelr ls create-dataset-file --input-file image_urls.txt --output-file dataset.json
 ```
 
 where `image_urls.txt` is a file containing the URLs of the images, one per line, and `dataset.json` is the output file.
@@ -80,7 +82,7 @@ where `image_urls.txt` is a file containing the URLs of the images, one per line
 Next, import the generated data to a project with the following command:
 
 ```bash
-labelr projects import-data --project-id PROJECT_ID --dataset-path dataset.json
+labelr ls import-data --project-id PROJECT_ID --dataset-path dataset.json
 ```
 
 where `PROJECT_ID` is the ID of the project you created.
@@ -96,7 +98,7 @@ To accelerate annotation, you can pre-annotate the images with an object detecti
 To pre-annotate the data with Triton, use the following command:
 
 ```bash
-labelr projects add-prediction --project-id PROJECT_ID --backend ultralytics --labels 'product' --labels 'price tag' --label-mapping '{"price tag": "price-tag"}'
+labelr ls add-prediction --project-id PROJECT_ID --backend ultralytics --labels 'product' --labels 'price tag' --label-mapping '{"price tag": "price-tag"}'
 ```
 
 where `labels` is the list of labels to use for the object detection task (you can add as many labels as you want).

{labelr-0.7.0 → labelr-0.9.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "labelr"
-version = "0.7.0"
+version = "0.9.0"
 description = "A command-line tool to manage labeling tasks with Label Studio."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -12,7 +12,12 @@ dependencies = [
     "openfoodfacts>=2.9.0",
     "typer>=0.15.1",
     "google-cloud-batch==0.18.0",
-    "huggingface-hub"
+    "huggingface-hub",
+    "deepdiff>=8.6.1",
+    "rapidfuzz>=3.14.3",
+    "aiohttp",
+    "aiofiles",
+    "orjson",
 ]
 
 [project.scripts]
@@ -25,6 +30,7 @@ ultralytics = [
 fiftyone = [
     "fiftyone~=1.10.0"
 ]
+google = ["google-genai >= 1.56.0", "gcloud-aio-storage", "google-cloud-storage"]
 
 [tool.uv]
 package = true

{labelr-0.7.0 → labelr-0.9.0}/src/labelr/apps/datasets.py

@@ -1,3 +1,6 @@
+"""Commands to manage datasets local datasets and export between platforms
+(Label Studio, HuggingFace Hub, local dataset,...)."""
+
 import json
 import random
 import shutil
@@ -21,45 +24,29 @@ logger = get_logger(__name__)
 
 @app.command()
 def check(
-    api_key: Annotated[
-        Optional[str], typer.Option(envvar="LABEL_STUDIO_API_KEY")
-    ] = None,
-    project_id: Annotated[
-        Optional[int], typer.Option(help="Label Studio Project ID")
-    ] = None,
-    label_studio_url: str = LABEL_STUDIO_DEFAULT_URL,
     dataset_dir: Annotated[
-        Optional[Path],
+        Path,
         typer.Option(
             help="Path to the dataset directory", exists=True, file_okay=False
         ),
-    ] = None,
+    ],
     remove: Annotated[
         bool,
-        typer.Option(
-            help="Remove duplicate images from the dataset, only for local datasets"
-        ),
+        typer.Option(help="Remove duplicate images from the dataset"),
     ] = False,
 ):
-    """Check a dataset for duplicate images."""
-    from label_studio_sdk.client import LabelStudio
+    """Check a local dataset in Ultralytics format for duplicate images."""
 
-    from ..check import check_local_dataset, check_ls_dataset
+    from ..check import check_local_dataset
 
-    if project_id is not None:
-        ls = LabelStudio(base_url=label_studio_url, api_key=api_key)
-        check_ls_dataset(ls, project_id)
-    elif dataset_dir is not None:
-        check_local_dataset(dataset_dir, remove=remove)
-    else:
-        raise typer.BadParameter("Either project ID or dataset directory is required")
+    check_local_dataset(dataset_dir, remove=remove)
 
 
 @app.command()
 def split_train_test(
     task_type: TaskType, dataset_dir: Path, output_dir: Path, train_ratio: float = 0.8
 ):
-    """Split a dataset into training and test sets.
+    """Split a local dataset into training and test sets.
 
     Only classification tasks are supported.
     """
@@ -112,7 +99,7 @@ def convert_object_detection_dataset(
     Studio format, and save it to a JSON file."""
     from datasets import load_dataset
 
-    from labelr.sample import format_object_detection_sample_from_hf
+    from labelr.sample import format_object_detection_sample_from_hf_to_ls
 
     logger.info("Loading dataset: %s", repo_id)
     ds = load_dataset(repo_id)
@@ -122,7 +109,7 @@ def convert_object_detection_dataset(
         for split in ds.keys():
             logger.info("Processing split: %s", split)
             for sample in ds[split]:
-                label_studio_sample = format_object_detection_sample_from_hf(
+                label_studio_sample = format_object_detection_sample_from_hf_to_ls(
                     sample, split=split
                 )
                 f.write(json.dumps(label_studio_sample) + "\n")

labelr-0.9.0/src/labelr/apps/evaluate.py

@@ -0,0 +1,41 @@
+from typing import Annotated
+
+import typer
+
+app = typer.Typer()
+
+
+@app.command()
+def visualize_object_detection(
+    hf_repo_id: Annotated[
+        str,
+        typer.Option(
+            ...,
+            help="Hugging Face repository ID of the trained model. "
+            "A `predictions.parquet` file is expected in the repo. Revision can be specified "
+            "by appending `@<revision>` to the repo ID.",
+        ),
+    ],
+    dataset_name: Annotated[
+        str | None, typer.Option(..., help="Name of the FiftyOne dataset to create.")
+    ] = None,
+    persistent: Annotated[
+        bool,
+        typer.Option(
+            ...,
+            help="Whether to make the FiftyOne dataset persistent (i.e., saved to disk).",
+        ),
+    ] = False,
+):
+    """Visualize object detection model predictions stored in a Hugging Face
+    repository using FiftyOne."""
+    from labelr.evaluate import object_detection
+
+    if dataset_name is None:
+        dataset_name = hf_repo_id.replace("/", "-").replace("@", "-")
+
+    object_detection.visualize(
+        hf_repo_id=hf_repo_id,
+        dataset_name=dataset_name,
+        persistent=persistent,
+    )

labelr-0.9.0/src/labelr/apps/google_batch.py

@@ -0,0 +1,289 @@
+import asyncio
+import importlib
+from pathlib import Path
+from typing import Annotated, Any
+
+import typer
+from google.genai.types import JSONSchema as GoogleJSONSchema
+from google.genai.types import Schema as GoogleSchema
+from openfoodfacts import Flavor
+from pydantic import BaseModel
+
+from labelr.google_genai import generate_batch_dataset, launch_batch_job
+
+app = typer.Typer()
+
+
+def convert_pydantic_model_to_google_schema(schema: type[BaseModel]) -> dict[str, Any]:
+    """Google doesn't support natively OpenAPI schemas, so we convert them to
+    Google `Schema` (a subset of OpenAPI)."""
+    return GoogleSchema.from_json_schema(
+        json_schema=GoogleJSONSchema.model_validate(schema.model_json_schema())
+    ).model_dump(mode="json", exclude_none=True, exclude_unset=True)
+
+
+@app.command()
+def generate_dataset(
+    data_path: Annotated[
+        Path,
+        typer.Option(
+            ...,
+            help="Path to a JSONL file containing the raw batch samples.",
+            exists=True,
+            dir_okay=False,
+            resolve_path=True,
+        ),
+    ],
+    output_path: Annotated[
+        Path,
+        typer.Option(
+            ...,
+            help="Path where to write the generated dataset file.",
+            exists=False,
+            dir_okay=False,
+            resolve_path=True,
+        ),
+    ],
+    config_module: Annotated[
+        str,
+        typer.Option(
+            ...,
+            help="Python module path (e.g., 'myschema') containing two variables: "
+            "OUTPUT_SCHEMA (a Pydantic class representing the output schema) and "
+            "INSTRUCTIONS (a str containing instructions to add before each sample).",
+        ),
+    ],
+    bucket_name: Annotated[
+        str,
+        typer.Option(
+            ...,
+            help="Name of the GCS bucket where the images are stored.",
+        ),
+    ] = "robotoff-batch",
+    bucket_dir_name: Annotated[
+        str,
+        typer.Option(
+            ...,
+            help="Directory name in the GCS bucket where the images are stored.",
+        ),
+    ] = "gemini-batch-images",
+    max_concurrent_uploads: Annotated[
+        int,
+        typer.Option(
+            ...,
+            help="Maximum number of concurrent uploads to GCS.",
+        ),
+    ] = 30,
+    base_image_dir: Annotated[
+        Path | None,
+        typer.Option(
+            ...,
+            help="Base directory to resolve local image paths from.",
+        ),
+    ] = None,
+    from_key: Annotated[
+        str | None,
+        typer.Option(
+            ...,
+            help="If specified, resume processing from this sample key.",
+        ),
+    ] = None,
+    skip_upload: Annotated[
+        bool, typer.Option(..., help="Skip uploading images to GCS")
+    ] = False,
+    thinking_level: Annotated[
+        str | None,
+        typer.Option(
+            ...,
+            help="Thinking level to use for the generation config.",
+        ),
+    ] = None,
+):
+    """Generate a dataset file in JSONL format to be used for batch
+    processing, using Gemini Batch Inference."""
+    typer.echo(f"Uploading images from '{data_path}' to GCS bucket '{bucket_name}'...")
+    typer.echo(f"Writing updated dataset to {output_path}...")
+    typer.echo(f"Max concurrent uploads: {max_concurrent_uploads}...")
+    typer.echo(f"Base image directory: {base_image_dir}...")
+    typer.echo(f"From key: {from_key}...")
+    typer.echo(f"Skip upload: {skip_upload}...")
+    typer.echo(f"Thinking level: {thinking_level}...")
+
+    module = importlib.import_module(config_module)
+    base_cls = getattr(module, "OUTPUT_SCHEMA")
+
+    if not issubclass(base_cls, BaseModel):
+        typer.echo(
+            f"Error: {config_module}.OUTPUT_SCHEMA is not a subclass of pydantic.BaseModel"
+        )
+        raise typer.Exit(code=1)
+
+    instructions = getattr(module, "INSTRUCTIONS", None) or None
+
+    if instructions:
+        typer.echo(f"Using instructions: '{instructions}'...")
+    else:
+        typer.echo("No instructions provided.")
+
+    # JSON Schema is supoorted natively by Vertex AI and Gemini APIs,
+    # but not yet on Batch Inference...
+    # So we convert the JSON schema to Google internal "Schema"
+    # google_json_schema = base_cls.model_json_schema()
+    google_json_schema = convert_pydantic_model_to_google_schema(base_cls)
+    asyncio.run(
+        generate_batch_dataset(
+            data_path=data_path,
+            output_path=output_path,
+            google_json_schema=google_json_schema,
+            instructions=instructions,
+            bucket_name=bucket_name,
+            bucket_dir_name=bucket_dir_name,
+            max_concurrent_uploads=max_concurrent_uploads,
+            base_image_dir=base_image_dir,
+            from_key=from_key,
+            skip_upload=skip_upload,
+            thinking_level=thinking_level,
+        )
+    )
+
+
+@app.command(name="launch-batch-job")
+def launch_batch_job_command(
+    run_name: Annotated[str, typer.Argument(..., help="Name of the batch job run")],
+    dataset_path: Annotated[Path, typer.Option(..., help="Path to the dataset file")],
+    model: Annotated[str, typer.Option(..., help="Model to use for the batch job")],
+    location: Annotated[
+        str,
+        typer.Option(..., help="GCP location where to run the batch job"),
+    ] = "europe-west4",
+):
+    """Launch a Gemini Batch Inference job."""
+    launch_batch_job(
+        run_name=run_name,
+        dataset_path=dataset_path,
+        model=model,
+        location=location,
+    )
+
+
+@app.command()
+def upload_training_dataset_from_predictions(
+    prediction_path: Annotated[
+        Path,
+        typer.Argument(
+            ...,
+            help="Path to the prediction JSONL file generated by Google Inference Batch",
+            exists=True,
+            dir_okay=False,
+            readable=True,
+        ),
+    ],
+    instructions_path: Annotated[
+        Path,
+        typer.Option(
+            ...,
+            help="Path to the file with the instruction prompt for the model",
+            exists=True,
+            dir_okay=False,
+            readable=True,
+        ),
+    ],
+    json_schema_path: Annotated[
+        Path,
+        typer.Option(
+            ...,
+            help="Path to the file with the JSON schema to follow",
+            dir_okay=False,
+            readable=True,
+        ),
+    ],
+    repo_id: Annotated[
+        str, typer.Option(help="Hugging Face Datasets repository ID to push to")
+    ],
+    revision: Annotated[
+        str,
+        typer.Option(
+            help="Revision (branch, tag or commit) to use for the Hugging Face Datasets repository"
+        ),
+    ] = "main",
+    is_openfoodfacts_dataset: Annotated[
+        bool, typer.Option(..., help="Whether this is an Open Food Facts dataset")
+    ] = False,
+    openfoodfacts_flavor: Annotated[
+        Flavor,
+        typer.Option(
+            ...,
+            help="Open Food Facts flavor of the dataset (if applicable)",
+        ),
+    ] = Flavor.off,
+    split: Annotated[str, typer.Option(..., help="Name of the split")] = "train",
+    tmp_dir: Annotated[
+        Path | None,
+        typer.Option(
+            ...,
+            help="Temporary directory to use for intermediate files, default to a temporary directory "
+            "generated automatically. This is useful to relaunch the command if it fails midway.",
+        ),
+    ] = None,
+    skip: Annotated[int, typer.Option(..., help="Number of samples to skip")] = 0,
+    limit: Annotated[
+        int | None,
+        typer.Option(
+            ..., help="Limit number of samples to process, or None for no limit"
+        ),
+    ] = None,
+    raise_on_invalid_sample: Annotated[
+        bool,
+        typer.Option(
+            ...,
+            help="Whether to raise an error on invalid samples instead of skipping them",
+        ),
+    ] = False,
+):
+    """Upload a training dataset to a Hugging Face Datasets repository from a
+    Gemini batch prediction file."""
+    import tempfile
+
+    import orjson
+    from huggingface_hub import HfApi
+
+    from labelr.export import export_to_hf_llm_image_extraction
+    from labelr.google_genai import generate_sample_iter
+
+    instructions = instructions_path.read_text()
+    print(f"Instructions: {instructions}")
+    json_schema = orjson.loads(json_schema_path.read_text())
+
+    api = HfApi()
+    config = {
+        "instructions": instructions,
+        "json_schema": json_schema,
+    }
+    with tempfile.TemporaryDirectory() as config_tmp_dir_str:
+        config_tmp_dir = Path(config_tmp_dir_str)
+        config_path = config_tmp_dir / "config.json"
+        config_path.write_text(
+            orjson.dumps(config, option=orjson.OPT_INDENT_2).decode("utf-8")
+        )
+        api.upload_file(
+            path_or_fileobj=config_path,
+            path_in_repo="config.json",
+            repo_id=repo_id,
+            repo_type="dataset",
+        )
+    sample_iter = generate_sample_iter(
+        prediction_path=prediction_path,
+        json_schema=json_schema,
+        is_openfoodfacts_dataset=is_openfoodfacts_dataset,
+        openfoodfacts_flavor=openfoodfacts_flavor,
+        skip=skip,
+        limit=limit,
+        raise_on_invalid_sample=raise_on_invalid_sample,
+    )
+    export_to_hf_llm_image_extraction(
+        sample_iter=sample_iter,
+        split=split,
+        repo_id=repo_id,
+        revision=revision,
+        tmp_dir=tmp_dir,
+    )

labelr-0.9.0/src/labelr/apps/hugging_face.py

@@ -0,0 +1,57 @@
+from pathlib import Path
+from typing import Annotated
+
+import typer
+
+app = typer.Typer()
+
+
+@app.command()
+def show_hf_sample(
+    repo_id: Annotated[
+        str,
+        typer.Argument(
+            ...,
+            help="Hugging Face Datasets repo ID. The revision can be specified by "
+            "appending `@<revision>` to the repo ID.",
+        ),
+    ],
+    image_id: Annotated[
+        str,
+        typer.Argument(
+            ...,
+            help="ID of the image associated with the sample to display (field: `image_id`)",
+        ),
+    ],
+    output_image_path: Annotated[
+        Path | None,
+        typer.Option(help="Path to save the sample image (optional)", exists=False),
+    ] = None,
+):
+    """Display a sample from a Hugging Face Datasets repository by image ID."""
+    from labelr.utils import parse_hf_repo_id
+
+    repo_id, revision = parse_hf_repo_id(repo_id)
+
+    from datasets import load_dataset
+
+    ds = load_dataset(repo_id, revision=revision)
+
+    sample = None
+    for split in ds.keys():
+        samples = ds[split].filter(lambda x: x == image_id, input_columns="image_id")
+        if len(samples) > 0:
+            sample = samples[0]
+            break
+    if sample is None:
+        typer.echo(f"Sample with image ID {image_id} not found in dataset {repo_id}")
+        raise typer.Exit(code=1)
+
+    else:
+        for key, value in sample.items():
+            typer.echo(f"{key}: {value}")
+
+        if output_image_path is not None:
+            image = sample["image"]
+            image.save(output_image_path)
+            typer.echo(f"Image saved to {output_image_path}")