hafnia 0.1.25__py3-none-any.whl → 0.1.26__py3-none-any.whl

{hafnia-0.1.25.dist-info → hafnia-0.1.26.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hafnia
-Version: 0.1.25
+Version: 0.1.26
 Summary: Python SDK for communication with Hafnia platform.
 Author-email: Milestone Systems <hafniaplatform@milestone.dk>
 License-File: LICENSE
@@ -17,10 +17,6 @@ Requires-Dist: pydantic>=2.10.4
 Requires-Dist: rich>=13.9.4
 Requires-Dist: seedir>=0.5.0
 Requires-Dist: tqdm>=4.67.1
-Provides-Extra: torch
-Requires-Dist: flatten-dict>=0.4.2; extra == 'torch'
-Requires-Dist: torch>=2.6.0; extra == 'torch'
-Requires-Dist: torchvision>=0.21.0; extra == 'torch'
 Description-Content-Type: text/markdown
 
 # Hafnia
@@ -91,14 +87,117 @@ and explore the dataset sample with a python script:
 from hafnia.data import load_dataset
 
 dataset_splits = load_dataset("mnist")
-print(dataset_splits)
-print(dataset_splits["train"])
 ```
+
+### Dataset Format
 The returned sample dataset is a [hugging face dataset](https://huggingface.co/docs/datasets/index) 
 and contains train, validation and test splits. 
 
+```python
+print(dataset_splits)
+
+# Output:
+>>> DatasetDict({
+    train: Dataset({
+        features: ['image_id', 'image', 'height', 'width', 'objects', 'Weather', 'Surface Conditions'],
+        num_rows: 172
+    })
+    validation: Dataset({
+        features: ['image_id', 'image', 'height', 'width', 'objects', 'Weather', 'Surface Conditions'],
+        num_rows: 21
+    })
+    test: Dataset({
+        features: ['image_id', 'image', 'height', 'width', 'objects', 'Weather', 'Surface Conditions'],
+        num_rows: 21
+    })
+})
+
+```
+
+A Hugging Face dataset is a dictionary with splits, where each split is a `Dataset` object.
+Each `Dataset` is structured as a table with a set of columns (also called features) and a row for each sample.
+
+The features of the dataset can be viewed with the `features` attribute.
+```python
+# View features of the train split
+pprint.pprint(dataset["train"].features)
+{'Surface Conditions': ClassLabel(names=['Dry', 'Wet'], id=None),
+ 'Weather': ClassLabel(names=['Clear', 'Foggy'], id=None),
+ 'height': Value(dtype='int64', id=None),
+ 'image': Image(mode=None, decode=True, id=None),
+ 'image_id': Value(dtype='int64', id=None),
+ 'objects': Sequence(feature={'bbox': Sequence(feature=Value(dtype='int64',
+                                                             id=None),
+                                               length=-1,
+                                               id=None),
+                              'class_idx': ClassLabel(names=['Vehicle.Bicycle',
+                                                             'Vehicle.Motorcycle',
+                                                             'Vehicle.Car',
+                                                             'Vehicle.Van',
+                                                             'Vehicle.RV',
+                                                             'Vehicle.Single_Truck',
+                                                             'Vehicle.Combo_Truck',
+                                                             'Vehicle.Pickup_Truck',
+                                                             'Vehicle.Trailer',
+                                                             'Vehicle.Emergency_Vehicle',
+                                                             'Vehicle.Bus',
+                                                             'Vehicle.Heavy_Duty_Vehicle'],
+                                                      id=None),
+                              'class_name': Value(dtype='string', id=None),
+                              'id': Value(dtype='string', id=None)},
+                     length=-1,
+                     id=None),
+ 'width': Value(dtype='int64', id=None)}
+```
+
+View the first sample in the training set:
+```python
+# Print sample from the training set
+pprint.pprint(dataset["train"][0])
+
+{'image': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=1920x1080 at 0x79D6292C5ED0>,
+ 'image_id': 4920,
+ 'height': 1080,
+ 'Weather': 0,
+ 'Surface Conditions': 0,
+ 'objects': {'bbox': [[441, 180, 121, 126],
+                      [549, 151, 131, 103],
+                      [1845, 722, 68, 130],
+                      [1810, 571, 110, 149]],
+             'class_idx': [7, 7, 2, 2],
+             'class_name': ['Vehicle.Pickup_Truck',
+                            'Vehicle.Pickup_Truck',
+                            'Vehicle.Car',
+                            'Vehicle.Car'],
+             'id': ['HW6WiLAJ', 'T/ccFpRi', 'CS0O8B6W', 'DKrJGzjp']},
+ 'width': 1920}
+
+```
+
+For hafnia based datasets, we want to standardized how a dataset and dataset tasks are represented.
+We have defined a set of features that are common across all datasets in the Hafnia data library.
+
+- `image`: The image itself, stored as a PIL image
+- `height`: The height of the image in pixels
+- `width`: The width of the image in pixels
+- `[IMAGE_CLASSIFICATION_TASK]`: [Optional] Image classification tasks are top-level `ClassLabel` feature. 
+  `ClassLabel` is a Hugging Face feature that maps class indices to class names. 
+  In above example we have two classification tasks:
+  - `Weather`: Classifies the weather conditions in the image, with possible values `Clear` and `Foggy`
+  - `Surface Conditions`: Classifies the surface conditions in the image, with possible values `Dry` and `Wet`
+- `objects`: A dictionary containing information about objects in the image, including:
+  - `bbox`: Bounding boxes for each object, represented with a list of bounding box coordinates 
+  `[xmin, ymin, bbox_width, bbox_height]`. Each bounding box is defined with a top-left corner coordinate 
+  `(xmin, ymin)` and bounding box width and height `(bbox_width, bbox_height)` in pixels.
+  - `class_idx`: Class indices for each detected object. This is a
+  `ClassLabel` feature that maps to the `class_name` feature.
+  - `class_name`: Class names for each detected object
+  - `id`: Unique identifiers for each detected object
+
+### Dataset Locally vs. Training-aaS
 An important feature of `load_dataset` is that it will return the full dataset 
-when loaded on the Hafnia platform. 
+when loaded with Training-aaS on the Hafnia platform. 
+
 This enables seamlessly switching between running/validating a training script 
 locally (on the sample dataset) and running full model trainings with Training-aaS (on the full dataset). 
 without changing code or configurations for the training script.
@@ -160,12 +259,58 @@ with a dataloader that performs data augmentations and batching of the dataset a
 To support this, we have provided a torch dataloader example script
 [example_torchvision_dataloader.py](./examples/example_torchvision_dataloader.py). 
 
-The script demonstrates how to make a dataloader with data augmentation (`torchvision.transforms.v2`)
-and a helper function for visualizing image and labels. 
+The script demonstrates how to load a dataset sample, apply data augmentations using
+`torchvision.transforms.v2`, and visualize the dataset with `torch_helpers.draw_image_and_targets`.
+
+Note also how `torch_helpers.TorchVisionCollateFn` is used in combination with the `DataLoader` from 
+`torch.utils.data` to handle the dataset's collate function.
 
 The dataloader and visualization function supports computer vision tasks 
 and datasets available in the data library. 
 
+```python
+# Load Hugging Face dataset
+dataset_splits = load_dataset("midwest-vehicle-detection")
+
+# Define transforms
+train_transforms = v2.Compose(
+    [
+        v2.RandomResizedCrop(size=(224, 224), antialias=True),
+        v2.RandomHorizontalFlip(p=0.5),
+        v2.ToDtype(torch.float32, scale=True),
+        v2.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
+    ]
+)
+test_transforms = v2.Compose(
+    [
+        v2.Resize(size=(224, 224), antialias=True),
+        v2.ToDtype(torch.float32, scale=True),
+        v2.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
+    ]
+)
+
+keep_metadata = True
+train_dataset = torch_helpers.TorchvisionDataset(
+    dataset_splits["train"], transforms=train_transforms, keep_metadata=keep_metadata
+)
+test_dataset = torch_helpers.TorchvisionDataset(
+    dataset_splits["test"], transforms=test_transforms, keep_metadata=keep_metadata
+)
+
+# Visualize sample
+image, targets = train_dataset[0]
+visualize_image = torch_helpers.draw_image_and_targets(image=image, targets=targets)
+pil_image = torchvision.transforms.functional.to_pil_image(visualize_image)
+pil_image.save("visualized_labels.png")
+
+# Create DataLoaders - using TorchVisionCollateFn
+collate_fn = torch_helpers.TorchVisionCollateFn(
+    skip_stacking=["objects.bbox", "objects.class_idx"]
+)
+train_loader = DataLoader(train_dataset, batch_size=8, shuffle=True, collate_fn=collate_fn)
+```
+
+
 ## Example: Training-aaS
 By combining logging and dataset loading, we can now construct our model training recipe. 
 
@@ -206,10 +351,10 @@ Install uv
 curl -LsSf https://astral.sh/uv/install.sh | sh
 ```
 
-Install python dependencies including developer (`--dev`) and optional dependencies (`--all-extras`).
+Create virtual environment and install python dependencies
 
 ```bash
-uv sync --all-extras --dev
+uv sync
 ```
 
  Run tests:

hafnia-0.1.26.dist-info/RECORD

@@ -0,0 +1,27 @@
+cli/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+cli/__main__.py,sha256=MX0RT1BP3t59rzCvdUqfw39Kw05HOF4OEtjDTwIU9h8,1594
+cli/config.py,sha256=R9w0NKIOtIxRKNs7ieeUrIKwRkrTlK5PqOVjc5VYljE,4923
+cli/consts.py,sha256=sj0MRwbbCT2Yl77FPddck1VWkFxp7QY6I9l1o75j_aE,963
+cli/data_cmds.py,sha256=BQiythAPwAwudgdUa68v50a345uw5flrcDiBHLGp9lo,1460
+cli/experiment_cmds.py,sha256=L-k_ZJ4B7I4cA8OvHcheSwXM6nx9aTF9G7eKBzAcOzQ,1961
+cli/profile_cmds.py,sha256=-HQcFgYI6Rqaefi0Nj-91KhiqPKUj7zOaiJWbHx_bac,3196
+cli/recipe_cmds.py,sha256=qnMfF-te47HXNkgyA0hm9X3etDQsqMnrVEGDCrzVjZU,1462
+cli/runc_cmds.py,sha256=6fHMi_dEd8g3Cx9PEfU4gJMZf5-G0IUPDcZh6DNq8Mw,4953
+hafnia/__init__.py,sha256=Zphq-cQoX95Z11zm4lkrU-YiAJxddR7IBfwDkxeHoDE,108
+hafnia/http.py,sha256=psCWdNKfKYiBrYD6bezat1AeHh77JJtJrPePiUAjTIk,2948
+hafnia/log.py,sha256=sWF8tz78yBtwZ9ddzm19L1MBSBJ3L4G704IGeT1_OEU,784
+hafnia/torch_helpers.py,sha256=P_Jl4IwqUebKVCOXNe6iTorJZA3S-3d92HV274UHIko,7456
+hafnia/utils.py,sha256=mJ5aOjSVSOrrQnpmaKLK71ld5jYpmtd3HciTIk_Wk88,4659
+hafnia/data/__init__.py,sha256=Pntmo_1fst8OhyrHB60jQ8mhJJ4hL38tdjLvt0YXEJo,73
+hafnia/data/factory.py,sha256=4fZDkWNyOK1QNCmsxsXfSztPJkJW_HBIa_PTdGCYHCM,2551
+hafnia/experiment/__init__.py,sha256=OEFE6HqhO5zcTCLZcPcPVjIg7wMFFnvZ1uOtAVhRz7M,85
+hafnia/experiment/hafnia_logger.py,sha256=usL5pl7XLJP-g1vZrwvbky5YiD6Bg-rOODYYAX5z43I,6830
+hafnia/platform/__init__.py,sha256=I-VIVXDxwBAUzxx8Zx0g_wykyDdFGTsjb_mYLmvxk2Y,443
+hafnia/platform/builder.py,sha256=OFPnOjE3bAbWjUgYErXtffDKTiW_9ol95eVzKqL27WM,5433
+hafnia/platform/download.py,sha256=t055axPNHlXTYCQgZHOS2YMQt1I2_bc4G8dltsOKttY,4760
+hafnia/platform/experiment.py,sha256=-nAfTmn1c8sE6pHDCTNZvWDTopkXndarJAPIGvsnk60,2389
+hafnia-0.1.26.dist-info/METADATA,sha256=Lds8gx_ffd8_l9kByvK_e-HPFehSSUv8E_85d8ZelSE,14990
+hafnia-0.1.26.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
+hafnia-0.1.26.dist-info/entry_points.txt,sha256=FCJVIQ8GP2VE9I3eeGVF5eLxVDNW_01pOJCpG_CGnMM,45
+hafnia-0.1.26.dist-info/licenses/LICENSE,sha256=wLZw1B7_mod_CO1H8LXqQgfqlWD6QceJR8--LJYRZGE,1078
+hafnia-0.1.26.dist-info/RECORD,,

hafnia/platform/api.py

@@ -1,12 +0,0 @@
-import urllib3
-
-from hafnia.http import fetch
-
-
-def get_organization_id(endpoint: str, api_key: str) -> str:
-    headers = {"X-APIKEY": api_key}
-    try:
-        org_info = fetch(endpoint, headers=headers)
-    except urllib3.exceptions.HTTPError as e:
-        raise ValueError("Failed to fetch organization ID. Verify platform URL and API key.") from e
-    return org_info[0]["id"]

hafnia/platform/executor.py

@@ -1,111 +0,0 @@
-import os
-import subprocess
-import sys
-from dataclasses import dataclass
-from pathlib import Path
-from typing import Dict
-
-from hafnia.log import logger
-
-
-@dataclass
-class PythonModule:
-    """Dataclass to store Python module details."""
-
-    module_name: str
-    runner_path: str
-
-
-def handle_mount(source: str) -> None:
-    """
-    Mounts the Hafnia environment by adding source directories to PYTHONPATH.
-
-    Args:
-        source (str): Path to the root directory containing 'src' and 'scripts' subdirectories
-
-    Raises:
-        FileNotFoundError: If the required directory structure is not found
-    """
-    source_path = Path(source)
-    src_dir = source_path / "src"
-    scripts_dir = source_path / "scripts"
-
-    if not src_dir.exists() and not scripts_dir.exists():
-        logger.error(f"Filestructure is not supported. Expected 'src' and 'scripts' directories in {source_path}.")
-        exit(1)
-
-    sys.path.extend([src_dir.as_posix(), scripts_dir.as_posix()])
-    python_path = os.getenv("PYTHONPATH", "")
-    os.environ["PYTHONPATH"] = f"{python_path}:{src_dir.as_posix()}:{scripts_dir.as_posix()}"
-    logger.info(f"Mounted codebase from {source_path}")
-
-
-def collect_python_modules(directory: Path) -> Dict[str, PythonModule]:
-    """
-    Collects Python modules from a directory and its subdirectories.
-
-    This function dynamically imports Python modules found in the specified directory,
-    excluding files that start with '_' or '.'. It's used to discover available tasks
-    in the Hafnia environment.
-
-    Args:
-        directory (Path): The directory to search for Python modules
-
-    Returns:
-        Dict[str, Dict[str, str]]: A dictionary mapping task names to module details, where each detail contains:
-            - module_name (str): The full module name
-            - runner_path (str): The absolute path to the module file
-    """
-    from importlib.util import module_from_spec, spec_from_file_location
-
-    modules = {}
-    for fname in directory.rglob("*.py"):
-        if fname.name.startswith("-"):
-            continue
-
-        task_name = fname.stem
-        module_name = f"{directory.name}.{task_name}"
-
-        spec = spec_from_file_location(module_name, fname)
-        if spec is None:
-            logger.warning(f"Was not able to load {module_name} from {fname}")
-            continue
-        if spec.loader is None:
-            logger.warning(f"Loader is None for {module_name} from {fname}")
-            continue
-        module = module_from_spec(spec)
-        spec.loader.exec_module(module)
-
-        modules[task_name] = PythonModule(module_name, str(fname.resolve()))
-
-    return modules
-
-
-def handle_launch(task: str) -> None:
-    """
-    Launch and execute a specified Hafnia task.
-
-    This function verifies the Hafnia environment status, locates the task script,
-    and executes it in a subprocess with output streaming.
-
-    Args:
-        task (str): Name of the task to execute
-
-    Raises:
-        ValueError: If the task is not found or scripts directory is not in PYTHONPATH
-    """
-    recipe_dir = os.getenv("RECIPE_DIR", None)
-    if recipe_dir is None:
-        raise ValueError("RECIPE_DIR environment variable not set.")
-    handle_mount(recipe_dir)
-    scripts_dir = [p for p in sys.path if "scripts" in p][0]
-    scripts = collect_python_modules(Path(scripts_dir))
-    if task not in scripts:
-        available_tasks = ", ".join(sorted(scripts.keys()))
-        logger.error(f"Task '{task}' not found. Available tasks: {available_tasks}")
-        exit(1)
-    try:
-        subprocess.check_call(["python", scripts[task].runner_path], stdout=sys.stdout, stderr=sys.stdout)
-    except subprocess.CalledProcessError as e:
-        logger.error(f"Error executing task: {str(e)}")
-        exit(1)

hafnia-0.1.25.dist-info/RECORD

@@ -1,29 +0,0 @@
-cli/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-cli/__main__.py,sha256=8JgZHtFpWAOUlEFvV0YWviLwesWSA-PTYH_v9COl2xw,1786
-cli/config.py,sha256=Js_dCn39l7hLhA3ovHorOyVqj-LCLzUg_figSy4jNjs,5279
-cli/consts.py,sha256=ybpWMhjkrqqevL7eVmYtdn_13a5-bV_5lCpA6_Wzcz0,964
-cli/data_cmds.py,sha256=BQiythAPwAwudgdUa68v50a345uw5flrcDiBHLGp9lo,1460
-cli/experiment_cmds.py,sha256=nJCnI0kzmFJ1_vmxIzOYWk_2eiiw1Ub0j02jXi2vW_s,2239
-cli/profile_cmds.py,sha256=Rg-5wLHSWlZhNPUZBO7LdyJS-Y-SgI6qKLoAac2gSdk,2534
-cli/recipe_cmds.py,sha256=TnUAoO643NeSio8akVUEJHs6Ttuu2JuprxyTPqzzb4k,1592
-cli/runc_cmds.py,sha256=6qvVfjxQ_1nkm7lrrIzYETdnBzfiXrmdnWo4jpbbdPk,4830
-hafnia/__init__.py,sha256=Zphq-cQoX95Z11zm4lkrU-YiAJxddR7IBfwDkxeHoDE,108
-hafnia/http.py,sha256=rID6Krn9wRGXwsJYvpffsFlt5cwxFgkcihYppqtdT-8,2974
-hafnia/log.py,sha256=ii--Q6IThsWOluRp_Br9WGhwBtKChU80BXk5pK_NU5A,819
-hafnia/torch_helpers.py,sha256=P_Jl4IwqUebKVCOXNe6iTorJZA3S-3d92HV274UHIko,7456
-hafnia/utils.py,sha256=jLq2S8n7W4HS7TsXnDgxTze463Mcatd_wC6pd54a7Os,4221
-hafnia/data/__init__.py,sha256=Pntmo_1fst8OhyrHB60jQ8mhJJ4hL38tdjLvt0YXEJo,73
-hafnia/data/factory.py,sha256=scsXrAHlBEP16AJH8RyQ1fyzhei5GxIwsmMgwEru3Pc,2536
-hafnia/experiment/__init__.py,sha256=OEFE6HqhO5zcTCLZcPcPVjIg7wMFFnvZ1uOtAVhRz7M,85
-hafnia/experiment/hafnia_logger.py,sha256=8baV6SUtCVIijypU-FfgAOIyWIf_eeJ5a62oFzQesmc,6794
-hafnia/platform/__init__.py,sha256=Oz1abs40hEKspLg6mVIokdtsp1tZJF9Pndv8uSMOgtQ,522
-hafnia/platform/api.py,sha256=aJvlQGjzqm-D3WYb2xTEYX60YoJoWN_kyYdlkvqt_MI,382
-hafnia/platform/builder.py,sha256=6xLy64a4cytMZEfqiA0kPzxiATEBbHXmDbf7igTMAiM,6595
-hafnia/platform/download.py,sha256=AWnlSYj9FD7GvZ_-9Sw5jrcxi3RyBSSUVph8U9T9ZbQ,4711
-hafnia/platform/executor.py,sha256=8E6cGmEMr5xYb3OReBuWj8ZnVXc0Es0UkfPamsmjH4g,3759
-hafnia/platform/experiment.py,sha256=951ppXdrp075pW2xGFOM0oiGYGE1I53tP9azQjjIUe8,2305
-hafnia-0.1.25.dist-info/METADATA,sha256=Q5dBhUXq-6lgaIVwR2ndWPsF7GFu4m8-G7dIjcW0iug,8660
-hafnia-0.1.25.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
-hafnia-0.1.25.dist-info/entry_points.txt,sha256=FCJVIQ8GP2VE9I3eeGVF5eLxVDNW_01pOJCpG_CGnMM,45
-hafnia-0.1.25.dist-info/licenses/LICENSE,sha256=wLZw1B7_mod_CO1H8LXqQgfqlWD6QceJR8--LJYRZGE,1078
-hafnia-0.1.25.dist-info/RECORD,,