hafnia 0.1.9__tar.gz → 0.1.12__tar.gz

hafnia-0.1.12/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Data-insight-Platform
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

hafnia-0.1.12/PKG-INFO

@@ -0,0 +1,197 @@
+Metadata-Version: 2.4
+Name: hafnia
+Version: 0.1.12
+Summary: Python tools for communication with Hafnia platform.
+Author-email: Ivan Sahumbaiev <ivsa@milestone.dk>
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: boto3>=1.35.91
+Requires-Dist: click>=8.1.8
+Requires-Dist: datasets>=3.2.0
+Requires-Dist: pillow>=11.1.0
+Requires-Dist: pyarrow>=18.1.0
+Requires-Dist: pydantic>=2.10.4
+Requires-Dist: rich>=13.9.4
+Requires-Dist: tqdm>=4.67.1
+Provides-Extra: torch
+Requires-Dist: flatten-dict>=0.4.2; extra == 'torch'
+Requires-Dist: torch>=2.6.0; extra == 'torch'
+Requires-Dist: torchvision>=0.21.0; extra == 'torch'
+Description-Content-Type: text/markdown
+
+# Hafnia
+
+The `hafnia` python package is a collection of tools to create and run model training recipes on
+the [Hafnia Platform](https://hafnia.milestonesys.com/). 
+
+The package includes the following interfaces: 
+
+- `cli`: A Command Line Interface (CLI) to 1) configure/connect to Hafnia and 2) create and 
+launch  [Training-aaS](https://hafnia.readme.io/docs/training-as-a-service) recipe scripts.
+- `hafnia`: A python package with helper functions to load and interact with sample datasets and an experiment
+ tracker (`HafniaLogger`). 
+
+
+## The Concept: Training as a Service (Training-aaS)
+`Training-aaS` is the concept of training models on the Hafnia platform on large 
+and *hidden* datasets. Hidden datasets refers to datasets that can be used for 
+training, but are not available for download or direct access. 
+
+This is a key feature of the Hafnia platform, as a hidden dataset ensures data 
+privacy, and allow models to be trained compliantly and ethically by third parties (you).
+
+The `script2model` approach is a Training-aaS concept, where you package your custom training 
+script as a *training recipe* and use the recipe to train models on the hidden datasets.
+
+To support local development of a training recipe, we have introduced a **sample dataset** 
+for each dataset available on the Hafnia platform. The sample dataset is a small 
+and anonymized subset of the full dataset and available for download. 
+
+With the sample dataset, you can seamlessly switch between local and Hafnia training. 
+Locally, you can create, validate and debug your training recipe. The recipe is then 
+launched with Hafnia Training-aaS, where the recipe runs on the full dataset and can be scaled to run on
+multiple GPUs and instances if needed. 
+
+
+## Getting started: Configuration
+To get started with Hafnia: 
+
+1. Install `hafnia` with your favorite python package manager. With pip do this:
+
+    `pip install hafnia`
+1. Sign in to the [Hafnia Platform](https://hafnia.milestonesys.com/). 
+1. Create an API KEY for Training aaS. For more instructions, follow this 
+[guide](https://hafnia.readme.io/docs/create-an-api-key). 
+Copy the key and save it for later use.
+1. From terminal, configure your machine to access Hafnia: 
+
+    ```
+    # Start configuration with
+    hafnia configure
+
+    # You are then prompted: 
+    Profile Name [default]:   # Press [Enter] or select an optional name
+    Hafnia API Key:  # Pass your HAFNIA API key
+    Hafnia Platform URL [https://api.mdi.milestonesys.com]:  # Press [Enter]
+    ```
+1. Download `mnist` from terminal to verify configuration is working.  
+
+    ```bash
+    hafnia data download mnist --force
+    ```
+
+## Getting started: Loading datasets samples
+With Hafnia configured on your local machine, it is now possible to download 
+and explore the dataset sample with a python script:
+
+```python
+from hafnia.data import load_dataset
+
+dataset_splits = load_dataset("midwest-vehicle-detection")
+print(dataset_splits)
+print(dataset_splits["train"])
+```
+
+Datasets with corresponding sample datasets can be found in [data library](https://hafnia.milestonesys.com/training-aas/datasets). It is early days for the data library, 
+but we are actively working on adding more datasets.
+
+The returned sample dataset is a [hugging face dataset](https://huggingface.co/docs/datasets/index) 
+and contains train, validation and test splits. 
+
+An important feature of `load_dataset` is that it will return the full dataset 
+when loaded on the Hafnia platform. 
+
+This enables seamlessly switching between running/validating a training script 
+locally (on the sample dataset) and running full model trainings in the cloud 
+without changing code or configurations for the training script.
+
+## Getting started: Experiment Tracking with HafniaLogger
+The `HafniaLogger` is an important part of the recipe script and enables you to track, log and
+reproduce your experiments.
+
+When integrated into your training script, the `HafniaLogger` is responsible for collecting:
+
+- **Trained Model**: The model trained during the experiment
+- **Model Checkpoints**: Intermediate model states saved during training
+- **Experiment Configurations**: Hyperparameters and other settings used in your experiment
+- **Training/Evaluation Metrics**: Performance data such as loss values, accuracy, and custom metrics
+
+### Basic Implementation Example
+
+Here's how to integrate the `HafniaLogger` into your training script:
+
+```python
+from hafnia.experiment import HafniaLogger
+
+batch_size = 128
+learning_rate = 0.001
+
+# Initialize Hafnia logger
+logger = HafniaLogger()
+
+# Log experiment parameters
+logger.log_configuration({"batch_size": 128, "learning_rate": 0.001})
+
+# Store checkpoints in this path
+ckpt_dir = logger.path_model_checkpoints()
+
+# Store the trained model in this path
+model_dir = logger.path_model()
+
+# Log scalar and metric values during training and validation
+logger.log_scalar("train/loss", value=0.1, step=100)
+logger.log_metric("train/accuracy", value=0.98, step=100)
+
+logger.log_scalar("validation/loss", value=0.1, step=100)
+logger.log_metric("validation/accuracy", value=0.95, step=100)
+```
+
+Similar to `load_dataset`, the tracker behaves differently when running locally or in the cloud. 
+Locally, experiment data is stored in a local folder `.data/experiments/{DATE_TIME}`. 
+
+In the cloud, the experiment data will be available in the Hafnia platform under 
+[experiments](https://hafnia.milestonesys.com/training-aas/experiments). 
+
+## Example: Torch Dataloader
+Commonly for `torch`-based training scripts, a dataset is used in combination 
+with a dataloader that performs data augmentations and batching of the dataset as torch tensors.
+
+To support this, we have provided a torch dataloader example script
+[example_torchvision_dataloader.py](./examples/example_torchvision_dataloader.py). 
+
+The script demonstrates how to make a dataloader with data augmentation (`torchvision.transforms.v2`)
+and a helper function for visualizing image and labels. 
+
+The dataloader and visualization function supports computer vision tasks 
+and datasets available in the data library. 
+
+## Example: Training-aaS
+By combining logging and dataset loading, we can now construct our model training recipe. 
+
+To demonstrate this, we have provided a recipe project that serves as a template for creating and structuring training recipes
+[recipe-classification](https://github.com/Data-insight-Platform/recipe-classification)
+
+The project also contains additional information on how to structure your training recipe, use the `HafniaLogger`, the `load_dataset` function and different approach for launching 
+the training recipe on the Hafnia platform.
+
+## Detailed Documentation
+For more information, go to our [documentation page](https://hafnia.readme.io/docs/welcome-to-hafnia) 
+or in below markdown pages. 
+
+- [CLI](docs/cli.md) - Detailed guide for the Hafnia command-line interface
+- [Script2Model Documentation](docs/s2m.md) - Detailed guide for script2model
+- [Release lifecycle](docs/release.md) - Details about package release lifecycle.
+
+## Development
+For development, we are using an uv based virtual python environment
+
+Install uv
+
+    curl -LsSf https://astral.sh/uv/install.sh | sh
+
+
+Install python dependencies including developer (`--dev`) and optional dependencies (`--all-extras`).
+
+    uv sync --all-extras --dev
+
+ 

hafnia-0.1.12/README.md

@@ -0,0 +1,176 @@
+# Hafnia
+
+The `hafnia` python package is a collection of tools to create and run model training recipes on
+the [Hafnia Platform](https://hafnia.milestonesys.com/). 
+
+The package includes the following interfaces: 
+
+- `cli`: A Command Line Interface (CLI) to 1) configure/connect to Hafnia and 2) create and 
+launch  [Training-aaS](https://hafnia.readme.io/docs/training-as-a-service) recipe scripts.
+- `hafnia`: A python package with helper functions to load and interact with sample datasets and an experiment
+ tracker (`HafniaLogger`). 
+
+
+## The Concept: Training as a Service (Training-aaS)
+`Training-aaS` is the concept of training models on the Hafnia platform on large 
+and *hidden* datasets. Hidden datasets refers to datasets that can be used for 
+training, but are not available for download or direct access. 
+
+This is a key feature of the Hafnia platform, as a hidden dataset ensures data 
+privacy, and allow models to be trained compliantly and ethically by third parties (you).
+
+The `script2model` approach is a Training-aaS concept, where you package your custom training 
+script as a *training recipe* and use the recipe to train models on the hidden datasets.
+
+To support local development of a training recipe, we have introduced a **sample dataset** 
+for each dataset available on the Hafnia platform. The sample dataset is a small 
+and anonymized subset of the full dataset and available for download. 
+
+With the sample dataset, you can seamlessly switch between local and Hafnia training. 
+Locally, you can create, validate and debug your training recipe. The recipe is then 
+launched with Hafnia Training-aaS, where the recipe runs on the full dataset and can be scaled to run on
+multiple GPUs and instances if needed. 
+
+
+## Getting started: Configuration
+To get started with Hafnia: 
+
+1. Install `hafnia` with your favorite python package manager. With pip do this:
+
+    `pip install hafnia`
+1. Sign in to the [Hafnia Platform](https://hafnia.milestonesys.com/). 
+1. Create an API KEY for Training aaS. For more instructions, follow this 
+[guide](https://hafnia.readme.io/docs/create-an-api-key). 
+Copy the key and save it for later use.
+1. From terminal, configure your machine to access Hafnia: 
+
+    ```
+    # Start configuration with
+    hafnia configure
+
+    # You are then prompted: 
+    Profile Name [default]:   # Press [Enter] or select an optional name
+    Hafnia API Key:  # Pass your HAFNIA API key
+    Hafnia Platform URL [https://api.mdi.milestonesys.com]:  # Press [Enter]
+    ```
+1. Download `mnist` from terminal to verify configuration is working.  
+
+    ```bash
+    hafnia data download mnist --force
+    ```
+
+## Getting started: Loading datasets samples
+With Hafnia configured on your local machine, it is now possible to download 
+and explore the dataset sample with a python script:
+
+```python
+from hafnia.data import load_dataset
+
+dataset_splits = load_dataset("midwest-vehicle-detection")
+print(dataset_splits)
+print(dataset_splits["train"])
+```
+
+Datasets with corresponding sample datasets can be found in [data library](https://hafnia.milestonesys.com/training-aas/datasets). It is early days for the data library, 
+but we are actively working on adding more datasets.
+
+The returned sample dataset is a [hugging face dataset](https://huggingface.co/docs/datasets/index) 
+and contains train, validation and test splits. 
+
+An important feature of `load_dataset` is that it will return the full dataset 
+when loaded on the Hafnia platform. 
+
+This enables seamlessly switching between running/validating a training script 
+locally (on the sample dataset) and running full model trainings in the cloud 
+without changing code or configurations for the training script.
+
+## Getting started: Experiment Tracking with HafniaLogger
+The `HafniaLogger` is an important part of the recipe script and enables you to track, log and
+reproduce your experiments.
+
+When integrated into your training script, the `HafniaLogger` is responsible for collecting:
+
+- **Trained Model**: The model trained during the experiment
+- **Model Checkpoints**: Intermediate model states saved during training
+- **Experiment Configurations**: Hyperparameters and other settings used in your experiment
+- **Training/Evaluation Metrics**: Performance data such as loss values, accuracy, and custom metrics
+
+### Basic Implementation Example
+
+Here's how to integrate the `HafniaLogger` into your training script:
+
+```python
+from hafnia.experiment import HafniaLogger
+
+batch_size = 128
+learning_rate = 0.001
+
+# Initialize Hafnia logger
+logger = HafniaLogger()
+
+# Log experiment parameters
+logger.log_configuration({"batch_size": 128, "learning_rate": 0.001})
+
+# Store checkpoints in this path
+ckpt_dir = logger.path_model_checkpoints()
+
+# Store the trained model in this path
+model_dir = logger.path_model()
+
+# Log scalar and metric values during training and validation
+logger.log_scalar("train/loss", value=0.1, step=100)
+logger.log_metric("train/accuracy", value=0.98, step=100)
+
+logger.log_scalar("validation/loss", value=0.1, step=100)
+logger.log_metric("validation/accuracy", value=0.95, step=100)
+```
+
+Similar to `load_dataset`, the tracker behaves differently when running locally or in the cloud. 
+Locally, experiment data is stored in a local folder `.data/experiments/{DATE_TIME}`. 
+
+In the cloud, the experiment data will be available in the Hafnia platform under 
+[experiments](https://hafnia.milestonesys.com/training-aas/experiments). 
+
+## Example: Torch Dataloader
+Commonly for `torch`-based training scripts, a dataset is used in combination 
+with a dataloader that performs data augmentations and batching of the dataset as torch tensors.
+
+To support this, we have provided a torch dataloader example script
+[example_torchvision_dataloader.py](./examples/example_torchvision_dataloader.py). 
+
+The script demonstrates how to make a dataloader with data augmentation (`torchvision.transforms.v2`)
+and a helper function for visualizing image and labels. 
+
+The dataloader and visualization function supports computer vision tasks 
+and datasets available in the data library. 
+
+## Example: Training-aaS
+By combining logging and dataset loading, we can now construct our model training recipe. 
+
+To demonstrate this, we have provided a recipe project that serves as a template for creating and structuring training recipes
+[recipe-classification](https://github.com/Data-insight-Platform/recipe-classification)
+
+The project also contains additional information on how to structure your training recipe, use the `HafniaLogger`, the `load_dataset` function and different approach for launching 
+the training recipe on the Hafnia platform.
+
+## Detailed Documentation
+For more information, go to our [documentation page](https://hafnia.readme.io/docs/welcome-to-hafnia) 
+or in below markdown pages. 
+
+- [CLI](docs/cli.md) - Detailed guide for the Hafnia command-line interface
+- [Script2Model Documentation](docs/s2m.md) - Detailed guide for script2model
+- [Release lifecycle](docs/release.md) - Details about package release lifecycle.
+
+## Development
+For development, we are using an uv based virtual python environment
+
+Install uv
+
+    curl -LsSf https://astral.sh/uv/install.sh | sh
+
+
+Install python dependencies including developer (`--dev`) and optional dependencies (`--all-extras`).
+
+    uv sync --all-extras --dev
+
+ 

hafnia-0.1.12/examples/example_logger.py

@@ -0,0 +1,23 @@
+from hafnia.experiment import HafniaLogger
+
+batch_size = 128
+learning_rate = 0.001
+
+# Initialize Hafnia logger
+logger = HafniaLogger()
+
+# Log experiment parameters
+logger.log_configuration({"batch_size": 128, "learning_rate": 0.001})
+
+# Store checkpoints in this path
+ckpt_dir = logger.path_model_checkpoints()
+
+# Store the trained model in this path
+model_dir = logger.path_model()
+
+# Log scalar and metric values during training and validation
+logger.log_scalar("train/loss", value=0.1, step=100)
+logger.log_metric("train/accuracy", value=0.98, step=100)
+
+logger.log_scalar("validation/loss", value=0.1, step=100)
+logger.log_metric("validation/accuracy", value=0.95, step=100)

{hafnia-0.1.9 → hafnia-0.1.12}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "hafnia"
-version = "0.1.9"
+version = "0.1.12"
 description = "Python tools for communication with Hafnia platform."
 readme = "README.md"
 authors = [{ name = "Ivan Sahumbaiev", email = "ivsa@milestone.dk" }]

{hafnia-0.1.9 → hafnia-0.1.12}/src/cli/config.py

@@ -140,6 +140,9 @@ class Config:
         del self.config_data.profiles[profile_name]
         self.save_config()
 
+    def is_configured(self) -> bool:
+        return self.config_data.active_profile is not None
+
     def clear(self) -> None:
         self.config_data = ConfigFileSchema(active_profile=None, profiles={})
         if self.config_path.exists():

hafnia-0.1.12/tests/test_check_example_scripts.py

@@ -0,0 +1,45 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+from cli.config import Config
+
+
+@pytest.mark.parametrize(
+    "script_path_str",
+    [
+        "examples/example_torchvision_dataloader.py",
+        "examples/example_load_dataset.py",
+        "examples/example_logger.py",
+        # Add other example scripts here
+    ],
+)
+def test_example_scripts(script_path_str: str):
+    if not Config().is_configured():
+        pytest.skip("Not logged in to Hafnia")
+
+    script_path = Path(script_path_str)
+    if not script_path.exists():
+        pytest.fail(f"Script {script_path} does not exist")
+
+    try:
+        result = subprocess.run(
+            [sys.executable, str(script_path)],
+            capture_output=True,
+            text=True,
+            timeout=120,  # 2-minute timeout
+        )
+
+        # Print output for debugging if there was an error
+        if result.returncode != 0:
+            print(f"STDOUT: {result.stdout}")
+            print(f"STDERR: {result.stderr}")
+
+        assert result.returncode == 0, f"Script {script_path} failed with return code {result.returncode}"
+
+    except subprocess.TimeoutExpired:
+        pytest.fail(f"Script {script_path} timed out")
+    except Exception as e:
+        pytest.fail(f"Failed to run script {script_path}: {str(e)}")

{hafnia-0.1.9 → hafnia-0.1.12}/tests/test_samples.py

@@ -13,7 +13,6 @@ from hafnia.data import load_dataset
 
 FORCE_REDOWNLOAD = False
 
-HAFNIA_LOGGED_IN = Config().config_data.active_profile is not None
 DATASETS_EXPECTED = [
     (
         "midwest-vehicle-detection",
@@ -36,7 +35,7 @@ DATASET_IDS = [dataset[0] for dataset in DATASETS_EXPECTED]
 @pytest.fixture(params=DATASETS_EXPECTED, ids=DATASET_IDS, scope="session")
 def loaded_dataset(request):
     """Fixture that loads a dataset and returns it along with metadata."""
-    if not HAFNIA_LOGGED_IN:
+    if not Config().is_configured():
         pytest.skip("Not logged in to Hafnia")
 
     dataset_name, expected_lengths, task_type = request.param

{hafnia-0.1.9 → hafnia-0.1.12}/uv.lock

@@ -549,7 +549,7 @@ http = [
 
 [[package]]
 name = "hafnia"
-version = "0.1.8"
+version = "0.1.12"
 source = { editable = "." }
 dependencies = [
     { name = "boto3" },

hafnia-0.1.9/PKG-INFO

@@ -1,42 +0,0 @@
-Metadata-Version: 2.4
-Name: hafnia
-Version: 0.1.9
-Summary: Python tools for communication with Hafnia platform.
-Author-email: Ivan Sahumbaiev <ivsa@milestone.dk>
-Requires-Python: >=3.10
-Requires-Dist: boto3>=1.35.91
-Requires-Dist: click>=8.1.8
-Requires-Dist: datasets>=3.2.0
-Requires-Dist: pillow>=11.1.0
-Requires-Dist: pyarrow>=18.1.0
-Requires-Dist: pydantic>=2.10.4
-Requires-Dist: rich>=13.9.4
-Requires-Dist: tqdm>=4.67.1
-Provides-Extra: torch
-Requires-Dist: flatten-dict>=0.4.2; extra == 'torch'
-Requires-Dist: torch>=2.6.0; extra == 'torch'
-Requires-Dist: torchvision>=0.21.0; extra == 'torch'
-Description-Content-Type: text/markdown
-
-# Project Hafnia 
-
-Project Hafnia is a comprehensive solution for managing data science experiments and resources. It provides tools and interfaces for experiment management, data handling, and container orchestration.
-
-
-## Documentation
-
-- [CLI Documentation](docs/cli.md) - Detailed guide for the Hafnia command-line interface
-- [Script2Model Documentation](docs/s2m.md) - Detailed guide for script2model
-- [Release lyfecycle](docs/release.md) - Details about package release lifecycle.
-
-
-## Key Components
-
-- **CLI Tool**: Command-line interface for platform interaction
-
-
-## Install
-
-```bash
-pip install hafnia
-```

hafnia-0.1.9/README.md

@@ -1,22 +0,0 @@
-# Project Hafnia 
-
-Project Hafnia is a comprehensive solution for managing data science experiments and resources. It provides tools and interfaces for experiment management, data handling, and container orchestration.
-
-
-## Documentation
-
-- [CLI Documentation](docs/cli.md) - Detailed guide for the Hafnia command-line interface
-- [Script2Model Documentation](docs/s2m.md) - Detailed guide for script2model
-- [Release lyfecycle](docs/release.md) - Details about package release lifecycle.
-
-
-## Key Components
-
-- **CLI Tool**: Command-line interface for platform interaction
-
-
-## Install
-
-```bash
-pip install hafnia
-```

hafnia-0.1.9/examples/core/example_run.ipynb

@@ -1,64 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from hafnia.data import load_dataset"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "dataset = load_dataset(\"mnist\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "for i in range(10):\n",
-    "    plt.subplot(2, 5, i + 1)\n",
-    "    plt.imshow(dataset[\"train\"][i][\"image\"], cmap=\"gray\")\n",
-    "    plt.title(dataset[\"train\"][i][\"label\"])\n",
-    "    plt.axis(\"off\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": ".venv",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.12"
-  },
-  "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}