active-vision 0.0.3__tar.gz → 0.0.5__tar.gz

{active_vision-0.0.3 → active_vision-0.0.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: active-vision
-Version: 0.0.3
+Version: 0.0.5
 Summary: Active learning for edge vision.
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
@@ -12,6 +12,7 @@ Requires-Dist: ipykernel>=6.29.5
 Requires-Dist: ipywidgets>=8.1.5
 Requires-Dist: loguru>=0.7.3
 Requires-Dist: seaborn>=0.13.2
+Requires-Dist: timm>=1.0.13
 
 ![Python Version](https://img.shields.io/badge/python-3.10%2B-blue?style=for-the-badge)
 ![License](https://img.shields.io/badge/License-Apache%202.0-green.svg?style=for-the-badge)
@@ -22,20 +23,38 @@ Requires-Dist: seaborn>=0.13.2
   <img src="https://raw.githubusercontent.com/dnth/active-vision/main/assets/logo.png" alt="active-vision">
 </p>
 
-Active learning at the edge for computer vision.
+The goal of this project is to create a framework for the active learning loop for computer vision. The diagram below shows a general workflow of how the active learning loop works.
 
-The goal of this project is to create a framework for the active learning loop for computer vision deployed on edge devices. 
+<p align="center">
+  <img src="https://raw.githubusercontent.com/dnth/active-vision/main/assets/data_flywheel.gif" alt="active-vision", width="700">
+</p>
 
-## Installation
-I recommend using [uv](https://docs.astral.sh/uv/) to set up a virtual environment and install the package. You can also use other virtual env of your choice.
+### Supported tasks:
+- [X] Image classification
+- [ ] Object detection
+- [ ] Segmentation
 
-If you're using uv:
+### Supported models:
+- [X] Fastai models
+- [X] Torchvision models
+- [X] Timm models
+- [ ] Hugging Face models
 
-```bash
-uv venv
-uv sync
-```
-Once the virtual environment is created, you can install the package using pip.
+### Supported Active Learning Strategies:
+
+Uncertainty Sampling:
+- [X] Least confidence
+- [ ] Margin of confidence
+- [ ] Ratio of confidence
+- [ ] Entropy
+
+Diverse Sampling:
+- [X] Random sampling
+- [ ] Model-based outlier
+- [ ] Cluster-based
+- [ ] Representative
+
+## 📦 Installation
 
 Get a release from PyPI
 ```bash
@@ -49,19 +68,31 @@ cd active-vision
 pip install -e .
 ```
 
+I recommend using [uv](https://docs.astral.sh/uv/) to set up a virtual environment and install the package. You can also use other virtual env of your choice.
+
+If you're using uv:
+
+```bash
+uv venv
+uv sync
+```
+Once the virtual environment is created, you can install the package using pip.
+
 > [!TIP]
-> If you're using uv add a uv before the pip install command to install into your virtual environment. Eg:
+> If you're using uv add a `uv` before the pip install command to install into your virtual environment. Eg:
 > ```bash
 > uv pip install active-vision
 > ```
 
-## Usage
+## 🛠️ Usage
 See the [notebook](./nbs/04_relabel_loop.ipynb) for a complete example.
 
-Be sure to prepared 3 datasets:
-- train: A dataframe of an existing labeled training dataset.
-- unlabeled: A dataframe of unlabeled data which we will sample from using active learning.
-- eval: A dataframe of labeled data which we will use to evaluate the performance of the model. (Optional)
+Be sure to prepared 3 subsets of the dataset:
+- [Initial samples](./nbs/initial_samples.parquet): A dataframe of a labeled images to train an initial model. If you don't have any labeled data, you can label some images yourself.
+- [Unlabeled samples](./nbs/unlabeled_samples.parquet): A dataframe of *unlabeled* images. We will continuously sample from this set using active learning strategies.
+- [Evaluation samples](./nbs/evaluation_samples.parquet): A dataframe of *labeled* images. We will use this set to evaluate the performance of the model. This is the test set, DO NOT use it for active learning. Split this out in the beginning.
+
+As a toy example I created the above 3 datasets from the imagenette dataset.
 
 ```python
 from active_vision import ActiveLearner
@@ -90,7 +121,7 @@ uncertain_df = al.sample_uncertain(pred_df, num_samples=10)
 al.label(uncertain_df, output_filename="uncertain")
 ```
 
-![Gradio UI](./assets/labeling_ui.png)
+![Gradio UI](https://raw.githubusercontent.com/dnth/active-vision/main/assets/labeling_ui.png)
 
 Once complete, the labeled samples will be save into a new df.
 We can now add the newly labeled data to the training set.
@@ -102,17 +133,90 @@ al.add_to_train_set(labeled_df, output_filename="active_labeled")
 
 Repeat the process until the model is good enough. Use the dataset to train a larger model and deploy.
 
-## Workflow
-There are two workflows for active learning at the edge that we can use depending on the availability of labeled data.
+> [!TIP]
+> For the toy dataset, I got to about 93% accuracy on the evaluation set with 200+ labeled images. The best performing model on the [leaderboard](https://github.com/fastai/imagenette) got 95.11% accuracy training on all 9469 labeled images.
+> 
+> This took me about 6 iterations of relabeling. Each iteration took about 5 minutes to complete including labeling and model training (resnet18). See the [notebook](./nbs/04_relabel_loop.ipynb) for more details.
+>
+> But using the dataset of 200+ images, I trained a more capable model (convnext_small_in22k) and got 99.3% accuracy on the evaluation set. See the [notebook](./nbs/05_retrain_larger.ipynb) for more details.
+
+
+## 📊 Benchmarks
+This section contains the benchmarks I ran using the active learning loop on various datasets.
+
+Column description:
+- `#Labeled Images`: The number of labeled images used to train the model.
+- `Evaluation Accuracy`: The accuracy of the model on the evaluation set.
+- `Train Epochs`: The number of epochs used to train the model.
+- `Model`: The model used to train.
+- `Active Learning`: Whether active learning was used to train the model.
+- `Source`: The source of the results.
+
+### Imagenette
+- num classes: 10
+- num images: 9469
+
+To start the active learning loop, I labeled 100 images (10 images from each class) and iteratively relabeled the most informative images until I hit 275 labeled images. 
+
+The active learning loop is a iterative process and can keep going until you hit a stopping point. You can decide your own stopping point based on your use case. It could be:
+- You ran out of data to label.
+- You hit a performance goal.
+- You hit a budget.
+- Other criteria.
+
+For this dataset,I decided to stop the active learning loop at 275 labeled images because the performance on the evaluation set is close to the top performing model on the leaderboard. 
+
+
+| #Labeled Images | Evaluation Accuracy | Train Epochs | Model                | Active Learning | Source |
+|-----------------|---------------------|--------------|----------------------|----------------|--------|
+| 9469            | 94.90%              | 80           | xse_resnext50        | ❌             | [Link](https://github.com/fastai/imagenette) |
+| 9469            | 95.11%              | 200          | xse_resnext50        | ❌             | [Link](https://github.com/fastai/imagenette) |
+| 275             | 99.33%               | 6            | convnext_small_in22k | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/05_retrain_larger.ipynb) |
+| 275             | 93.40%               | 4            | resnet18             | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/04_relabel_loop.ipynb) |
+
+### Dog Food
+- num classes: 2
+- num images: 2100
+
+To start the active learning loop, I labeled 20 images (10 images from each class) and iteratively relabeled the most informative images until I hit 160 labeled images. 
+
+I decided to stop the active learning loop at 160 labeled images because the performance on the evaluation set is close to the top performing model on the leaderboard. You can decide your own stopping point based on your use case.
+
+| #Labeled Images | Evaluation Accuracy | Train Epochs | Model | Active Learning | Source |
+|-----------------|---------------------|--------------|-------|----------------|--------|
+| 2100            | 99.70%              | ?            | vit-base-patch16-224   | ❌             | [Link](https://huggingface.co/abhishek/autotrain-dog-vs-food) |
+| 160             | 100.00%             | 6            | convnext_small_in22k   | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/dog_food_dataset/02_train.ipynb) |
+| 160             | 97.60%              | 4            | resnet18              | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/dog_food_dataset/01_label.ipynb) |
+
+### Oxford-IIIT Pet
+- num classes: 37
+- num images: 3680
+
+To start the active learning loop, I labeled 370 images (10 images from each class) and iteratively relabeled the most informative images until I hit 612 labeled images. 
+
+I decided to stop the active learning loop at 612 labeled images because the performance on the evaluation set is close to the top performing model on the leaderboard. You can decide your own stopping point based on your use case.
+
+| #Labeled Images | Evaluation Accuracy | Train Epochs | Model | Active Learning | Source |
+|-----------------|---------------------|--------------|-------|----------------|--------|
+| 3680             | 95.40%              | 5           | vit-base-patch16-224   | ❌              | [Link](https://huggingface.co/walterg777/vit-base-oxford-iiit-pets) |
+| 612             | 90.26%              | 11            | convnext_small_in22k              | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/oxford_iiit_pets/02_train.ipynb) |
+| 612             | 91.38%              | 11            | vit-base-patch16-224              | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/oxford_iiit_pets/03_train_vit.ipynb) |
+
+
+
+## ➿ Workflow
+This section describes a more detailed workflow for active learning. There are two workflows for active learning that we can use depending on the availability of labeled data.
 
 ### With unlabeled data
-If we have no labeled data, we can use active learning to iteratively improve the model and build a labeled dataset.
+If we have no labeled data, the goal of the active learning loop is to build a resonably good labeled dataset to train a larger model.
+
+Steps:
 
 1. Load a small proxy model.
-2. Label an initial dataset.
+2. Label an initial dataset. If there is none, you'll have to label some images.
 3. Train the proxy model on the labeled dataset.
 4. Run inference on the unlabeled dataset.
-5. Evaluate the performance of the proxy model on the unlabeled dataset.
+5. Evaluate the performance of the proxy model.
 6. Is model good enough?    
     - Yes: Save the proxy model and the dataset.
     - No: Select the most informative images to label using active learning.
@@ -138,24 +242,25 @@ graph TD
 ```
     
 ### With labeled data
-If we have a labeled dataset, we can use active learning to iteratively improve the dataset and the model by fixing the most important label errors.
+If we already have a labeled dataset, the goal of the active learning loop is to iteratively improve the dataset and the model by fixing the most important label errors.
+
+Steps:
 
 1. Load a small proxy model.
 2. Train the proxy model on the labeled dataset.
 3. Run inference on the entire labeled dataset.
-4. Get the most important label errors with active learning.
+4. Get the most impactful label errors with active learning.
 5. Fix the label errors.
 6. Repeat steps 2-5 until the dataset is good enough.
 7. Save the labeled dataset.
 8. Train a larger model on the saved labeled dataset.
 
 
-
 ```mermaid
 graph TD
     A[Load a small proxy model] --> B[Train proxy model on labeled dataset]
     B --> C[Run inference on labeled dataset]
-    C --> D[Get important label errors using active learning]
+    C --> D[Get label errors using active learning]
     D --> E[Fix label errors]
     E --> F{Dataset good enough?}
     F -->|No| B
@@ -164,7 +269,8 @@ graph TD
 ```
 
 
-## Methodology
+
+<!-- ## Methodology
 To test out the workflows we will use the [imagenette dataset](https://huggingface.co/datasets/frgfm/imagenette). But this will be applicable to any dataset.
 
 Imagenette is a subset of the ImageNet dataset with 10 classes. We will use this dataset to test out the workflows. Additionally, Imagenette has an existing leaderboard which we can use to evaluate the performance of the models.
@@ -215,4 +321,4 @@ After the first iteration we got 94.57% accuracy on the validation set. See the
 > [!TIP]
 > | Train Epochs | Number of Images | Validation Accuracy |      Source      |
 > |--------------|-----------------|----------------------|------------------|
-> | 10           | 200             | 94.57%               | First relabeling [notebook](./nbs/03_retrain_model.ipynb) |
+> | 10           | 200             | 94.57%               | First relabeling [notebook](./nbs/03_retrain_model.ipynb) | -->

{active_vision-0.0.3 → active_vision-0.0.5}/README.md

@@ -7,20 +7,38 @@
   <img src="https://raw.githubusercontent.com/dnth/active-vision/main/assets/logo.png" alt="active-vision">
 </p>
 
-Active learning at the edge for computer vision.
+The goal of this project is to create a framework for the active learning loop for computer vision. The diagram below shows a general workflow of how the active learning loop works.
 
-The goal of this project is to create a framework for the active learning loop for computer vision deployed on edge devices. 
+<p align="center">
+  <img src="https://raw.githubusercontent.com/dnth/active-vision/main/assets/data_flywheel.gif" alt="active-vision", width="700">
+</p>
 
-## Installation
-I recommend using [uv](https://docs.astral.sh/uv/) to set up a virtual environment and install the package. You can also use other virtual env of your choice.
+### Supported tasks:
+- [X] Image classification
+- [ ] Object detection
+- [ ] Segmentation
 
-If you're using uv:
+### Supported models:
+- [X] Fastai models
+- [X] Torchvision models
+- [X] Timm models
+- [ ] Hugging Face models
 
-```bash
-uv venv
-uv sync
-```
-Once the virtual environment is created, you can install the package using pip.
+### Supported Active Learning Strategies:
+
+Uncertainty Sampling:
+- [X] Least confidence
+- [ ] Margin of confidence
+- [ ] Ratio of confidence
+- [ ] Entropy
+
+Diverse Sampling:
+- [X] Random sampling
+- [ ] Model-based outlier
+- [ ] Cluster-based
+- [ ] Representative
+
+## 📦 Installation
 
 Get a release from PyPI
 ```bash
@@ -34,19 +52,31 @@ cd active-vision
 pip install -e .
 ```
 
+I recommend using [uv](https://docs.astral.sh/uv/) to set up a virtual environment and install the package. You can also use other virtual env of your choice.
+
+If you're using uv:
+
+```bash
+uv venv
+uv sync
+```
+Once the virtual environment is created, you can install the package using pip.
+
 > [!TIP]
-> If you're using uv add a uv before the pip install command to install into your virtual environment. Eg:
+> If you're using uv add a `uv` before the pip install command to install into your virtual environment. Eg:
 > ```bash
 > uv pip install active-vision
 > ```
 
-## Usage
+## 🛠️ Usage
 See the [notebook](./nbs/04_relabel_loop.ipynb) for a complete example.
 
-Be sure to prepared 3 datasets:
-- train: A dataframe of an existing labeled training dataset.
-- unlabeled: A dataframe of unlabeled data which we will sample from using active learning.
-- eval: A dataframe of labeled data which we will use to evaluate the performance of the model. (Optional)
+Be sure to prepared 3 subsets of the dataset:
+- [Initial samples](./nbs/initial_samples.parquet): A dataframe of a labeled images to train an initial model. If you don't have any labeled data, you can label some images yourself.
+- [Unlabeled samples](./nbs/unlabeled_samples.parquet): A dataframe of *unlabeled* images. We will continuously sample from this set using active learning strategies.
+- [Evaluation samples](./nbs/evaluation_samples.parquet): A dataframe of *labeled* images. We will use this set to evaluate the performance of the model. This is the test set, DO NOT use it for active learning. Split this out in the beginning.
+
+As a toy example I created the above 3 datasets from the imagenette dataset.
 
 ```python
 from active_vision import ActiveLearner
@@ -75,7 +105,7 @@ uncertain_df = al.sample_uncertain(pred_df, num_samples=10)
 al.label(uncertain_df, output_filename="uncertain")
 ```
 
-![Gradio UI](./assets/labeling_ui.png)
+![Gradio UI](https://raw.githubusercontent.com/dnth/active-vision/main/assets/labeling_ui.png)
 
 Once complete, the labeled samples will be save into a new df.
 We can now add the newly labeled data to the training set.
@@ -87,17 +117,90 @@ al.add_to_train_set(labeled_df, output_filename="active_labeled")
 
 Repeat the process until the model is good enough. Use the dataset to train a larger model and deploy.
 
-## Workflow
-There are two workflows for active learning at the edge that we can use depending on the availability of labeled data.
+> [!TIP]
+> For the toy dataset, I got to about 93% accuracy on the evaluation set with 200+ labeled images. The best performing model on the [leaderboard](https://github.com/fastai/imagenette) got 95.11% accuracy training on all 9469 labeled images.
+> 
+> This took me about 6 iterations of relabeling. Each iteration took about 5 minutes to complete including labeling and model training (resnet18). See the [notebook](./nbs/04_relabel_loop.ipynb) for more details.
+>
+> But using the dataset of 200+ images, I trained a more capable model (convnext_small_in22k) and got 99.3% accuracy on the evaluation set. See the [notebook](./nbs/05_retrain_larger.ipynb) for more details.
+
+
+## 📊 Benchmarks
+This section contains the benchmarks I ran using the active learning loop on various datasets.
+
+Column description:
+- `#Labeled Images`: The number of labeled images used to train the model.
+- `Evaluation Accuracy`: The accuracy of the model on the evaluation set.
+- `Train Epochs`: The number of epochs used to train the model.
+- `Model`: The model used to train.
+- `Active Learning`: Whether active learning was used to train the model.
+- `Source`: The source of the results.
+
+### Imagenette
+- num classes: 10
+- num images: 9469
+
+To start the active learning loop, I labeled 100 images (10 images from each class) and iteratively relabeled the most informative images until I hit 275 labeled images. 
+
+The active learning loop is a iterative process and can keep going until you hit a stopping point. You can decide your own stopping point based on your use case. It could be:
+- You ran out of data to label.
+- You hit a performance goal.
+- You hit a budget.
+- Other criteria.
+
+For this dataset,I decided to stop the active learning loop at 275 labeled images because the performance on the evaluation set is close to the top performing model on the leaderboard. 
+
+
+| #Labeled Images | Evaluation Accuracy | Train Epochs | Model                | Active Learning | Source |
+|-----------------|---------------------|--------------|----------------------|----------------|--------|
+| 9469            | 94.90%              | 80           | xse_resnext50        | ❌             | [Link](https://github.com/fastai/imagenette) |
+| 9469            | 95.11%              | 200          | xse_resnext50        | ❌             | [Link](https://github.com/fastai/imagenette) |
+| 275             | 99.33%               | 6            | convnext_small_in22k | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/05_retrain_larger.ipynb) |
+| 275             | 93.40%               | 4            | resnet18             | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/04_relabel_loop.ipynb) |
+
+### Dog Food
+- num classes: 2
+- num images: 2100
+
+To start the active learning loop, I labeled 20 images (10 images from each class) and iteratively relabeled the most informative images until I hit 160 labeled images. 
+
+I decided to stop the active learning loop at 160 labeled images because the performance on the evaluation set is close to the top performing model on the leaderboard. You can decide your own stopping point based on your use case.
+
+| #Labeled Images | Evaluation Accuracy | Train Epochs | Model | Active Learning | Source |
+|-----------------|---------------------|--------------|-------|----------------|--------|
+| 2100            | 99.70%              | ?            | vit-base-patch16-224   | ❌             | [Link](https://huggingface.co/abhishek/autotrain-dog-vs-food) |
+| 160             | 100.00%             | 6            | convnext_small_in22k   | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/dog_food_dataset/02_train.ipynb) |
+| 160             | 97.60%              | 4            | resnet18              | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/dog_food_dataset/01_label.ipynb) |
+
+### Oxford-IIIT Pet
+- num classes: 37
+- num images: 3680
+
+To start the active learning loop, I labeled 370 images (10 images from each class) and iteratively relabeled the most informative images until I hit 612 labeled images. 
+
+I decided to stop the active learning loop at 612 labeled images because the performance on the evaluation set is close to the top performing model on the leaderboard. You can decide your own stopping point based on your use case.
+
+| #Labeled Images | Evaluation Accuracy | Train Epochs | Model | Active Learning | Source |
+|-----------------|---------------------|--------------|-------|----------------|--------|
+| 3680             | 95.40%              | 5           | vit-base-patch16-224   | ❌              | [Link](https://huggingface.co/walterg777/vit-base-oxford-iiit-pets) |
+| 612             | 90.26%              | 11            | convnext_small_in22k              | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/oxford_iiit_pets/02_train.ipynb) |
+| 612             | 91.38%              | 11            | vit-base-patch16-224              | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/oxford_iiit_pets/03_train_vit.ipynb) |
+
+
+
+## ➿ Workflow
+This section describes a more detailed workflow for active learning. There are two workflows for active learning that we can use depending on the availability of labeled data.
 
 ### With unlabeled data
-If we have no labeled data, we can use active learning to iteratively improve the model and build a labeled dataset.
+If we have no labeled data, the goal of the active learning loop is to build a resonably good labeled dataset to train a larger model.
+
+Steps:
 
 1. Load a small proxy model.
-2. Label an initial dataset.
+2. Label an initial dataset. If there is none, you'll have to label some images.
 3. Train the proxy model on the labeled dataset.
 4. Run inference on the unlabeled dataset.
-5. Evaluate the performance of the proxy model on the unlabeled dataset.
+5. Evaluate the performance of the proxy model.
 6. Is model good enough?    
     - Yes: Save the proxy model and the dataset.
     - No: Select the most informative images to label using active learning.
@@ -123,24 +226,25 @@ graph TD
 ```
     
 ### With labeled data
-If we have a labeled dataset, we can use active learning to iteratively improve the dataset and the model by fixing the most important label errors.
+If we already have a labeled dataset, the goal of the active learning loop is to iteratively improve the dataset and the model by fixing the most important label errors.
+
+Steps:
 
 1. Load a small proxy model.
 2. Train the proxy model on the labeled dataset.
 3. Run inference on the entire labeled dataset.
-4. Get the most important label errors with active learning.
+4. Get the most impactful label errors with active learning.
 5. Fix the label errors.
 6. Repeat steps 2-5 until the dataset is good enough.
 7. Save the labeled dataset.
 8. Train a larger model on the saved labeled dataset.
 
 
-
 ```mermaid
 graph TD
     A[Load a small proxy model] --> B[Train proxy model on labeled dataset]
     B --> C[Run inference on labeled dataset]
-    C --> D[Get important label errors using active learning]
+    C --> D[Get label errors using active learning]
     D --> E[Fix label errors]
     E --> F{Dataset good enough?}
     F -->|No| B
@@ -149,7 +253,8 @@ graph TD
 ```
 
 
-## Methodology
+
+<!-- ## Methodology
 To test out the workflows we will use the [imagenette dataset](https://huggingface.co/datasets/frgfm/imagenette). But this will be applicable to any dataset.
 
 Imagenette is a subset of the ImageNet dataset with 10 classes. We will use this dataset to test out the workflows. Additionally, Imagenette has an existing leaderboard which we can use to evaluate the performance of the models.
@@ -200,4 +305,4 @@ After the first iteration we got 94.57% accuracy on the validation set. See the
 > [!TIP]
 > | Train Epochs | Number of Images | Validation Accuracy |      Source      |
 > |--------------|-----------------|----------------------|------------------|
-> | 10           | 200             | 94.57%               | First relabeling [notebook](./nbs/03_retrain_model.ipynb) |
+> | 10           | 200             | 94.57%               | First relabeling [notebook](./nbs/03_retrain_model.ipynb) | -->

{active_vision-0.0.3 → active_vision-0.0.5}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "active-vision"
-version = "0.0.3"
+version = "0.0.5"
 description = "Active learning for edge vision."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -12,4 +12,5 @@ dependencies = [
     "ipywidgets>=8.1.5",
     "loguru>=0.7.3",
     "seaborn>=0.13.2",
-]
+    "timm>=1.0.13",
+]

active_vision-0.0.5/src/active_vision/__init__.py

@@ -0,0 +1,3 @@
+__version__ = "0.0.5"
+
+from .core import *

{active_vision-0.0.3 → active_vision-0.0.5}/src/active_vision/core.py

@@ -1,6 +1,5 @@
 import pandas as pd
 from loguru import logger
-from fastai.vision.models import resnet18, resnet34
 from fastai.callback.all import ShowGraphCallback
 from fastai.vision.all import (
     ImageDataLoaders,
@@ -17,6 +16,7 @@ import torch
 import torch.nn.functional as F
 
 import warnings
+from typing import Callable
 
 warnings.filterwarnings("ignore", category=FutureWarning)
 
@@ -25,13 +25,14 @@ class ActiveLearner:
     def __init__(self, model_name: str):
         self.model = self.load_model(model_name)
 
-    def load_model(self, model_name: str):
-        models = {"resnet18": resnet18, "resnet34": resnet34}
-        logger.info(f"Loading model {model_name}")
-        if model_name not in models:
-            logger.error(f"Model {model_name} not found")
-            raise ValueError(f"Model {model_name} not found")
-        return models[model_name]
+    def load_model(self, model_name: str | Callable):
+        if isinstance(model_name, Callable):
+            logger.info(f"Loading fastai model {model_name.__name__}")
+            return model_name
+
+        if isinstance(model_name, str):
+            logger.info(f"Loading timm model {model_name}")
+            return model_name
 
     def load_dataset(
         self,
@@ -41,6 +42,7 @@ class ActiveLearner:
         valid_pct: float = 0.2,
         batch_size: int = 16,
         image_size: int = 224,
+        batch_tfms: Callable = None,
     ):
         logger.info(f"Loading dataset from {filepath_col} and {label_col}")
         self.train_set = df.copy()
@@ -54,13 +56,16 @@ class ActiveLearner:
             label_col=label_col,
             bs=batch_size,
             item_tfms=Resize(image_size),
-            batch_tfms=aug_transforms(size=image_size, min_scale=0.75),
+            batch_tfms=batch_tfms,
         )
         logger.info("Creating learner")
         self.learn = vision_learner(self.dls, self.model, metrics=accuracy).to_fp16()
         self.class_names = self.dls.vocab
         logger.info("Done. Ready to train.")
 
+    def show_batch(self):
+        self.dls.show_batch()
+
     def lr_find(self):
         logger.info("Finding optimal learning rate")
         self.lrs = self.learn.lr_find(suggest_funcs=(minimum, steep, valley, slide))
@@ -112,13 +117,69 @@ class ActiveLearner:
         logger.info(f"Accuracy: {accuracy:.2%}")
         return accuracy
 
-    def sample_uncertain(self, df: pd.DataFrame, num_samples: int):
+    def sample_uncertain(
+        self, df: pd.DataFrame, num_samples: int, strategy: str = "least-confidence"
+    ):
         """
         Sample top `num_samples` low confidence samples. Returns a df with filepaths and predicted labels, and confidence scores.
+
+        Strategies:
+        - least-confidence: Get top `num_samples` low confidence samples.
+        - margin-of-confidence: Get top `num_samples` samples with the smallest margin between the top two predictions.
+        - ratio-of-confidence: Get top `num_samples` samples with the highest ratio between the top two predictions.
+        - entropy: Get top `num_samples` samples with the highest entropy.
         """
-        logger.info(f"Getting top {num_samples} low confidence samples")
-        uncertain_df = df.sort_values(by="pred_conf", ascending=True).head(num_samples)
-        return uncertain_df
+
+        # Remove samples that is already in the training set
+        df = df[~df["filepath"].isin(self.train_set["filepath"])]
+
+        if strategy == "least-confidence":
+            logger.info(f"Getting top {num_samples} low confidence samples")
+            uncertain_df = df.sort_values(by="pred_conf", ascending=True).head(
+                num_samples
+            )
+            return uncertain_df
+
+        # TODO: Implement margin of confidence strategy
+        elif strategy == "margin-of-confidence":
+            logger.error("Margin of confidence strategy not implemented")
+            raise NotImplementedError("Margin of confidence strategy not implemented")
+
+        # TODO: Implement ratio of confidence strategy
+        elif strategy == "ratio-of-confidence":
+            logger.error("Ratio of confidence strategy not implemented")
+            raise NotImplementedError("Ratio of confidence strategy not implemented")
+
+        # TODO: Implement entropy strategy
+        elif strategy == "entropy":
+            logger.error("Entropy strategy not implemented")
+            raise NotImplementedError("Entropy strategy not implemented")
+
+        else:
+            logger.error(f"Unknown strategy: {strategy}")
+            raise ValueError(f"Unknown strategy: {strategy}")
+
+    def sample_diverse(self, df: pd.DataFrame, num_samples: int):
+        """
+        Sample top `num_samples` diverse samples. Returns a df with filepaths and predicted labels, and confidence scores.
+
+        Strategies:
+        - model-based-outlier: Get top `num_samples` samples with lowest activation of the model's last layer.
+        - cluster-based: Get top `num_samples` samples with the highest distance to the nearest neighbor.
+        - representative: Get top `num_samples` samples with the highest distance to the centroid of the training set.
+        """
+        logger.error("Diverse sampling strategy not implemented")
+        raise NotImplementedError("Diverse sampling strategy not implemented")
+
+    def sample_random(self, df: pd.DataFrame, num_samples: int, seed: int = None):
+        """
+        Sample `num_samples` random samples. Returns a df with filepaths and predicted labels, and confidence scores.
+        """
+
+        logger.info(f"Sampling {num_samples} random samples")
+        if seed is not None:
+            logger.info(f"Using seed: {seed}")
+        return df.sample(n=num_samples, random_state=seed)
 
     def label(self, df: pd.DataFrame, output_filename: str = "labeled"):
         """

{active_vision-0.0.3 → active_vision-0.0.5}/src/active_vision.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: active-vision
-Version: 0.0.3
+Version: 0.0.5
 Summary: Active learning for edge vision.
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
@@ -12,6 +12,7 @@ Requires-Dist: ipykernel>=6.29.5
 Requires-Dist: ipywidgets>=8.1.5
 Requires-Dist: loguru>=0.7.3
 Requires-Dist: seaborn>=0.13.2
+Requires-Dist: timm>=1.0.13
 
 ![Python Version](https://img.shields.io/badge/python-3.10%2B-blue?style=for-the-badge)
 ![License](https://img.shields.io/badge/License-Apache%202.0-green.svg?style=for-the-badge)
@@ -22,20 +23,38 @@ Requires-Dist: seaborn>=0.13.2
   <img src="https://raw.githubusercontent.com/dnth/active-vision/main/assets/logo.png" alt="active-vision">
 </p>
 
-Active learning at the edge for computer vision.
+The goal of this project is to create a framework for the active learning loop for computer vision. The diagram below shows a general workflow of how the active learning loop works.
 
-The goal of this project is to create a framework for the active learning loop for computer vision deployed on edge devices. 
+<p align="center">
+  <img src="https://raw.githubusercontent.com/dnth/active-vision/main/assets/data_flywheel.gif" alt="active-vision", width="700">
+</p>
 
-## Installation
-I recommend using [uv](https://docs.astral.sh/uv/) to set up a virtual environment and install the package. You can also use other virtual env of your choice.
+### Supported tasks:
+- [X] Image classification
+- [ ] Object detection
+- [ ] Segmentation
 
-If you're using uv:
+### Supported models:
+- [X] Fastai models
+- [X] Torchvision models
+- [X] Timm models
+- [ ] Hugging Face models
 
-```bash
-uv venv
-uv sync
-```
-Once the virtual environment is created, you can install the package using pip.
+### Supported Active Learning Strategies:
+
+Uncertainty Sampling:
+- [X] Least confidence
+- [ ] Margin of confidence
+- [ ] Ratio of confidence
+- [ ] Entropy
+
+Diverse Sampling:
+- [X] Random sampling
+- [ ] Model-based outlier
+- [ ] Cluster-based
+- [ ] Representative
+
+## 📦 Installation
 
 Get a release from PyPI
 ```bash
@@ -49,19 +68,31 @@ cd active-vision
 pip install -e .
 ```
 
+I recommend using [uv](https://docs.astral.sh/uv/) to set up a virtual environment and install the package. You can also use other virtual env of your choice.
+
+If you're using uv:
+
+```bash
+uv venv
+uv sync
+```
+Once the virtual environment is created, you can install the package using pip.
+
 > [!TIP]
-> If you're using uv add a uv before the pip install command to install into your virtual environment. Eg:
+> If you're using uv add a `uv` before the pip install command to install into your virtual environment. Eg:
 > ```bash
 > uv pip install active-vision
 > ```
 
-## Usage
+## 🛠️ Usage
 See the [notebook](./nbs/04_relabel_loop.ipynb) for a complete example.
 
-Be sure to prepared 3 datasets:
-- train: A dataframe of an existing labeled training dataset.
-- unlabeled: A dataframe of unlabeled data which we will sample from using active learning.
-- eval: A dataframe of labeled data which we will use to evaluate the performance of the model. (Optional)
+Be sure to prepared 3 subsets of the dataset:
+- [Initial samples](./nbs/initial_samples.parquet): A dataframe of a labeled images to train an initial model. If you don't have any labeled data, you can label some images yourself.
+- [Unlabeled samples](./nbs/unlabeled_samples.parquet): A dataframe of *unlabeled* images. We will continuously sample from this set using active learning strategies.
+- [Evaluation samples](./nbs/evaluation_samples.parquet): A dataframe of *labeled* images. We will use this set to evaluate the performance of the model. This is the test set, DO NOT use it for active learning. Split this out in the beginning.
+
+As a toy example I created the above 3 datasets from the imagenette dataset.
 
 ```python
 from active_vision import ActiveLearner
@@ -90,7 +121,7 @@ uncertain_df = al.sample_uncertain(pred_df, num_samples=10)
 al.label(uncertain_df, output_filename="uncertain")
 ```
 
-![Gradio UI](./assets/labeling_ui.png)
+![Gradio UI](https://raw.githubusercontent.com/dnth/active-vision/main/assets/labeling_ui.png)
 
 Once complete, the labeled samples will be save into a new df.
 We can now add the newly labeled data to the training set.
@@ -102,17 +133,90 @@ al.add_to_train_set(labeled_df, output_filename="active_labeled")
 
 Repeat the process until the model is good enough. Use the dataset to train a larger model and deploy.
 
-## Workflow
-There are two workflows for active learning at the edge that we can use depending on the availability of labeled data.
+> [!TIP]
+> For the toy dataset, I got to about 93% accuracy on the evaluation set with 200+ labeled images. The best performing model on the [leaderboard](https://github.com/fastai/imagenette) got 95.11% accuracy training on all 9469 labeled images.
+> 
+> This took me about 6 iterations of relabeling. Each iteration took about 5 minutes to complete including labeling and model training (resnet18). See the [notebook](./nbs/04_relabel_loop.ipynb) for more details.
+>
+> But using the dataset of 200+ images, I trained a more capable model (convnext_small_in22k) and got 99.3% accuracy on the evaluation set. See the [notebook](./nbs/05_retrain_larger.ipynb) for more details.
+
+
+## 📊 Benchmarks
+This section contains the benchmarks I ran using the active learning loop on various datasets.
+
+Column description:
+- `#Labeled Images`: The number of labeled images used to train the model.
+- `Evaluation Accuracy`: The accuracy of the model on the evaluation set.
+- `Train Epochs`: The number of epochs used to train the model.
+- `Model`: The model used to train.
+- `Active Learning`: Whether active learning was used to train the model.
+- `Source`: The source of the results.
+
+### Imagenette
+- num classes: 10
+- num images: 9469
+
+To start the active learning loop, I labeled 100 images (10 images from each class) and iteratively relabeled the most informative images until I hit 275 labeled images. 
+
+The active learning loop is a iterative process and can keep going until you hit a stopping point. You can decide your own stopping point based on your use case. It could be:
+- You ran out of data to label.
+- You hit a performance goal.
+- You hit a budget.
+- Other criteria.
+
+For this dataset,I decided to stop the active learning loop at 275 labeled images because the performance on the evaluation set is close to the top performing model on the leaderboard. 
+
+
+| #Labeled Images | Evaluation Accuracy | Train Epochs | Model                | Active Learning | Source |
+|-----------------|---------------------|--------------|----------------------|----------------|--------|
+| 9469            | 94.90%              | 80           | xse_resnext50        | ❌             | [Link](https://github.com/fastai/imagenette) |
+| 9469            | 95.11%              | 200          | xse_resnext50        | ❌             | [Link](https://github.com/fastai/imagenette) |
+| 275             | 99.33%               | 6            | convnext_small_in22k | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/05_retrain_larger.ipynb) |
+| 275             | 93.40%               | 4            | resnet18             | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/04_relabel_loop.ipynb) |
+
+### Dog Food
+- num classes: 2
+- num images: 2100
+
+To start the active learning loop, I labeled 20 images (10 images from each class) and iteratively relabeled the most informative images until I hit 160 labeled images. 
+
+I decided to stop the active learning loop at 160 labeled images because the performance on the evaluation set is close to the top performing model on the leaderboard. You can decide your own stopping point based on your use case.
+
+| #Labeled Images | Evaluation Accuracy | Train Epochs | Model | Active Learning | Source |
+|-----------------|---------------------|--------------|-------|----------------|--------|
+| 2100            | 99.70%              | ?            | vit-base-patch16-224   | ❌             | [Link](https://huggingface.co/abhishek/autotrain-dog-vs-food) |
+| 160             | 100.00%             | 6            | convnext_small_in22k   | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/dog_food_dataset/02_train.ipynb) |
+| 160             | 97.60%              | 4            | resnet18              | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/dog_food_dataset/01_label.ipynb) |
+
+### Oxford-IIIT Pet
+- num classes: 37
+- num images: 3680
+
+To start the active learning loop, I labeled 370 images (10 images from each class) and iteratively relabeled the most informative images until I hit 612 labeled images. 
+
+I decided to stop the active learning loop at 612 labeled images because the performance on the evaluation set is close to the top performing model on the leaderboard. You can decide your own stopping point based on your use case.
+
+| #Labeled Images | Evaluation Accuracy | Train Epochs | Model | Active Learning | Source |
+|-----------------|---------------------|--------------|-------|----------------|--------|
+| 3680             | 95.40%              | 5           | vit-base-patch16-224   | ❌              | [Link](https://huggingface.co/walterg777/vit-base-oxford-iiit-pets) |
+| 612             | 90.26%              | 11            | convnext_small_in22k              | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/oxford_iiit_pets/02_train.ipynb) |
+| 612             | 91.38%              | 11            | vit-base-patch16-224              | ✓              | [Link](https://github.com/dnth/active-vision/blob/main/nbs/oxford_iiit_pets/03_train_vit.ipynb) |
+
+
+
+## ➿ Workflow
+This section describes a more detailed workflow for active learning. There are two workflows for active learning that we can use depending on the availability of labeled data.
 
 ### With unlabeled data
-If we have no labeled data, we can use active learning to iteratively improve the model and build a labeled dataset.
+If we have no labeled data, the goal of the active learning loop is to build a resonably good labeled dataset to train a larger model.
+
+Steps:
 
 1. Load a small proxy model.
-2. Label an initial dataset.
+2. Label an initial dataset. If there is none, you'll have to label some images.
 3. Train the proxy model on the labeled dataset.
 4. Run inference on the unlabeled dataset.
-5. Evaluate the performance of the proxy model on the unlabeled dataset.
+5. Evaluate the performance of the proxy model.
 6. Is model good enough?    
     - Yes: Save the proxy model and the dataset.
     - No: Select the most informative images to label using active learning.
@@ -138,24 +242,25 @@ graph TD
 ```
     
 ### With labeled data
-If we have a labeled dataset, we can use active learning to iteratively improve the dataset and the model by fixing the most important label errors.
+If we already have a labeled dataset, the goal of the active learning loop is to iteratively improve the dataset and the model by fixing the most important label errors.
+
+Steps:
 
 1. Load a small proxy model.
 2. Train the proxy model on the labeled dataset.
 3. Run inference on the entire labeled dataset.
-4. Get the most important label errors with active learning.
+4. Get the most impactful label errors with active learning.
 5. Fix the label errors.
 6. Repeat steps 2-5 until the dataset is good enough.
 7. Save the labeled dataset.
 8. Train a larger model on the saved labeled dataset.
 
 
-
 ```mermaid
 graph TD
     A[Load a small proxy model] --> B[Train proxy model on labeled dataset]
     B --> C[Run inference on labeled dataset]
-    C --> D[Get important label errors using active learning]
+    C --> D[Get label errors using active learning]
     D --> E[Fix label errors]
     E --> F{Dataset good enough?}
     F -->|No| B
@@ -164,7 +269,8 @@ graph TD
 ```
 
 
-## Methodology
+
+<!-- ## Methodology
 To test out the workflows we will use the [imagenette dataset](https://huggingface.co/datasets/frgfm/imagenette). But this will be applicable to any dataset.
 
 Imagenette is a subset of the ImageNet dataset with 10 classes. We will use this dataset to test out the workflows. Additionally, Imagenette has an existing leaderboard which we can use to evaluate the performance of the models.
@@ -215,4 +321,4 @@ After the first iteration we got 94.57% accuracy on the validation set. See the
 > [!TIP]
 > | Train Epochs | Number of Images | Validation Accuracy |      Source      |
 > |--------------|-----------------|----------------------|------------------|
-> | 10           | 200             | 94.57%               | First relabeling [notebook](./nbs/03_retrain_model.ipynb) |
+> | 10           | 200             | 94.57%               | First relabeling [notebook](./nbs/03_retrain_model.ipynb) | -->

{active_vision-0.0.3 → active_vision-0.0.5}/src/active_vision.egg-info/requires.txt

@@ -5,3 +5,4 @@ ipykernel>=6.29.5
 ipywidgets>=8.1.5
 loguru>=0.7.3
 seaborn>=0.13.2
+timm>=1.0.13

active_vision-0.0.3/src/active_vision/__init__.py

@@ -1,3 +0,0 @@
-__version__ = "0.0.3"
-
-from .core import *