kaiko-eva 0.0.0.dev8__tar.gz → 0.0.2__tar.gz

{kaiko_eva-0.0.0.dev8 → kaiko_eva-0.0.2}/PKG-INFO

@@ -1,10 +1,10 @@
 Metadata-Version: 2.1
 Name: kaiko-eva
-Version: 0.0.0.dev8
+Version: 0.0.2
 Summary: Evaluation Framework for oncology foundation models.
-Keywords: machine-learning evaluation-framework oncology foundation-models
-Author-Email: Ioannis Gatopoulos <ioannis@kaiko.ai>, Nicolas Känzig <nicolas@kaiko.ai>, Roman Moser <roman@kaiko.ai>
-Maintainer-Email: Ioannis Gatopoulos <ioannis@kaiko.ai>, Nicolas Känzig <nicolas@kaiko.ai>, Roman Moser <roman@kaiko.ai>
+Keywords: machine-learning,evaluation-framework,oncology,foundation-models
+Author-Email: Ioannis Gatopoulos <ioannis@kaiko.ai>, =?utf-8?q?Nicolas_K=C3=A4nzig?= <nicolas@kaiko.ai>, Roman Moser <roman@kaiko.ai>
+Maintainer-Email: Ioannis Gatopoulos <ioannis@kaiko.ai>, =?utf-8?q?Nicolas_K=C3=A4nzig?= <nicolas@kaiko.ai>, Roman Moser <roman@kaiko.ai>
 License:                                  Apache License
                                    Version 2.0, January 2004
                                 http://www.apache.org/licenses/
@@ -215,15 +215,17 @@ Project-URL: Homepage, https://kaiko-ai.github.io/eva/dev/
 Project-URL: Repository, https://github.com/kaiko-ai/eva
 Project-URL: Documentation, https://kaiko-ai.github.io/eva/dev/
 Requires-Python: >=3.10
-Requires-Dist: lightning>=2.2.1
-Requires-Dist: jsonargparse[omegaconf]>=4.27.4
+Requires-Dist: torch==2.3.0
+Requires-Dist: lightning>=2.2.2
+Requires-Dist: jsonargparse[omegaconf]==4.28
 Requires-Dist: tensorboard>=2.16.2
 Requires-Dist: loguru>=0.7.2
 Requires-Dist: pandas>=2.2.0
 Requires-Dist: transformers>=4.38.2
 Requires-Dist: onnxruntime>=1.17.1
-Requires-Dist: onnx>=1.15.0
+Requires-Dist: onnx>=1.16.0
 Requires-Dist: toolz>=0.12.1
+Requires-Dist: rich>=13.7.1
 Requires-Dist: h5py>=3.10.0; extra == "vision"
 Requires-Dist: nibabel>=5.2.0; extra == "vision"
 Requires-Dist: opencv-python-headless>=4.9.0.80; extra == "vision"
@@ -240,15 +242,19 @@ Description-Content-Type: text/markdown
 
 <div align="center">
 
-<img src="https://github.com/kaiko-ai/eva/blob/main/docs/images/eva-logo.png?raw=true" width="400">
+<br />
+
+<img src="https://github.com/kaiko-ai/eva/blob/main/docs/images/eva-logo.png?raw=true" width="340">
 
+<br />
 <br />
 
 _Oncology FM Evaluation Framework by kaiko.ai_
 
 [![PyPI](https://img.shields.io/pypi/v/kaiko-eva.svg?logo=python)](https://pypi.python.org/pypi/kaiko-eva)
-[![CI](https://github.com/kaiko-ai/eva/workflows/CI/badge.svg)](https://github.com/kaiko-ai/eva/actions?query=workflow%3ACI)
-[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg?labelColor=gray)](https://github.com/kaiko-ai/eva#license)
+[![docs](https://img.shields.io/badge/📚_docs-latest-green)](https://kaiko-ai.github.io/eva/latest)
+[![license](https://img.shields.io/badge/⚖️_License-Apache%202.0-blue.svg?labelColor=gray)](https://github.com/kaiko-ai/eva#license)<br>
+[![paper](http://img.shields.io/badge/OpenReview-MIDL_2024-B31B1B.svg)](https://openreview.net/forum?id=FNBQOPj18N&noteId=FNBQOPj18N)
 
 <p align="center">
   <a href="https://github.com/kaiko-ai/eva#installation">Installation</a> •
@@ -264,7 +270,8 @@ _Oncology FM Evaluation Framework by kaiko.ai_
 
 <br />
 
-_`eva`_ is an evaluation framework for oncology foundation models (FMs) by [kaiko.ai](https://kaiko.ai/). Check out the [documentation](https://kaiko-ai.github.io/eva/) for more information.
+_`eva`_ is an evaluation framework for oncology foundation models (FMs) by [kaiko.ai](https://kaiko.ai/).
+Check out the [documentation](https://kaiko-ai.github.io/eva/) for more information.
 
 ### Highlights:
 - Easy and reliable benchmark of Oncology FMs
@@ -298,18 +305,33 @@ eva --version
 
 ## How To Use
 
-_eva_ can be used directly from the terminal as a CLI tool as follows:
+_`eva`_ can be used directly from the terminal as a CLI tool as follows:
 ```sh
 eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml 
 ```
 
-For example, to perform a downstream evaluation of DINO ViT-S/16 on the BACH dataset with linear probing by first inferring the embeddings and performing 5 sequential fits, execute:
+When used as a CLI tool, _`eva`_ supports configuration files (`.yaml`) as an argument to define its functionality.
+Native supported configs can be found at the [configs](https://github.com/kaiko-ai/eva/tree/main/configs) directory
+of the repo. Apart from cloning the repo, you can download the latest config folder as `.zip` from your browser from
+[here](https://download-directory.github.io/?url=https://github.com/kaiko-ai/eva/tree/main/configs). Alternatively,
+from a specific release the configs can be downloaded from the terminal as follows:
+```sh
+curl -LO https://github.com/kaiko-ai/eva/releases/download/0.0.1/configs.zip | unzip configs
+```
+
+For example, to perform a downstream evaluation of DINO ViT-S/16 on the BACH dataset with
+linear probing by first inferring the embeddings and performing 5 sequential fits, execute:
 ```sh
+# from a locally stored config file
+eva predict_fit --config ./configs/vision/dino_vit/offline/bach.yaml
+
+# from a remote stored config file
 eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/offline/bach.yaml
 ```
 
 > [!NOTE] 
-> All the datasets that support automatic download in the repo have by default the option to automatically download set to false. For automatic download you have to manually set download=true.
+> All the datasets that support automatic download in the repo have by default the option to automatically download set to false.
+> For automatic download you have to manually set download=true.
 
 
 To view all the possibles, execute:
@@ -317,11 +339,12 @@ To view all the possibles, execute:
 eva --help
 ```
 
-For more information, please refer to the [documentation](https://kaiko-ai.github.io/eva/dev/user-guide/tutorials/offline_vs_online/) and [tutorials](https://kaiko-ai.github.io/eva/dev/user-guide/advanced/replicate_evaluations/).
+For more information, please refer to the [documentation](https://kaiko-ai.github.io/eva/dev/user-guide/tutorials/offline_vs_online/)
+and [tutorials](https://kaiko-ai.github.io/eva/dev/user-guide/advanced/replicate_evaluations/).
 
 ## Benchmarks
 
-In this section you will find model benchmarks which were generated with _eva_.
+In this section you will find model benchmarks which were generated with _`eva`_.
 
 ### Table I: WSI patch-level benchmark
 
@@ -334,13 +357,15 @@ In this section you will find model benchmarks which were generated with _eva_.
 | ViT-S/16 _(random)_	<sup>[1]</sup>               | 0.410 | 0.617 | 0.501 | 0.753    | 0.728     |
 | ViT-S/16 _(ImageNet)_ <sup>[1]</sup>             | 0.695 | 0.935 | 0.831 | 0.864    | 0.849     |
 | ViT-B/8 _(ImageNet)_ <sup>[1]</sup>              | 0.710 | 0.939 | 0.814 | 0.870    | 0.856     |
+| ViT-L/14 _(ImageNet)_ <sup>[1]</sup>             | 0.707 | 0.916 | 0.832 | 0.873    | 0.888     |
 | DINO<sub>(p=16)</sub> <sup>[2]</sup>             | 0.801 | 0.934 | 0.768 | 0.889    | 0.895     |
 | Phikon <sup>[3]</sup>                            | 0.725 | 0.935 | 0.777 | 0.912    | 0.915     |
-| ViT-S/16 _(kaiko.ai)_ <sup>[4]</sup>             | 0.797 | 0.943 | 0.828 | 0.903    | 0.893     |
-| ViT-S/8 _(kaiko.ai)_ <sup>[4]</sup>              | 0.834 | 0.946 | 0.832 | 0.897    | 0.887     |
-| ViT-B/16 _(kaiko.ai)_	<sup>[4]</sup>             | 0.810 | 0.960 | 0.826 | 0.900    | 0.898     |
-| ViT-B/8 _(kaiko.ai)_ <sup>[4]</sup>              | 0.865 | 0.956 | 0.809 | 0.913    | 0.921     |
-| ViT-L/14 _(kaiko.ai)_ <sup>[4]</sup>             | 0.870 | 0.930 | 0.809 | 0.908    | 0.898     |
+| UNI <sup>[4]</sup>                               | 0.814 | 0.950 | 0.837 | 0.936    | 0.938     |
+| ViT-S/16 _(kaiko.ai)_ <sup>[5]</sup>             | 0.797 | 0.943 | 0.828 | 0.903    | 0.893     |
+| ViT-S/8 _(kaiko.ai)_ <sup>[5]</sup>              | 0.834 | 0.946 | 0.832 | 0.897    | 0.887     |
+| ViT-B/16 _(kaiko.ai)_	<sup>[5]</sup>             | 0.810 | 0.960 | 0.826 | 0.900    | 0.898     |
+| ViT-B/8 _(kaiko.ai)_ <sup>[5]</sup>              | 0.865 | 0.956 | 0.809 | 0.913    | 0.921     |
+| ViT-L/14 _(kaiko.ai)_ <sup>[5]</sup>             | 0.870 | 0.930 | 0.809 | 0.908    | 0.898     |
 
 _Table I: Linear probing evaluation of FMs on patch-level downstream datasets.<br> We report averaged balanced accuracy
 over 5 runs, with an average standard deviation of ±0.003._
@@ -350,14 +375,16 @@ over 5 runs, with an average standard deviation of ±0.003._
 <br />
 
 _References_:
-1. _"Emerging properties in self-supervised vision transformers”_
-2. _"Benchmarking self-supervised learning on diverse pathology datasets”_
-3. _"Scaling self-supervised learning for histopathology with masked image modeling”_
-4. _"Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scale”_
+1. _"Emerging properties in self-supervised vision transformers”_, [arXiv](https://arxiv.org/abs/2104.14294)
+2. _"Benchmarking self-supervised learning on diverse pathology datasets”_, [arXiv](https://arxiv.org/abs/2212.04690)
+3. _"Scaling self-supervised learning for histopathology with masked image modeling”_, [medRxiv](https://www.medrxiv.org/content/10.1101/2023.07.21.23292757v1)
+4. _"A General-Purpose Self-Supervised Model for Computational Pathology”_, [arXiv](https://arxiv.org/abs/2308.15474)
+5. _"Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scale”_, [arXiv](https://arxiv.org/pdf/2404.15217)
 
 ## Contributing
 
-_eva_ is an open source project and welcomes contributions of all kinds. Please checkout the [developer](./docs/DEVELOPER_GUIDE.md) and [contributing guide](./docs/CONTRIBUTING.md) for help on how to do so.
+_`eva`_ is an open source project and welcomes contributions of all kinds. Please checkout the [developer](./docs/DEVELOPER_GUIDE.md)
+and [contributing guide](./docs/CONTRIBUTING.md) for help on how to do so.
 
 All contributors must follow the [code of conduct](./docs/CODE_OF_CONDUCT.md).
 
@@ -381,7 +408,24 @@ Our codebase is built using multiple opensource contributions
 
 </div>
 
----
+
+## Citation
+
+If you find this repository useful, please consider giving a star ⭐ and adding the following citation:
+
+```
+@inproceedings{
+kaiko.ai2024eva,
+title={eva: Evaluation framework for pathology foundation models},
+author={kaiko.ai and Ioannis Gatopoulos and Nicolas K{\"a}nzig and Roman Moser and Sebastian Ot{\'a}lora},
+booktitle={Medical Imaging with Deep Learning},
+year={2024},
+url={https://openreview.net/forum?id=FNBQOPj18N}
+}
+```
+
+<br />
+
 <div align="center">
   <img src="https://github.com/kaiko-ai/eva/blob/main/docs/images/kaiko-logo.png?raw=true" width="200">
 </div>

{kaiko_eva-0.0.0.dev8 → kaiko_eva-0.0.2}/README.md

@@ -1,14 +1,18 @@
 <div align="center">
 
-<img src="https://github.com/kaiko-ai/eva/blob/main/docs/images/eva-logo.png?raw=true" width="400">
+<br />
+
+<img src="https://github.com/kaiko-ai/eva/blob/main/docs/images/eva-logo.png?raw=true" width="340">
 
+<br />
 <br />
 
 _Oncology FM Evaluation Framework by kaiko.ai_
 
 [![PyPI](https://img.shields.io/pypi/v/kaiko-eva.svg?logo=python)](https://pypi.python.org/pypi/kaiko-eva)
-[![CI](https://github.com/kaiko-ai/eva/workflows/CI/badge.svg)](https://github.com/kaiko-ai/eva/actions?query=workflow%3ACI)
-[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg?labelColor=gray)](https://github.com/kaiko-ai/eva#license)
+[![docs](https://img.shields.io/badge/📚_docs-latest-green)](https://kaiko-ai.github.io/eva/latest)
+[![license](https://img.shields.io/badge/⚖️_License-Apache%202.0-blue.svg?labelColor=gray)](https://github.com/kaiko-ai/eva#license)<br>
+[![paper](http://img.shields.io/badge/OpenReview-MIDL_2024-B31B1B.svg)](https://openreview.net/forum?id=FNBQOPj18N&noteId=FNBQOPj18N)
 
 <p align="center">
   <a href="https://github.com/kaiko-ai/eva#installation">Installation</a> •
@@ -24,7 +28,8 @@ _Oncology FM Evaluation Framework by kaiko.ai_
 
 <br />
 
-_`eva`_ is an evaluation framework for oncology foundation models (FMs) by [kaiko.ai](https://kaiko.ai/). Check out the [documentation](https://kaiko-ai.github.io/eva/) for more information.
+_`eva`_ is an evaluation framework for oncology foundation models (FMs) by [kaiko.ai](https://kaiko.ai/).
+Check out the [documentation](https://kaiko-ai.github.io/eva/) for more information.
 
 ### Highlights:
 - Easy and reliable benchmark of Oncology FMs
@@ -58,18 +63,33 @@ eva --version
 
 ## How To Use
 
-_eva_ can be used directly from the terminal as a CLI tool as follows:
+_`eva`_ can be used directly from the terminal as a CLI tool as follows:
 ```sh
 eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml 
 ```
 
-For example, to perform a downstream evaluation of DINO ViT-S/16 on the BACH dataset with linear probing by first inferring the embeddings and performing 5 sequential fits, execute:
+When used as a CLI tool, _`eva`_ supports configuration files (`.yaml`) as an argument to define its functionality.
+Native supported configs can be found at the [configs](https://github.com/kaiko-ai/eva/tree/main/configs) directory
+of the repo. Apart from cloning the repo, you can download the latest config folder as `.zip` from your browser from
+[here](https://download-directory.github.io/?url=https://github.com/kaiko-ai/eva/tree/main/configs). Alternatively,
+from a specific release the configs can be downloaded from the terminal as follows:
+```sh
+curl -LO https://github.com/kaiko-ai/eva/releases/download/0.0.1/configs.zip | unzip configs
+```
+
+For example, to perform a downstream evaluation of DINO ViT-S/16 on the BACH dataset with
+linear probing by first inferring the embeddings and performing 5 sequential fits, execute:
 ```sh
+# from a locally stored config file
+eva predict_fit --config ./configs/vision/dino_vit/offline/bach.yaml
+
+# from a remote stored config file
 eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/offline/bach.yaml
 ```
 
 > [!NOTE] 
-> All the datasets that support automatic download in the repo have by default the option to automatically download set to false. For automatic download you have to manually set download=true.
+> All the datasets that support automatic download in the repo have by default the option to automatically download set to false.
+> For automatic download you have to manually set download=true.
 
 
 To view all the possibles, execute:
@@ -77,11 +97,12 @@ To view all the possibles, execute:
 eva --help
 ```
 
-For more information, please refer to the [documentation](https://kaiko-ai.github.io/eva/dev/user-guide/tutorials/offline_vs_online/) and [tutorials](https://kaiko-ai.github.io/eva/dev/user-guide/advanced/replicate_evaluations/).
+For more information, please refer to the [documentation](https://kaiko-ai.github.io/eva/dev/user-guide/tutorials/offline_vs_online/)
+and [tutorials](https://kaiko-ai.github.io/eva/dev/user-guide/advanced/replicate_evaluations/).
 
 ## Benchmarks
 
-In this section you will find model benchmarks which were generated with _eva_.
+In this section you will find model benchmarks which were generated with _`eva`_.
 
 ### Table I: WSI patch-level benchmark
 
@@ -94,13 +115,15 @@ In this section you will find model benchmarks which were generated with _eva_.
 | ViT-S/16 _(random)_	<sup>[1]</sup>               | 0.410 | 0.617 | 0.501 | 0.753    | 0.728     |
 | ViT-S/16 _(ImageNet)_ <sup>[1]</sup>             | 0.695 | 0.935 | 0.831 | 0.864    | 0.849     |
 | ViT-B/8 _(ImageNet)_ <sup>[1]</sup>              | 0.710 | 0.939 | 0.814 | 0.870    | 0.856     |
+| ViT-L/14 _(ImageNet)_ <sup>[1]</sup>             | 0.707 | 0.916 | 0.832 | 0.873    | 0.888     |
 | DINO<sub>(p=16)</sub> <sup>[2]</sup>             | 0.801 | 0.934 | 0.768 | 0.889    | 0.895     |
 | Phikon <sup>[3]</sup>                            | 0.725 | 0.935 | 0.777 | 0.912    | 0.915     |
-| ViT-S/16 _(kaiko.ai)_ <sup>[4]</sup>             | 0.797 | 0.943 | 0.828 | 0.903    | 0.893     |
-| ViT-S/8 _(kaiko.ai)_ <sup>[4]</sup>              | 0.834 | 0.946 | 0.832 | 0.897    | 0.887     |
-| ViT-B/16 _(kaiko.ai)_	<sup>[4]</sup>             | 0.810 | 0.960 | 0.826 | 0.900    | 0.898     |
-| ViT-B/8 _(kaiko.ai)_ <sup>[4]</sup>              | 0.865 | 0.956 | 0.809 | 0.913    | 0.921     |
-| ViT-L/14 _(kaiko.ai)_ <sup>[4]</sup>             | 0.870 | 0.930 | 0.809 | 0.908    | 0.898     |
+| UNI <sup>[4]</sup>                               | 0.814 | 0.950 | 0.837 | 0.936    | 0.938     |
+| ViT-S/16 _(kaiko.ai)_ <sup>[5]</sup>             | 0.797 | 0.943 | 0.828 | 0.903    | 0.893     |
+| ViT-S/8 _(kaiko.ai)_ <sup>[5]</sup>              | 0.834 | 0.946 | 0.832 | 0.897    | 0.887     |
+| ViT-B/16 _(kaiko.ai)_	<sup>[5]</sup>             | 0.810 | 0.960 | 0.826 | 0.900    | 0.898     |
+| ViT-B/8 _(kaiko.ai)_ <sup>[5]</sup>              | 0.865 | 0.956 | 0.809 | 0.913    | 0.921     |
+| ViT-L/14 _(kaiko.ai)_ <sup>[5]</sup>             | 0.870 | 0.930 | 0.809 | 0.908    | 0.898     |
 
 _Table I: Linear probing evaluation of FMs on patch-level downstream datasets.<br> We report averaged balanced accuracy
 over 5 runs, with an average standard deviation of ±0.003._
@@ -110,14 +133,16 @@ over 5 runs, with an average standard deviation of ±0.003._
 <br />
 
 _References_:
-1. _"Emerging properties in self-supervised vision transformers”_
-2. _"Benchmarking self-supervised learning on diverse pathology datasets”_
-3. _"Scaling self-supervised learning for histopathology with masked image modeling”_
-4. _"Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scale”_
+1. _"Emerging properties in self-supervised vision transformers”_, [arXiv](https://arxiv.org/abs/2104.14294)
+2. _"Benchmarking self-supervised learning on diverse pathology datasets”_, [arXiv](https://arxiv.org/abs/2212.04690)
+3. _"Scaling self-supervised learning for histopathology with masked image modeling”_, [medRxiv](https://www.medrxiv.org/content/10.1101/2023.07.21.23292757v1)
+4. _"A General-Purpose Self-Supervised Model for Computational Pathology”_, [arXiv](https://arxiv.org/abs/2308.15474)
+5. _"Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scale”_, [arXiv](https://arxiv.org/pdf/2404.15217)
 
 ## Contributing
 
-_eva_ is an open source project and welcomes contributions of all kinds. Please checkout the [developer](./docs/DEVELOPER_GUIDE.md) and [contributing guide](./docs/CONTRIBUTING.md) for help on how to do so.
+_`eva`_ is an open source project and welcomes contributions of all kinds. Please checkout the [developer](./docs/DEVELOPER_GUIDE.md)
+and [contributing guide](./docs/CONTRIBUTING.md) for help on how to do so.
 
 All contributors must follow the [code of conduct](./docs/CODE_OF_CONDUCT.md).
 
@@ -141,7 +166,24 @@ Our codebase is built using multiple opensource contributions
 
 </div>
 
----
+
+## Citation
+
+If you find this repository useful, please consider giving a star ⭐ and adding the following citation:
+
+```
+@inproceedings{
+kaiko.ai2024eva,
+title={eva: Evaluation framework for pathology foundation models},
+author={kaiko.ai and Ioannis Gatopoulos and Nicolas K{\"a}nzig and Roman Moser and Sebastian Ot{\'a}lora},
+booktitle={Medical Imaging with Deep Learning},
+year={2024},
+url={https://openreview.net/forum?id=FNBQOPj18N}
+}
+```
+
+<br />
+
 <div align="center">
   <img src="https://github.com/kaiko-ai/eva/blob/main/docs/images/kaiko-logo.png?raw=true" width="200">
 </div>

{kaiko_eva-0.0.0.dev8 → kaiko_eva-0.0.2}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "pdm.backend"
 
 [project]
 name = "kaiko-eva"
-version = "0.0.0.dev8"
+version = "0.0.2"
 description = "Evaluation Framework for oncology foundation models."
 keywords = [
     "machine-learning",
@@ -34,15 +34,17 @@ maintainers = [
 ]
 requires-python = ">=3.10"
 dependencies = [
-    "lightning>=2.2.1",
-    "jsonargparse[omegaconf]>=4.27.4",
+    "torch==2.3.0",
+    "lightning>=2.2.2",
+    "jsonargparse[omegaconf]==4.28",
     "tensorboard>=2.16.2",
     "loguru>=0.7.2",
     "pandas>=2.2.0",
     "transformers>=4.38.2",
     "onnxruntime>=1.17.1",
-    "onnx>=1.15.0",
+    "onnx>=1.16.0",
     "toolz>=0.12.1",
+    "rich>=13.7.1",
 ]
 
 [project.urls]

kaiko_eva-0.0.2/src/eva/core/callbacks/__init__.py

@@ -0,0 +1,6 @@
+"""Callbacks API."""
+
+from eva.core.callbacks.config import ConfigurationLogger
+from eva.core.callbacks.writers import EmbeddingsWriter
+
+__all__ = ["ConfigurationLogger", "EmbeddingsWriter"]

kaiko_eva-0.0.2/src/eva/core/callbacks/config.py

@@ -0,0 +1,143 @@
+"""Configuration logger callback."""
+
+import ast
+import os
+import sys
+from types import BuiltinFunctionType
+from typing import Any, Dict, List
+
+import lightning.pytorch as pl
+import yaml
+from lightning_fabric.utilities import cloud_io
+from loguru import logger as cli_logger
+from omegaconf import OmegaConf
+from typing_extensions import TypeGuard, override
+
+from eva.core import loggers
+
+
+class ConfigurationLogger(pl.Callback):
+    """Logs the submitted configuration to the experimental logger."""
+
+    _save_as: str = "config.yaml"
+
+    def __init__(self, verbose: bool = True) -> None:
+        """Initializes the callback.
+
+        Args:
+            verbose: Whether to print the configurations to print the
+                configuration to the terminal.
+        """
+        super().__init__()
+
+        self._verbose = verbose
+
+    @override
+    def setup(
+        self,
+        trainer: pl.Trainer,
+        pl_module: pl.LightningModule,
+        stage: str | None = None,
+    ) -> None:
+        log_dir = trainer.log_dir
+        if not _logdir_exists(log_dir):
+            return
+
+        configuration = _load_submitted_config()
+
+        if self._verbose:
+            config_as_text = yaml.dump(configuration, sort_keys=False)
+            print(f"Configuration:\033[94m\n---\n{config_as_text}\033[0m")
+
+        save_as = os.path.join(log_dir, self._save_as)
+        fs = cloud_io.get_filesystem(log_dir)
+        with fs.open(save_as, "w") as output_file:
+            yaml.dump(configuration, output_file, sort_keys=False)
+
+        loggers.log_parameters(trainer.loggers, tag="configuration", parameters=configuration)
+
+
+def _logdir_exists(logdir: str | None, verbose: bool = True) -> TypeGuard[str]:
+    """Checks if the trainer has a log directory.
+
+    Args:
+        logdir: Trainer's logdir.
+        name: The name to log with.
+        verbose: Whether to log if it does not exist.
+
+    Returns:
+        A bool indicating if the log directory exists or not.
+    """
+    exists = isinstance(logdir, str)
+    if not exists and verbose:
+        print("\n")
+        cli_logger.warning("Log directory is `None`. Configuration file will not be logged.\n")
+    return exists
+
+
+def _load_submitted_config() -> Dict[str, Any]:
+    """Retrieves and loads the submitted configuration.
+
+    Returns:
+        The path to the configuration file.
+    """
+    config_paths = _fetch_submitted_config_path()
+    return _load_yaml_files(config_paths)
+
+
+def _fetch_submitted_config_path() -> List[str]:
+    """Fetches the config path from command line arguments.
+
+    Returns:
+        The path to the configuration file.
+    """
+    return list(filter(lambda f: f.endswith(".yaml"), sys.argv))
+
+
+def _load_yaml_files(paths: List[str]) -> Dict[str, Any]:
+    """Loads yaml files and merge them from multiple paths.
+
+    Args:
+        paths: The paths to the yaml files.
+
+    Returns:
+        The merged configurations as a dictionary.
+    """
+    merged_config = {}
+    for config_path in paths:
+        fs = cloud_io.get_filesystem(config_path)
+        with fs.open(config_path, "r") as file:
+            omegaconf_file = OmegaConf.load(file)  # type: ignore
+            config_dict = OmegaConf.to_object(omegaconf_file)  # type: ignore
+            parsed_config = _type_resolver(config_dict)  # type: ignore
+            merged_config.update(parsed_config)
+    return merged_config
+
+
+def _type_resolver(mapping: Dict[str, Any]) -> Dict[str, Any]:
+    """Parses the string values of a dictionary in-place.
+
+    Args:
+        mapping: A dictionary object.
+
+    Returns:
+        The mapping with the formatted values.
+    """
+    for key, value in mapping.items():
+        if isinstance(value, dict):
+            formatted_value = _type_resolver(value)
+        elif isinstance(value, list) and isinstance(value[0], dict):
+            formatted_value = [_type_resolver(subvalue) for subvalue in value]
+        else:
+            try:
+                parsed_value = ast.literal_eval(value)  # type: ignore
+                formatted_value = (
+                    value if isinstance(parsed_value, BuiltinFunctionType) else parsed_value
+                )
+
+            except Exception:
+                formatted_value = value
+
+        mapping[key] = formatted_value
+
+    return mapping

kaiko_eva-0.0.2/src/eva/core/data/datasets/__init__.py

@@ -0,0 +1,15 @@
+"""Datasets API."""
+
+from eva.core.data.datasets.base import Dataset
+from eva.core.data.datasets.dataset import TorchDataset
+from eva.core.data.datasets.embeddings import (
+    EmbeddingsClassificationDataset,
+    MultiEmbeddingsClassificationDataset,
+)
+
+__all__ = [
+    "Dataset",
+    "EmbeddingsClassificationDataset",
+    "MultiEmbeddingsClassificationDataset",
+    "TorchDataset",
+]

kaiko_eva-0.0.2/src/eva/core/data/datasets/embeddings/__init__.py

@@ -0,0 +1,13 @@
+"""Datasets API."""
+
+from eva.core.data.datasets.embeddings.base import EmbeddingsDataset
+from eva.core.data.datasets.embeddings.classification import (
+    EmbeddingsClassificationDataset,
+    MultiEmbeddingsClassificationDataset,
+)
+
+__all__ = [
+    "EmbeddingsDataset",
+    "EmbeddingsClassificationDataset",
+    "MultiEmbeddingsClassificationDataset",
+]