kaiko-eva 0.0.1__tar.gz → 0.0.2__tar.gz

{kaiko_eva-0.0.1 → kaiko_eva-0.0.2}/PKG-INFO

@@ -1,10 +1,10 @@
 Metadata-Version: 2.1
 Name: kaiko-eva
-Version: 0.0.1
+Version: 0.0.2
 Summary: Evaluation Framework for oncology foundation models.
-Keywords: machine-learning evaluation-framework oncology foundation-models
-Author-Email: Ioannis Gatopoulos <ioannis@kaiko.ai>, Nicolas Känzig <nicolas@kaiko.ai>, Roman Moser <roman@kaiko.ai>
-Maintainer-Email: Ioannis Gatopoulos <ioannis@kaiko.ai>, Nicolas Känzig <nicolas@kaiko.ai>, Roman Moser <roman@kaiko.ai>
+Keywords: machine-learning,evaluation-framework,oncology,foundation-models
+Author-Email: Ioannis Gatopoulos <ioannis@kaiko.ai>, =?utf-8?q?Nicolas_K=C3=A4nzig?= <nicolas@kaiko.ai>, Roman Moser <roman@kaiko.ai>
+Maintainer-Email: Ioannis Gatopoulos <ioannis@kaiko.ai>, =?utf-8?q?Nicolas_K=C3=A4nzig?= <nicolas@kaiko.ai>, Roman Moser <roman@kaiko.ai>
 License:                                  Apache License
                                    Version 2.0, January 2004
                                 http://www.apache.org/licenses/
@@ -215,15 +215,17 @@ Project-URL: Homepage, https://kaiko-ai.github.io/eva/dev/
 Project-URL: Repository, https://github.com/kaiko-ai/eva
 Project-URL: Documentation, https://kaiko-ai.github.io/eva/dev/
 Requires-Python: >=3.10
-Requires-Dist: lightning>=2.2.1
-Requires-Dist: jsonargparse[omegaconf]>=4.27.4
+Requires-Dist: torch==2.3.0
+Requires-Dist: lightning>=2.2.2
+Requires-Dist: jsonargparse[omegaconf]==4.28
 Requires-Dist: tensorboard>=2.16.2
 Requires-Dist: loguru>=0.7.2
 Requires-Dist: pandas>=2.2.0
 Requires-Dist: transformers>=4.38.2
 Requires-Dist: onnxruntime>=1.17.1
-Requires-Dist: onnx>=1.15.0
+Requires-Dist: onnx>=1.16.0
 Requires-Dist: toolz>=0.12.1
+Requires-Dist: rich>=13.7.1
 Requires-Dist: h5py>=3.10.0; extra == "vision"
 Requires-Dist: nibabel>=5.2.0; extra == "vision"
 Requires-Dist: opencv-python-headless>=4.9.0.80; extra == "vision"
@@ -240,15 +242,19 @@ Description-Content-Type: text/markdown
 
 <div align="center">
 
-<img src="https://github.com/kaiko-ai/eva/blob/main/docs/images/eva-logo.png?raw=true" width="400">
+<br />
+
+<img src="https://github.com/kaiko-ai/eva/blob/main/docs/images/eva-logo.png?raw=true" width="340">
 
+<br />
 <br />
 
 _Oncology FM Evaluation Framework by kaiko.ai_
 
 [![PyPI](https://img.shields.io/pypi/v/kaiko-eva.svg?logo=python)](https://pypi.python.org/pypi/kaiko-eva)
-[![CI](https://github.com/kaiko-ai/eva/workflows/CI/badge.svg)](https://github.com/kaiko-ai/eva/actions?query=workflow%3ACI)
-[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg?labelColor=gray)](https://github.com/kaiko-ai/eva#license)
+[![docs](https://img.shields.io/badge/📚_docs-latest-green)](https://kaiko-ai.github.io/eva/latest)
+[![license](https://img.shields.io/badge/⚖️_License-Apache%202.0-blue.svg?labelColor=gray)](https://github.com/kaiko-ai/eva#license)<br>
+[![paper](http://img.shields.io/badge/OpenReview-MIDL_2024-B31B1B.svg)](https://openreview.net/forum?id=FNBQOPj18N&noteId=FNBQOPj18N)
 
 <p align="center">
   <a href="https://github.com/kaiko-ai/eva#installation">Installation</a> •
@@ -299,18 +305,18 @@ eva --version
 
 ## How To Use
 
-_eva_ can be used directly from the terminal as a CLI tool as follows:
+_`eva`_ can be used directly from the terminal as a CLI tool as follows:
 ```sh
 eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml 
 ```
 
-When used as a CLI tool, `_eva_` supports configuration files (`.yaml`) as an argument to define its functionality.
+When used as a CLI tool, _`eva`_ supports configuration files (`.yaml`) as an argument to define its functionality.
 Native supported configs can be found at the [configs](https://github.com/kaiko-ai/eva/tree/main/configs) directory
 of the repo. Apart from cloning the repo, you can download the latest config folder as `.zip` from your browser from
 [here](https://download-directory.github.io/?url=https://github.com/kaiko-ai/eva/tree/main/configs). Alternatively,
 from a specific release the configs can be downloaded from the terminal as follows:
 ```sh
-curl -LO https://github.com/kaiko-ai/eva/releases/download/0.0.1/configs.zip | unzip configs.zip
+curl -LO https://github.com/kaiko-ai/eva/releases/download/0.0.1/configs.zip | unzip configs
 ```
 
 For example, to perform a downstream evaluation of DINO ViT-S/16 on the BACH dataset with
@@ -338,7 +344,7 @@ and [tutorials](https://kaiko-ai.github.io/eva/dev/user-guide/advanced/replicate
 
 ## Benchmarks
 
-In this section you will find model benchmarks which were generated with _eva_.
+In this section you will find model benchmarks which were generated with _`eva`_.
 
 ### Table I: WSI patch-level benchmark
 
@@ -351,13 +357,15 @@ In this section you will find model benchmarks which were generated with _eva_.
 | ViT-S/16 _(random)_	<sup>[1]</sup>               | 0.410 | 0.617 | 0.501 | 0.753    | 0.728     |
 | ViT-S/16 _(ImageNet)_ <sup>[1]</sup>             | 0.695 | 0.935 | 0.831 | 0.864    | 0.849     |
 | ViT-B/8 _(ImageNet)_ <sup>[1]</sup>              | 0.710 | 0.939 | 0.814 | 0.870    | 0.856     |
+| ViT-L/14 _(ImageNet)_ <sup>[1]</sup>             | 0.707 | 0.916 | 0.832 | 0.873    | 0.888     |
 | DINO<sub>(p=16)</sub> <sup>[2]</sup>             | 0.801 | 0.934 | 0.768 | 0.889    | 0.895     |
 | Phikon <sup>[3]</sup>                            | 0.725 | 0.935 | 0.777 | 0.912    | 0.915     |
-| ViT-S/16 _(kaiko.ai)_ <sup>[4]</sup>             | 0.797 | 0.943 | 0.828 | 0.903    | 0.893     |
-| ViT-S/8 _(kaiko.ai)_ <sup>[4]</sup>              | 0.834 | 0.946 | 0.832 | 0.897    | 0.887     |
-| ViT-B/16 _(kaiko.ai)_	<sup>[4]</sup>             | 0.810 | 0.960 | 0.826 | 0.900    | 0.898     |
-| ViT-B/8 _(kaiko.ai)_ <sup>[4]</sup>              | 0.865 | 0.956 | 0.809 | 0.913    | 0.921     |
-| ViT-L/14 _(kaiko.ai)_ <sup>[4]</sup>             | 0.870 | 0.930 | 0.809 | 0.908    | 0.898     |
+| UNI <sup>[4]</sup>                               | 0.814 | 0.950 | 0.837 | 0.936    | 0.938     |
+| ViT-S/16 _(kaiko.ai)_ <sup>[5]</sup>             | 0.797 | 0.943 | 0.828 | 0.903    | 0.893     |
+| ViT-S/8 _(kaiko.ai)_ <sup>[5]</sup>              | 0.834 | 0.946 | 0.832 | 0.897    | 0.887     |
+| ViT-B/16 _(kaiko.ai)_	<sup>[5]</sup>             | 0.810 | 0.960 | 0.826 | 0.900    | 0.898     |
+| ViT-B/8 _(kaiko.ai)_ <sup>[5]</sup>              | 0.865 | 0.956 | 0.809 | 0.913    | 0.921     |
+| ViT-L/14 _(kaiko.ai)_ <sup>[5]</sup>             | 0.870 | 0.930 | 0.809 | 0.908    | 0.898     |
 
 _Table I: Linear probing evaluation of FMs on patch-level downstream datasets.<br> We report averaged balanced accuracy
 over 5 runs, with an average standard deviation of ±0.003._
@@ -367,14 +375,15 @@ over 5 runs, with an average standard deviation of ±0.003._
 <br />
 
 _References_:
-1. _"Emerging properties in self-supervised vision transformers”_
-2. _"Benchmarking self-supervised learning on diverse pathology datasets”_
-3. _"Scaling self-supervised learning for histopathology with masked image modeling”_
-4. _"Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scale”_
+1. _"Emerging properties in self-supervised vision transformers”_, [arXiv](https://arxiv.org/abs/2104.14294)
+2. _"Benchmarking self-supervised learning on diverse pathology datasets”_, [arXiv](https://arxiv.org/abs/2212.04690)
+3. _"Scaling self-supervised learning for histopathology with masked image modeling”_, [medRxiv](https://www.medrxiv.org/content/10.1101/2023.07.21.23292757v1)
+4. _"A General-Purpose Self-Supervised Model for Computational Pathology”_, [arXiv](https://arxiv.org/abs/2308.15474)
+5. _"Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scale”_, [arXiv](https://arxiv.org/pdf/2404.15217)
 
 ## Contributing
 
-_eva_ is an open source project and welcomes contributions of all kinds. Please checkout the [developer](./docs/DEVELOPER_GUIDE.md)
+_`eva`_ is an open source project and welcomes contributions of all kinds. Please checkout the [developer](./docs/DEVELOPER_GUIDE.md)
 and [contributing guide](./docs/CONTRIBUTING.md) for help on how to do so.
 
 All contributors must follow the [code of conduct](./docs/CODE_OF_CONDUCT.md).
@@ -399,7 +408,24 @@ Our codebase is built using multiple opensource contributions
 
 </div>
 
----
+
+## Citation
+
+If you find this repository useful, please consider giving a star ⭐ and adding the following citation:
+
+```
+@inproceedings{
+kaiko.ai2024eva,
+title={eva: Evaluation framework for pathology foundation models},
+author={kaiko.ai and Ioannis Gatopoulos and Nicolas K{\"a}nzig and Roman Moser and Sebastian Ot{\'a}lora},
+booktitle={Medical Imaging with Deep Learning},
+year={2024},
+url={https://openreview.net/forum?id=FNBQOPj18N}
+}
+```
+
+<br />
+
 <div align="center">
   <img src="https://github.com/kaiko-ai/eva/blob/main/docs/images/kaiko-logo.png?raw=true" width="200">
 </div>

{kaiko_eva-0.0.1 → kaiko_eva-0.0.2}/README.md

@@ -1,14 +1,18 @@
 <div align="center">
 
-<img src="https://github.com/kaiko-ai/eva/blob/main/docs/images/eva-logo.png?raw=true" width="400">
+<br />
+
+<img src="https://github.com/kaiko-ai/eva/blob/main/docs/images/eva-logo.png?raw=true" width="340">
 
+<br />
 <br />
 
 _Oncology FM Evaluation Framework by kaiko.ai_
 
 [![PyPI](https://img.shields.io/pypi/v/kaiko-eva.svg?logo=python)](https://pypi.python.org/pypi/kaiko-eva)
-[![CI](https://github.com/kaiko-ai/eva/workflows/CI/badge.svg)](https://github.com/kaiko-ai/eva/actions?query=workflow%3ACI)
-[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg?labelColor=gray)](https://github.com/kaiko-ai/eva#license)
+[![docs](https://img.shields.io/badge/📚_docs-latest-green)](https://kaiko-ai.github.io/eva/latest)
+[![license](https://img.shields.io/badge/⚖️_License-Apache%202.0-blue.svg?labelColor=gray)](https://github.com/kaiko-ai/eva#license)<br>
+[![paper](http://img.shields.io/badge/OpenReview-MIDL_2024-B31B1B.svg)](https://openreview.net/forum?id=FNBQOPj18N&noteId=FNBQOPj18N)
 
 <p align="center">
   <a href="https://github.com/kaiko-ai/eva#installation">Installation</a> •
@@ -59,18 +63,18 @@ eva --version
 
 ## How To Use
 
-_eva_ can be used directly from the terminal as a CLI tool as follows:
+_`eva`_ can be used directly from the terminal as a CLI tool as follows:
 ```sh
 eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml 
 ```
 
-When used as a CLI tool, `_eva_` supports configuration files (`.yaml`) as an argument to define its functionality.
+When used as a CLI tool, _`eva`_ supports configuration files (`.yaml`) as an argument to define its functionality.
 Native supported configs can be found at the [configs](https://github.com/kaiko-ai/eva/tree/main/configs) directory
 of the repo. Apart from cloning the repo, you can download the latest config folder as `.zip` from your browser from
 [here](https://download-directory.github.io/?url=https://github.com/kaiko-ai/eva/tree/main/configs). Alternatively,
 from a specific release the configs can be downloaded from the terminal as follows:
 ```sh
-curl -LO https://github.com/kaiko-ai/eva/releases/download/0.0.1/configs.zip | unzip configs.zip
+curl -LO https://github.com/kaiko-ai/eva/releases/download/0.0.1/configs.zip | unzip configs
 ```
 
 For example, to perform a downstream evaluation of DINO ViT-S/16 on the BACH dataset with
@@ -98,7 +102,7 @@ and [tutorials](https://kaiko-ai.github.io/eva/dev/user-guide/advanced/replicate
 
 ## Benchmarks
 
-In this section you will find model benchmarks which were generated with _eva_.
+In this section you will find model benchmarks which were generated with _`eva`_.
 
 ### Table I: WSI patch-level benchmark
 
@@ -111,13 +115,15 @@ In this section you will find model benchmarks which were generated with _eva_.
 | ViT-S/16 _(random)_	<sup>[1]</sup>               | 0.410 | 0.617 | 0.501 | 0.753    | 0.728     |
 | ViT-S/16 _(ImageNet)_ <sup>[1]</sup>             | 0.695 | 0.935 | 0.831 | 0.864    | 0.849     |
 | ViT-B/8 _(ImageNet)_ <sup>[1]</sup>              | 0.710 | 0.939 | 0.814 | 0.870    | 0.856     |
+| ViT-L/14 _(ImageNet)_ <sup>[1]</sup>             | 0.707 | 0.916 | 0.832 | 0.873    | 0.888     |
 | DINO<sub>(p=16)</sub> <sup>[2]</sup>             | 0.801 | 0.934 | 0.768 | 0.889    | 0.895     |
 | Phikon <sup>[3]</sup>                            | 0.725 | 0.935 | 0.777 | 0.912    | 0.915     |
-| ViT-S/16 _(kaiko.ai)_ <sup>[4]</sup>             | 0.797 | 0.943 | 0.828 | 0.903    | 0.893     |
-| ViT-S/8 _(kaiko.ai)_ <sup>[4]</sup>              | 0.834 | 0.946 | 0.832 | 0.897    | 0.887     |
-| ViT-B/16 _(kaiko.ai)_	<sup>[4]</sup>             | 0.810 | 0.960 | 0.826 | 0.900    | 0.898     |
-| ViT-B/8 _(kaiko.ai)_ <sup>[4]</sup>              | 0.865 | 0.956 | 0.809 | 0.913    | 0.921     |
-| ViT-L/14 _(kaiko.ai)_ <sup>[4]</sup>             | 0.870 | 0.930 | 0.809 | 0.908    | 0.898     |
+| UNI <sup>[4]</sup>                               | 0.814 | 0.950 | 0.837 | 0.936    | 0.938     |
+| ViT-S/16 _(kaiko.ai)_ <sup>[5]</sup>             | 0.797 | 0.943 | 0.828 | 0.903    | 0.893     |
+| ViT-S/8 _(kaiko.ai)_ <sup>[5]</sup>              | 0.834 | 0.946 | 0.832 | 0.897    | 0.887     |
+| ViT-B/16 _(kaiko.ai)_	<sup>[5]</sup>             | 0.810 | 0.960 | 0.826 | 0.900    | 0.898     |
+| ViT-B/8 _(kaiko.ai)_ <sup>[5]</sup>              | 0.865 | 0.956 | 0.809 | 0.913    | 0.921     |
+| ViT-L/14 _(kaiko.ai)_ <sup>[5]</sup>             | 0.870 | 0.930 | 0.809 | 0.908    | 0.898     |
 
 _Table I: Linear probing evaluation of FMs on patch-level downstream datasets.<br> We report averaged balanced accuracy
 over 5 runs, with an average standard deviation of ±0.003._
@@ -127,14 +133,15 @@ over 5 runs, with an average standard deviation of ±0.003._
 <br />
 
 _References_:
-1. _"Emerging properties in self-supervised vision transformers”_
-2. _"Benchmarking self-supervised learning on diverse pathology datasets”_
-3. _"Scaling self-supervised learning for histopathology with masked image modeling”_
-4. _"Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scale”_
+1. _"Emerging properties in self-supervised vision transformers”_, [arXiv](https://arxiv.org/abs/2104.14294)
+2. _"Benchmarking self-supervised learning on diverse pathology datasets”_, [arXiv](https://arxiv.org/abs/2212.04690)
+3. _"Scaling self-supervised learning for histopathology with masked image modeling”_, [medRxiv](https://www.medrxiv.org/content/10.1101/2023.07.21.23292757v1)
+4. _"A General-Purpose Self-Supervised Model for Computational Pathology”_, [arXiv](https://arxiv.org/abs/2308.15474)
+5. _"Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scale”_, [arXiv](https://arxiv.org/pdf/2404.15217)
 
 ## Contributing
 
-_eva_ is an open source project and welcomes contributions of all kinds. Please checkout the [developer](./docs/DEVELOPER_GUIDE.md)
+_`eva`_ is an open source project and welcomes contributions of all kinds. Please checkout the [developer](./docs/DEVELOPER_GUIDE.md)
 and [contributing guide](./docs/CONTRIBUTING.md) for help on how to do so.
 
 All contributors must follow the [code of conduct](./docs/CODE_OF_CONDUCT.md).
@@ -159,7 +166,24 @@ Our codebase is built using multiple opensource contributions
 
 </div>
 
----
+
+## Citation
+
+If you find this repository useful, please consider giving a star ⭐ and adding the following citation:
+
+```
+@inproceedings{
+kaiko.ai2024eva,
+title={eva: Evaluation framework for pathology foundation models},
+author={kaiko.ai and Ioannis Gatopoulos and Nicolas K{\"a}nzig and Roman Moser and Sebastian Ot{\'a}lora},
+booktitle={Medical Imaging with Deep Learning},
+year={2024},
+url={https://openreview.net/forum?id=FNBQOPj18N}
+}
+```
+
+<br />
+
 <div align="center">
   <img src="https://github.com/kaiko-ai/eva/blob/main/docs/images/kaiko-logo.png?raw=true" width="200">
 </div>

{kaiko_eva-0.0.1 → kaiko_eva-0.0.2}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "pdm.backend"
 
 [project]
 name = "kaiko-eva"
-version = "0.0.1"
+version = "0.0.2"
 description = "Evaluation Framework for oncology foundation models."
 keywords = [
     "machine-learning",
@@ -34,15 +34,17 @@ maintainers = [
 ]
 requires-python = ">=3.10"
 dependencies = [
-    "lightning>=2.2.1",
-    "jsonargparse[omegaconf]>=4.27.4",
+    "torch==2.3.0",
+    "lightning>=2.2.2",
+    "jsonargparse[omegaconf]==4.28",
     "tensorboard>=2.16.2",
     "loguru>=0.7.2",
     "pandas>=2.2.0",
     "transformers>=4.38.2",
     "onnxruntime>=1.17.1",
-    "onnx>=1.15.0",
+    "onnx>=1.16.0",
     "toolz>=0.12.1",
+    "rich>=13.7.1",
 ]
 
 [project.urls]

kaiko_eva-0.0.2/src/eva/core/callbacks/__init__.py

@@ -0,0 +1,6 @@
+"""Callbacks API."""
+
+from eva.core.callbacks.config import ConfigurationLogger
+from eva.core.callbacks.writers import EmbeddingsWriter
+
+__all__ = ["ConfigurationLogger", "EmbeddingsWriter"]

kaiko_eva-0.0.2/src/eva/core/callbacks/config.py

@@ -0,0 +1,143 @@
+"""Configuration logger callback."""
+
+import ast
+import os
+import sys
+from types import BuiltinFunctionType
+from typing import Any, Dict, List
+
+import lightning.pytorch as pl
+import yaml
+from lightning_fabric.utilities import cloud_io
+from loguru import logger as cli_logger
+from omegaconf import OmegaConf
+from typing_extensions import TypeGuard, override
+
+from eva.core import loggers
+
+
+class ConfigurationLogger(pl.Callback):
+    """Logs the submitted configuration to the experimental logger."""
+
+    _save_as: str = "config.yaml"
+
+    def __init__(self, verbose: bool = True) -> None:
+        """Initializes the callback.
+
+        Args:
+            verbose: Whether to print the configurations to print the
+                configuration to the terminal.
+        """
+        super().__init__()
+
+        self._verbose = verbose
+
+    @override
+    def setup(
+        self,
+        trainer: pl.Trainer,
+        pl_module: pl.LightningModule,
+        stage: str | None = None,
+    ) -> None:
+        log_dir = trainer.log_dir
+        if not _logdir_exists(log_dir):
+            return
+
+        configuration = _load_submitted_config()
+
+        if self._verbose:
+            config_as_text = yaml.dump(configuration, sort_keys=False)
+            print(f"Configuration:\033[94m\n---\n{config_as_text}\033[0m")
+
+        save_as = os.path.join(log_dir, self._save_as)
+        fs = cloud_io.get_filesystem(log_dir)
+        with fs.open(save_as, "w") as output_file:
+            yaml.dump(configuration, output_file, sort_keys=False)
+
+        loggers.log_parameters(trainer.loggers, tag="configuration", parameters=configuration)
+
+
+def _logdir_exists(logdir: str | None, verbose: bool = True) -> TypeGuard[str]:
+    """Checks if the trainer has a log directory.
+
+    Args:
+        logdir: Trainer's logdir.
+        name: The name to log with.
+        verbose: Whether to log if it does not exist.
+
+    Returns:
+        A bool indicating if the log directory exists or not.
+    """
+    exists = isinstance(logdir, str)
+    if not exists and verbose:
+        print("\n")
+        cli_logger.warning("Log directory is `None`. Configuration file will not be logged.\n")
+    return exists
+
+
+def _load_submitted_config() -> Dict[str, Any]:
+    """Retrieves and loads the submitted configuration.
+
+    Returns:
+        The path to the configuration file.
+    """
+    config_paths = _fetch_submitted_config_path()
+    return _load_yaml_files(config_paths)
+
+
+def _fetch_submitted_config_path() -> List[str]:
+    """Fetches the config path from command line arguments.
+
+    Returns:
+        The path to the configuration file.
+    """
+    return list(filter(lambda f: f.endswith(".yaml"), sys.argv))
+
+
+def _load_yaml_files(paths: List[str]) -> Dict[str, Any]:
+    """Loads yaml files and merge them from multiple paths.
+
+    Args:
+        paths: The paths to the yaml files.
+
+    Returns:
+        The merged configurations as a dictionary.
+    """
+    merged_config = {}
+    for config_path in paths:
+        fs = cloud_io.get_filesystem(config_path)
+        with fs.open(config_path, "r") as file:
+            omegaconf_file = OmegaConf.load(file)  # type: ignore
+            config_dict = OmegaConf.to_object(omegaconf_file)  # type: ignore
+            parsed_config = _type_resolver(config_dict)  # type: ignore
+            merged_config.update(parsed_config)
+    return merged_config
+
+
+def _type_resolver(mapping: Dict[str, Any]) -> Dict[str, Any]:
+    """Parses the string values of a dictionary in-place.
+
+    Args:
+        mapping: A dictionary object.
+
+    Returns:
+        The mapping with the formatted values.
+    """
+    for key, value in mapping.items():
+        if isinstance(value, dict):
+            formatted_value = _type_resolver(value)
+        elif isinstance(value, list) and isinstance(value[0], dict):
+            formatted_value = [_type_resolver(subvalue) for subvalue in value]
+        else:
+            try:
+                parsed_value = ast.literal_eval(value)  # type: ignore
+                formatted_value = (
+                    value if isinstance(parsed_value, BuiltinFunctionType) else parsed_value
+                )
+
+            except Exception:
+                formatted_value = value
+
+        mapping[key] = formatted_value
+
+    return mapping

kaiko_eva-0.0.2/src/eva/core/data/datasets/__init__.py

@@ -0,0 +1,15 @@
+"""Datasets API."""
+
+from eva.core.data.datasets.base import Dataset
+from eva.core.data.datasets.dataset import TorchDataset
+from eva.core.data.datasets.embeddings import (
+    EmbeddingsClassificationDataset,
+    MultiEmbeddingsClassificationDataset,
+)
+
+__all__ = [
+    "Dataset",
+    "EmbeddingsClassificationDataset",
+    "MultiEmbeddingsClassificationDataset",
+    "TorchDataset",
+]

kaiko_eva-0.0.2/src/eva/core/data/datasets/embeddings/__init__.py

@@ -0,0 +1,13 @@
+"""Datasets API."""
+
+from eva.core.data.datasets.embeddings.base import EmbeddingsDataset
+from eva.core.data.datasets.embeddings.classification import (
+    EmbeddingsClassificationDataset,
+    MultiEmbeddingsClassificationDataset,
+)
+
+__all__ = [
+    "EmbeddingsDataset",
+    "EmbeddingsClassificationDataset",
+    "MultiEmbeddingsClassificationDataset",
+]

kaiko_eva-0.0.1/src/eva/core/data/datasets/classification/embeddings.py → kaiko_eva-0.0.2/src/eva/core/data/datasets/embeddings/base.py

@@ -1,7 +1,8 @@
-"""Embeddings classification dataset."""
+"""Base dataset class for Embeddings."""
 
+import abc
 import os
-from typing import Callable, Dict, Tuple
+from typing import Callable, Dict, Literal, Tuple
 
 import numpy as np
 import pandas as pd
@@ -11,22 +12,23 @@ from typing_extensions import override
 from eva.core.data.datasets import base
 from eva.core.utils import io
 
+default_column_mapping: Dict[str, str] = {
+    "path": "embeddings",
+    "target": "target",
+    "split": "split",
+    "multi_id": "slide_id",
+}
+"""The default column mapping of the variables to the manifest columns."""
 
-class EmbeddingsClassificationDataset(base.Dataset):
-    """Embeddings classification dataset."""
 
-    default_column_mapping: Dict[str, str] = {
-        "data": "embeddings",
-        "target": "target",
-        "split": "split",
-    }
-    """The default column mapping of the variables to the manifest columns."""
+class EmbeddingsDataset(base.Dataset):
+    """Abstract base class for embedding datasets."""
 
     def __init__(
         self,
         root: str,
         manifest_file: str,
-        split: str | None = None,
+        split: Literal["train", "val", "test"] | None = None,
         column_mapping: Dict[str, str] = default_column_mapping,
         embeddings_transforms: Callable | None = None,
         target_transforms: Callable | None = None,
@@ -54,12 +56,38 @@ class EmbeddingsClassificationDataset(base.Dataset):
         self._root = root
         self._manifest_file = manifest_file
         self._split = split
-        self._column_mapping = self.default_column_mapping | column_mapping
+        self._column_mapping = default_column_mapping | column_mapping
         self._embeddings_transforms = embeddings_transforms
         self._target_transforms = target_transforms
 
         self._data: pd.DataFrame
 
+    @abc.abstractmethod
+    def _load_embeddings(self, index: int) -> torch.Tensor:
+        """Returns the `index`'th embedding sample.
+
+        Args:
+            index: The index of the data sample to load.
+
+        Returns:
+            The embedding sample as a tensor.
+        """
+
+    @abc.abstractmethod
+    def _load_target(self, index: int) -> np.ndarray:
+        """Returns the `index`'th target sample.
+
+        Args:
+            index: The index of the data sample to load.
+
+        Returns:
+            The sample target as an array.
+        """
+
+    @abc.abstractmethod
+    def __len__(self) -> int:
+        """Returns the total length of the data."""
+
     def filename(self, index: int) -> str:
         """Returns the filename of the `index`'th data sample.
 
@@ -71,7 +99,7 @@ class EmbeddingsClassificationDataset(base.Dataset):
         Returns:
             The filename of the `index`'th data sample.
         """
-        return self._data.at[index, self._column_mapping["data"]]
+        return self._data.at[index, self._column_mapping["path"]]
 
     @override
     def setup(self):
@@ -90,36 +118,6 @@ class EmbeddingsClassificationDataset(base.Dataset):
         target = self._load_target(index)
         return self._apply_transforms(embeddings, target)
 
-    def __len__(self) -> int:
-        """Returns the total length of the data."""
-        return len(self._data)
-
-    def _load_embeddings(self, index: int) -> torch.Tensor:
-        """Returns the `index`'th embedding sample.
-
-        Args:
-            index: The index of the data sample to load.
-
-        Returns:
-            The sample embedding as an array.
-        """
-        filename = self.filename(index)
-        embeddings_path = os.path.join(self._root, filename)
-        tensor = torch.load(embeddings_path, map_location="cpu")
-        return tensor.squeeze(0)
-
-    def _load_target(self, index: int) -> np.ndarray:
-        """Returns the `index`'th target sample.
-
-        Args:
-            index: The index of the data sample to load.
-
-        Returns:
-            The sample target as an array.
-        """
-        target = self._data.at[index, self._column_mapping["target"]]
-        return np.asarray(target, dtype=np.int64)
-
     def _load_manifest(self) -> pd.DataFrame:
         """Loads manifest file and filters the data based on the split column.
 

kaiko_eva-0.0.2/src/eva/core/data/datasets/embeddings/classification/__init__.py

@@ -0,0 +1,10 @@
+"""Embedding cllassification datasets API."""
+
+from eva.core.data.datasets.embeddings.classification.embeddings import (
+    EmbeddingsClassificationDataset,
+)
+from eva.core.data.datasets.embeddings.classification.multi_embeddings import (
+    MultiEmbeddingsClassificationDataset,
+)
+
+__all__ = ["EmbeddingsClassificationDataset", "MultiEmbeddingsClassificationDataset"]