kaiko-eva 0.2.1__tar.gz → 0.2.2__tar.gz

{kaiko_eva-0.2.1 → kaiko_eva-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: kaiko-eva
-Version: 0.2.1
+Version: 0.2.2
 Summary: Evaluation Framework for oncology foundation models.
 Keywords: machine-learning,evaluation-framework,oncology,foundation-models
 Author-Email: Ioannis Gatopoulos <ioannis@kaiko.ai>, =?utf-8?q?Nicolas_K=C3=A4nzig?= <nicolas@kaiko.ai>, Roman Moser <roman@kaiko.ai>
@@ -293,7 +293,7 @@ Check out the [documentation](https://kaiko-ai.github.io/eva/) for more informat
 
 ### Highlights:
 - Easy and reliable benchmark of Oncology FMs
-- Supports path-level classification, slide-level classification and semantic segmentation downstream tasks
+- Supports patch-level classification, slide-level classification and semantic segmentation downstream tasks
 - Automatic embedding inference and evaluation of a downstream task
 - Native support of popular medical [datasets](https://kaiko-ai.github.io/eva/dev/datasets/) and models
 - Produce statistics over multiple evaluation fits and multiple metrics
@@ -448,26 +448,28 @@ input, resulting in a faster evaluation.
 Here are some examples to get you started:
 
 - Perform a downstream offline **classification** evaluation of `DINO ViT-S/16`
-on the `BACH` dataset with linear probing by first inferring the embeddings
-and then performing 5 sequential fits:
+on the `BACH` dataset with linear probing by first pre-calculating the embeddings:
   ```sh
-  export DOWNLOAD_DATA=true
-  eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/offline/bach.yaml
+  DOWNLOAD_DATA=true \
+  MODEL_NAME=universal/vit_small_patch16_224_dino \
+  eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/pathology/offline/classification/bach.yaml
   ```
 
-- Perform a downstream online **segmentation** evaluation of `DINO ViT-S/16` on the
-`MoNuSAC` dataset with the `ConvDecoderMS` decoder:
+- Perform a downstream online **segmentation** evaluation of `DINO ViT-S/16` on the `MoNuSAC` dataset with the `ConvDecoderWithImage` decoder:
   ```sh
-  export DOWNLOAD_DATA=true
-  eva fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/online/monusac.yaml
+  DOWNLOAD_DATA=true \
+  MODEL_NAME=universal/vit_small_patch16_224_dino \
+  eva fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/pathology/online/segmentation/monusac.yaml
   ```
 
+By default `eva` will perform 5 evaluation runs using different seeds, however, you can control the number of runs through the `N_RUNS` environment variable or in the configuration file. The results will be saved to `./logs` by default, or to `OUTPUT_ROOT` if specified.
+
 For more examples, take a look at the [configs](https://github.com/kaiko-ai/eva/tree/main/configs)
-and [tutorials](https://kaiko-ai.github.io/eva/dev/user-guide/advanced/replicate_evaluations/).
+and [tutorials](https://kaiko-ai.github.io/eva/main/user-guide/advanced/replicate_evaluations/).
 
 > [!NOTE]
 > All the datasets that support automatic download in the repo have by default the option to automatically download set to false.
-> For automatic download you have to manually set the environmental variable `DOWNLOAD_DATA=true` or in the configuration file `download=true`.
+> For automatic download you have to manually set the environment variable `DOWNLOAD_DATA=true` or in the configuration file `download=true`.
 
 ## Leaderboards
 

{kaiko_eva-0.2.1 → kaiko_eva-0.2.2}/README.md

@@ -34,7 +34,7 @@ Check out the [documentation](https://kaiko-ai.github.io/eva/) for more informat
 
 ### Highlights:
 - Easy and reliable benchmark of Oncology FMs
-- Supports path-level classification, slide-level classification and semantic segmentation downstream tasks
+- Supports patch-level classification, slide-level classification and semantic segmentation downstream tasks
 - Automatic embedding inference and evaluation of a downstream task
 - Native support of popular medical [datasets](https://kaiko-ai.github.io/eva/dev/datasets/) and models
 - Produce statistics over multiple evaluation fits and multiple metrics
@@ -189,26 +189,28 @@ input, resulting in a faster evaluation.
 Here are some examples to get you started:
 
 - Perform a downstream offline **classification** evaluation of `DINO ViT-S/16`
-on the `BACH` dataset with linear probing by first inferring the embeddings
-and then performing 5 sequential fits:
+on the `BACH` dataset with linear probing by first pre-calculating the embeddings:
   ```sh
-  export DOWNLOAD_DATA=true
-  eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/offline/bach.yaml
+  DOWNLOAD_DATA=true \
+  MODEL_NAME=universal/vit_small_patch16_224_dino \
+  eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/pathology/offline/classification/bach.yaml
   ```
 
-- Perform a downstream online **segmentation** evaluation of `DINO ViT-S/16` on the
-`MoNuSAC` dataset with the `ConvDecoderMS` decoder:
+- Perform a downstream online **segmentation** evaluation of `DINO ViT-S/16` on the `MoNuSAC` dataset with the `ConvDecoderWithImage` decoder:
   ```sh
-  export DOWNLOAD_DATA=true
-  eva fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/online/monusac.yaml
+  DOWNLOAD_DATA=true \
+  MODEL_NAME=universal/vit_small_patch16_224_dino \
+  eva fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/pathology/online/segmentation/monusac.yaml
   ```
 
+By default `eva` will perform 5 evaluation runs using different seeds, however, you can control the number of runs through the `N_RUNS` environment variable or in the configuration file. The results will be saved to `./logs` by default, or to `OUTPUT_ROOT` if specified.
+
 For more examples, take a look at the [configs](https://github.com/kaiko-ai/eva/tree/main/configs)
-and [tutorials](https://kaiko-ai.github.io/eva/dev/user-guide/advanced/replicate_evaluations/).
+and [tutorials](https://kaiko-ai.github.io/eva/main/user-guide/advanced/replicate_evaluations/).
 
 > [!NOTE]
 > All the datasets that support automatic download in the repo have by default the option to automatically download set to false.
-> For automatic download you have to manually set the environmental variable `DOWNLOAD_DATA=true` or in the configuration file `download=true`.
+> For automatic download you have to manually set the environment variable `DOWNLOAD_DATA=true` or in the configuration file `download=true`.
 
 ## Leaderboards
 

{kaiko_eva-0.2.1 → kaiko_eva-0.2.2}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "pdm.backend"
 
 [project]
 name = "kaiko-eva"
-version = "0.2.1"
+version = "0.2.2"
 description = "Evaluation Framework for oncology foundation models."
 keywords = [
     "machine-learning",

{kaiko_eva-0.2.1 → kaiko_eva-0.2.2}/src/eva/core/trainers/_recorder.py

@@ -129,7 +129,10 @@ class SessionRecorder:
     def _save_config(self) -> None:
         """Saves the config yaml with resolved env placeholders to the output directory."""
         if self.config_path:
-            config = OmegaConf.load(self.config_path)
+            config_fs = cloud_io.get_filesystem(self.config_path)
+            with config_fs.open(self.config_path, "r") as config_file:
+                config = OmegaConf.load(config_file)  # type: ignore
+
             fs = cloud_io.get_filesystem(self._output_dir, anon=False)
             with fs.open(os.path.join(self._output_dir, self._config_file), "w") as file:
                 config_yaml = OmegaConf.to_yaml(config, resolve=True)

{kaiko_eva-0.2.1 → kaiko_eva-0.2.2}/src/eva/vision/models/modules/semantic_segmentation.py

@@ -1,6 +1,7 @@
 """"Neural Network Semantic Segmentation Module."""
 
-from typing import Any, Callable, Dict, Iterable, List
+import functools
+from typing import Any, Callable, Dict, Iterable, List, Tuple
 
 import torch
 from lightning.pytorch.cli import LRSchedulerCallable, OptimizerCallable
@@ -35,6 +36,7 @@ class SemanticSegmentationModule(module.ModelModule):
         metrics: metrics_lib.MetricsSchema | None = None,
         postprocess: batch_postprocess.BatchPostProcess | None = None,
         save_decoder_only: bool = True,
+        spatial_dims: int = 2,
     ) -> None:
         """Initializes the neural net head module.
 
@@ -57,6 +59,8 @@ class SemanticSegmentationModule(module.ModelModule):
                 predictions and targets.
             save_decoder_only: Whether to save only the decoder during checkpointing. If False,
                 will also save the encoder (not recommended when frozen).
+            spatial_dims: The number of spatial dimensions, 2 for 2D
+                and 3 for 3D segmentation.
         """
         super().__init__(metrics=metrics, postprocess=postprocess)
 
@@ -68,6 +72,7 @@ class SemanticSegmentationModule(module.ModelModule):
         self.lr_scheduler = lr_scheduler
         self.save_decoder_only = save_decoder_only
         self.inferer = inferer
+        self.spatial_dims = spatial_dims
 
     @override
     def configure_model(self) -> None:
@@ -111,13 +116,14 @@ class SemanticSegmentationModule(module.ModelModule):
     def forward(
         self,
         tensor: torch.Tensor,
+        to_size: Tuple[int, int],
         *args: Any,
         **kwargs: Any,
     ) -> torch.Tensor:
         return (
-            self.inferer(tensor, network=self._forward_networks)
+            self.inferer(tensor, network=functools.partial(self._forward_networks, to_size=to_size))
             if self.inferer is not None and not self.training
-            else self._forward_networks(tensor)
+            else self._forward_networks(tensor, to_size=to_size)
         )
 
     @override
@@ -168,7 +174,7 @@ class SemanticSegmentationModule(module.ModelModule):
             The batch step output.
         """
         data, targets, metadata = INPUT_TENSOR_BATCH(*batch)
-        predictions = self(data)
+        predictions = self(data, to_size=targets.shape[-self.spatial_dims :])
         loss = self.criterion(predictions, targets)
         return {
             "loss": loss,
@@ -177,12 +183,11 @@ class SemanticSegmentationModule(module.ModelModule):
             "metadata": metadata,
         }
 
-    def _forward_networks(self, tensor: torch.Tensor) -> torch.Tensor:
+    def _forward_networks(self, tensor: torch.Tensor, to_size: Tuple[int, int]) -> torch.Tensor:
         """Passes the input tensor through the encoder and decoder."""
         features = self.encoder(tensor) if self.encoder else tensor
         if isinstance(self.decoder, segmentation.Decoder):
             if not isinstance(features, list):
                 raise ValueError(f"Expected a list of feature map tensors, got {type(features)}.")
-            image_size = (tensor.shape[-2], tensor.shape[-1])
-            return self.decoder(DecoderInputs(features, image_size, tensor))
+            return self.decoder(DecoderInputs(features, to_size, tensor))
         return self.decoder(features)

{kaiko_eva-0.2.1 → kaiko_eva-0.2.2}/tests/eva/vision/models/modules/test_semantic_segmentation.py

@@ -43,17 +43,18 @@ def test_semantic_segmentation_module_forward_with_inferer(
     # Get a sample from dataset
     sample_data, _ = segmentation_dataset[0]
     sample_data = sample_data.unsqueeze(0)
+    to_size = sample_data.shape[2:]
 
     # Execute the forward pass in train & eval mode
     assert model_with_inferer.inferer is not None, "Inferer should not be None"
     with mock.patch.object(
         type(model_with_inferer.inferer), "__call__", wraps=model_with_inferer.inferer.__call__
     ) as mock_inferer_call:
-        output = model_with_inferer(sample_data)
+        output = model_with_inferer(sample_data, to_size=to_size)
         mock_inferer_call.assert_not_called()
 
         model_with_inferer.eval()
-        output = model_with_inferer(sample_data)
+        output = model_with_inferer(sample_data, to_size=to_size)
         mock_inferer_call.assert_called_once()
 
     assert output.shape[0] == 1