oodeel 0.1.1__tar.gz → 0.2.0__tar.gz

{oodeel-0.1.1 → oodeel-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: oodeel
-Version: 0.1.1
+Version: 0.2.0
 Summary: Simple, compact, and hackable post-hoc deep OOD detection for alreadytrained tensorflow or pytorch image classifiers.
 Author: DEEL Core Team
 Author-email: paul.novello@irt-saintexupery.com
@@ -194,14 +194,14 @@ Currently, **oodeel** includes the following baselines:
 | MSP | [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](http://arxiv.org/abs/1610.02136) | ICLR 2017 | avail [tensorflow & torch](docs/pages/getting_started.ipynb)|
 | Mahalanobis | [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](http://arxiv.org/abs/1807.03888) | NeurIPS 2018 | avail [tensorflow](docs/notebooks/tensorflow/demo_mahalanobis_tf.ipynb) or  [torch](docs/notebooks/torch/demo_mahalanobis_torch.ipynb)|
 | Energy | [Energy-based Out-of-distribution Detection](http://arxiv.org/abs/2010.03759) | NeurIPS 2020 |avail [tensorflow](docs/notebooks/tensorflow/demo_energy_tf.ipynb) or  [torch](docs/notebooks/torch/demo_energy_torch.ipynb)  |
-| Odin | [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](http://arxiv.org/abs/1706.02690) | ICLR 2018 |avail [tensorflow](docs/notebooks/tensorflow/demo_odin_tf.ipynb) or  [torch](docs/notebooks/torch/demo_odin_torch.ipynb) |
+| Odin | [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](http://arxiv.org/abs/1706.02690) | ICLR 2018 | avail [tensorflow](docs/notebooks/tensorflow/demo_odin_tf.ipynb) or  [torch](docs/notebooks/torch/demo_odin_torch.ipynb) |
 | DKNN | [Out-of-Distribution Detection with Deep Nearest Neighbors](http://arxiv.org/abs/2204.06507) | ICML 2022 | avail [tensorflow](docs/notebooks/tensorflow/demo_dknn_tf.ipynb) or  [torch](docs/notebooks/torch/demo_dknn_torch.ipynb) |
 | VIM | [ViM: Out-Of-Distribution with Virtual-logit Matching](http://arxiv.org/abs/2203.10807) | CVPR 2022 |avail [tensorflow](docs/notebooks/tensorflow/demo_vim_tf.ipynb) or  [torch](docs/notebooks/torch/demo_vim_torch.ipynb)  |
 | Entropy | [Likelihood Ratios for Out-of-Distribution Detection](https://proceedings.neurips.cc/paper/2019/hash/1e79596878b2320cac26dd792a6c51c9-Abstract.html) | NeurIPS 2019 |avail [tensorflow](docs/notebooks/tensorflow/demo_entropy_tf.ipynb) or  [torch](docs/notebooks/torch/demo_entropy_torch.ipynb)  |
 | GODIN | [Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data](https://ieeexplore.ieee.org/document/9156473/) | CVPR 2020 | planned |
 | ReAct | [ReAct: Out-of-distribution Detection With Rectified Activations](http://arxiv.org/abs/2111.12797) | NeurIPS 2021 | avail [tensorflow](docs/notebooks/tensorflow/demo_react_tf.ipynb) or  [torch](docs/notebooks/torch/demo_react_torch.ipynb) |
 | NMD | [Neural Mean Discrepancy for Efficient Out-of-Distribution Detection](https://openaccess.thecvf.com/content/CVPR2022/html/Dong_Neural_Mean_Discrepancy_for_Efficient_Out-of-Distribution_Detection_CVPR_2022_paper.html) | CVPR 2022 | planned |
-| Gram | [Detecting Out-of-Distribution Examples with Gram Matrices](https://proceedings.mlr.press/v119/sastry20a.html) | ICML 2020 | planned |
+| Gram | [Detecting Out-of-Distribution Examples with Gram Matrices](https://proceedings.mlr.press/v119/sastry20a.html) | ICML 2020 | avail [tensorflow](docs/notebooks/tensorflow/demo_gram_tf.ipynb) or  [torch](docs/notebooks/torch/demo_gram_torch.ipynb) |
 
 
 

{oodeel-0.1.1 → oodeel-0.2.0}/README.md

@@ -173,14 +173,14 @@ Currently, **oodeel** includes the following baselines:
 | MSP | [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](http://arxiv.org/abs/1610.02136) | ICLR 2017 | avail [tensorflow & torch](docs/pages/getting_started.ipynb)|
 | Mahalanobis | [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](http://arxiv.org/abs/1807.03888) | NeurIPS 2018 | avail [tensorflow](docs/notebooks/tensorflow/demo_mahalanobis_tf.ipynb) or  [torch](docs/notebooks/torch/demo_mahalanobis_torch.ipynb)|
 | Energy | [Energy-based Out-of-distribution Detection](http://arxiv.org/abs/2010.03759) | NeurIPS 2020 |avail [tensorflow](docs/notebooks/tensorflow/demo_energy_tf.ipynb) or  [torch](docs/notebooks/torch/demo_energy_torch.ipynb)  |
-| Odin | [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](http://arxiv.org/abs/1706.02690) | ICLR 2018 |avail [tensorflow](docs/notebooks/tensorflow/demo_odin_tf.ipynb) or  [torch](docs/notebooks/torch/demo_odin_torch.ipynb) |
+| Odin | [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](http://arxiv.org/abs/1706.02690) | ICLR 2018 | avail [tensorflow](docs/notebooks/tensorflow/demo_odin_tf.ipynb) or  [torch](docs/notebooks/torch/demo_odin_torch.ipynb) |
 | DKNN | [Out-of-Distribution Detection with Deep Nearest Neighbors](http://arxiv.org/abs/2204.06507) | ICML 2022 | avail [tensorflow](docs/notebooks/tensorflow/demo_dknn_tf.ipynb) or  [torch](docs/notebooks/torch/demo_dknn_torch.ipynb) |
 | VIM | [ViM: Out-Of-Distribution with Virtual-logit Matching](http://arxiv.org/abs/2203.10807) | CVPR 2022 |avail [tensorflow](docs/notebooks/tensorflow/demo_vim_tf.ipynb) or  [torch](docs/notebooks/torch/demo_vim_torch.ipynb)  |
 | Entropy | [Likelihood Ratios for Out-of-Distribution Detection](https://proceedings.neurips.cc/paper/2019/hash/1e79596878b2320cac26dd792a6c51c9-Abstract.html) | NeurIPS 2019 |avail [tensorflow](docs/notebooks/tensorflow/demo_entropy_tf.ipynb) or  [torch](docs/notebooks/torch/demo_entropy_torch.ipynb)  |
 | GODIN | [Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data](https://ieeexplore.ieee.org/document/9156473/) | CVPR 2020 | planned |
 | ReAct | [ReAct: Out-of-distribution Detection With Rectified Activations](http://arxiv.org/abs/2111.12797) | NeurIPS 2021 | avail [tensorflow](docs/notebooks/tensorflow/demo_react_tf.ipynb) or  [torch](docs/notebooks/torch/demo_react_torch.ipynb) |
 | NMD | [Neural Mean Discrepancy for Efficient Out-of-Distribution Detection](https://openaccess.thecvf.com/content/CVPR2022/html/Dong_Neural_Mean_Discrepancy_for_Efficient_Out-of-Distribution_Detection_CVPR_2022_paper.html) | CVPR 2022 | planned |
-| Gram | [Detecting Out-of-Distribution Examples with Gram Matrices](https://proceedings.mlr.press/v119/sastry20a.html) | ICML 2020 | planned |
+| Gram | [Detecting Out-of-Distribution Examples with Gram Matrices](https://proceedings.mlr.press/v119/sastry20a.html) | ICML 2020 | avail [tensorflow](docs/notebooks/tensorflow/demo_gram_tf.ipynb) or  [torch](docs/notebooks/torch/demo_gram_torch.ipynb) |
 
 
 

{oodeel-0.1.1 → oodeel-0.2.0}/oodeel/__init__.py

@@ -25,4 +25,4 @@ oodeel
 -------
 """
 
-__version__ = "0.1.1"
+__version__ = "0.2.0"

{oodeel-0.1.1 → oodeel-0.2.0}/oodeel/eval/plots/features.py

@@ -168,7 +168,7 @@ def _plot_features(
     # === extract id features ===
     # features
     in_features, _ = feature_extractor.predict(in_dataset)
-    in_features = op.convert_to_numpy(op.flatten(in_features))[:max_samples]
+    in_features = op.convert_to_numpy(op.flatten(in_features[0]))[:max_samples]
 
     # labels
     in_labels = []
@@ -181,7 +181,7 @@ def _plot_features(
     if out_dataset is not None:
         # features
         out_features, _ = feature_extractor.predict(out_dataset)
-        out_features = op.convert_to_numpy(op.flatten(out_features))[:max_samples]
+        out_features = op.convert_to_numpy(op.flatten(out_features[0]))[:max_samples]
 
         # labels
         out_labels_str = np.array(["unknown"] * len(out_features))

{oodeel-0.1.1 → oodeel-0.2.0}/oodeel/eval/plots/plotly.py

@@ -82,7 +82,7 @@ def plotly_3D_features(
     # === extract id features ===
     # features
     in_features, _ = feature_extractor.predict(in_dataset)
-    in_features = op.convert_to_numpy(op.flatten(in_features))[:max_samples]
+    in_features = op.convert_to_numpy(op.flatten(in_features[0]))[:max_samples]
 
     # labels
     in_labels = []
@@ -95,7 +95,7 @@ def plotly_3D_features(
     if out_dataset is not None:
         # features
         out_features, _ = feature_extractor.predict(out_dataset)
-        out_features = op.convert_to_numpy(op.flatten(out_features))[:max_samples]
+        out_features = op.convert_to_numpy(op.flatten(out_features[0]))[:max_samples]
 
         # labels
         out_labels = np.array(["unknown"] * len(out_features))

{oodeel-0.1.1 → oodeel-0.2.0}/oodeel/extractor/feature_extractor.py

@@ -25,6 +25,7 @@ from abc import abstractmethod
 
 from ..types import Callable
 from ..types import DatasetType
+from ..types import ItemType
 from ..types import List
 from ..types import Optional
 from ..types import TensorType
@@ -91,28 +92,37 @@ class FeatureExtractor(ABC):
         raise NotImplementedError()
 
     @abstractmethod
-    def predict_tensor(self, tensor: TensorType) -> Tuple[List[TensorType], TensorType]:
-        """
-        Projects input samples "inputs" into the feature space
+    def predict_tensor(
+        self,
+        tensor: TensorType,
+        postproc_fns: Optional[List[Callable]] = None,
+    ) -> Tuple[List[TensorType], TensorType]:
+        """Get the projection of tensor in the feature space of self.model
 
         Args:
-            tensor (TensorType): input tensor
+            tensor (TensorType): input tensor (or dataset elem)
+            postproc_fns (Optional[Callable]): postprocessing function to apply to each
+                feature immediately after forward. Default to None.
 
         Returns:
-            List[TensorType], TensorType: features, logits
+            Tuple[List[TensorType], TensorType]: features, logits
         """
         raise NotImplementedError()
 
     @abstractmethod
     def predict(
         self,
-        dataset: Union[DatasetType, TensorType],
+        dataset: Union[ItemType, DatasetType],
+        postproc_fns: Optional[List[Callable]] = None,
+        **kwargs,
     ) -> Tuple[List[TensorType], dict]:
-        """
-        Projects input samples "inputs" into the feature space for a batched dataset
+        """Get the projection of the dataset in the feature space of self.model
 
         Args:
-            dataset (Union[DatasetType, TensorType]): iterable of tensor batches
+            dataset (Union[ItemType, DatasetType]): input dataset
+            postproc_fns (Optional[Callable]): postprocessing function to apply to each
+                feature immediately after forward. Default to None.
+            kwargs (dict): additional arguments not considered for prediction
 
         Returns:
             List[TensorType], dict: features and extra information (logits, labels) as a

{oodeel-0.1.1 → oodeel-0.2.0}/oodeel/extractor/keras_feature_extractor.py

@@ -150,11 +150,17 @@ class KerasFeatureExtractor(FeatureExtractor):
         return extractor
 
     @sanitize_input
-    def predict_tensor(self, tensor: TensorType) -> Tuple[List[tf.Tensor], tf.Tensor]:
+    def predict_tensor(
+        self,
+        tensor: TensorType,
+        postproc_fns: Optional[List[Callable]] = None,
+    ) -> Tuple[List[tf.Tensor], tf.Tensor]:
         """Get the projection of tensor in the feature space of self.model
 
         Args:
             tensor (TensorType): input tensor (or dataset elem)
+            postproc_fns (Optional[List[Callable]]): postprocessing function to apply to
+                each feature immediately after forward. Default to None.
 
         Returns:
             Tuple[List[tf.Tensor], tf.Tensor]: features, logits
@@ -166,8 +172,12 @@ class KerasFeatureExtractor(FeatureExtractor):
 
         # split features and logits
         logits = features.pop()
-        if len(features) == 1:
-            features = features[0]
+
+        if postproc_fns is not None:
+            features = [
+                postproc_fn(feature)
+                for feature, postproc_fn in zip(features, postproc_fns)
+            ]
 
         self._last_logits = logits
         return features, logits
@@ -179,12 +189,15 @@ class KerasFeatureExtractor(FeatureExtractor):
     def predict(
         self,
         dataset: Union[ItemType, tf.data.Dataset],
+        postproc_fns: Optional[List[Callable]] = None,
         **kwargs,
     ) -> Tuple[List[tf.Tensor], dict]:
         """Get the projection of the dataset in the feature space of self.model
 
         Args:
             dataset (Union[ItemType, tf.data.Dataset]): input dataset
+            postproc_fns (Optional[Callable]): postprocessing function to apply to each
+                feature immediately after forward. Default to None.
             kwargs (dict): additional arguments not considered for prediction
 
         Returns:
@@ -195,7 +208,7 @@ class KerasFeatureExtractor(FeatureExtractor):
 
         if isinstance(dataset, get_args(ItemType)):
             tensor = TFDataHandler.get_input_from_dataset_item(dataset)
-            features, logits = self.predict_tensor(tensor)
+            features, logits = self.predict_tensor(tensor, postproc_fns)
 
             # Get labels if dataset is a tuple/list
             if isinstance(dataset, (list, tuple)):
@@ -207,10 +220,8 @@ class KerasFeatureExtractor(FeatureExtractor):
             contains_labels = TFDataHandler.get_item_length(dataset) > 1
             for elem in dataset:
                 tensor = TFDataHandler.get_input_from_dataset_item(elem)
-                features_batch, logits_batch = self.predict_tensor(tensor)
-                # concatenate features
-                if len(features) == 1:
-                    features_batch = [features_batch]
+                features_batch, logits_batch = self.predict_tensor(tensor, postproc_fns)
+
                 for i, f in enumerate(features_batch):
                     features[i] = (
                         f
@@ -234,10 +245,6 @@ class KerasFeatureExtractor(FeatureExtractor):
 
         # store extra information in a dict
         info = dict(labels=labels, logits=logits)
-
-        if len(features) == 1:
-            features = features[0]
-
         return features, info
 
     def get_weights(self, layer_id: Union[int, str]) -> List[tf.Tensor]:

{oodeel-0.1.1 → oodeel-0.2.0}/oodeel/extractor/torch_feature_extractor.py

@@ -181,12 +181,17 @@ class TorchFeatureExtractor(FeatureExtractor):
 
     @sanitize_input
     def predict_tensor(
-        self, x: TensorType, detach: bool = True
+        self,
+        x: TensorType,
+        postproc_fns: Optional[List[Callable]] = None,
+        detach: bool = True,
     ) -> Tuple[List[torch.Tensor], torch.Tensor]:
         """Get the projection of tensor in the feature space of self.model
 
         Args:
             x (TensorType): input tensor (or dataset elem)
+            postproc_fns (Optional[List[Callable]]): postprocessing function to apply to
+                each feature immediately after forward. Default to None.
             detach (bool): if True, return features detached from the computational
                 graph. Defaults to True.
 
@@ -206,8 +211,12 @@ class TorchFeatureExtractor(FeatureExtractor):
 
         # split features and logits
         logits = features.pop()
-        if len(features) == 1:
-            features = features[0]
+
+        if postproc_fns is not None:
+            features = [
+                postproc_fn(feature)
+                for feature, postproc_fn in zip(features, postproc_fns)
+            ]
 
         self._last_logits = logits
         return features, logits
@@ -215,6 +224,7 @@ class TorchFeatureExtractor(FeatureExtractor):
     def predict(
         self,
         dataset: Union[DataLoader, ItemType],
+        postproc_fns: Optional[List[Callable]] = None,
         detach: bool = True,
         **kwargs,
     ) -> Tuple[List[torch.Tensor], dict]:
@@ -222,6 +232,8 @@ class TorchFeatureExtractor(FeatureExtractor):
 
         Args:
             dataset (Union[DataLoader, ItemType]): input dataset
+            postproc_fns (Optional[List[Callable]]): postprocessing function to apply to
+                each feature immediately after forward. Default to None.
             detach (bool): if True, return features detached from the computational
                 graph. Defaults to True.
             kwargs (dict): additional arguments not considered for prediction
@@ -234,7 +246,7 @@ class TorchFeatureExtractor(FeatureExtractor):
 
         if isinstance(dataset, get_args(ItemType)):
             tensor = TorchDataHandler.get_input_from_dataset_item(dataset)
-            features, logits = self.predict_tensor(tensor, detach=detach)
+            features, logits = self.predict_tensor(tensor, postproc_fns, detach=detach)
 
             # Get labels if dataset is a tuple/list
             if isinstance(dataset, (list, tuple)) and len(dataset) > 1:
@@ -248,11 +260,8 @@ class TorchFeatureExtractor(FeatureExtractor):
             for elem in dataset:
                 tensor = TorchDataHandler.get_input_from_dataset_item(elem)
                 features_batch, logits_batch = self.predict_tensor(
-                    tensor, detach=detach
+                    tensor, postproc_fns, detach=detach
                 )
-                # concatenate features
-                if len(features) == 1:
-                    features_batch = [features_batch]
                 for i, f in enumerate(features_batch):
                     features[i] = (
                         f if features[i] is None else torch.cat([features[i], f], dim=0)
@@ -274,10 +283,6 @@ class TorchFeatureExtractor(FeatureExtractor):
 
         # store extra information in a dict
         info = dict(labels=labels, logits=logits)
-
-        if len(features) == 1:
-            features = features[0]
-
         return features, info
 
     def get_weights(self, layer_id: Union[str, int]) -> List[torch.Tensor]:

{oodeel-0.1.1 → oodeel-0.2.0}/oodeel/methods/__init__.py

@@ -23,9 +23,10 @@
 from .dknn import DKNN
 from .energy import Energy
 from .entropy import Entropy
+from .gram import Gram
 from .mahalanobis import Mahalanobis
 from .mls import MLS
 from .odin import ODIN
 from .vim import VIM
 
-__all__ = ["MLS", "DKNN", "ODIN", "Energy", "VIM", "Mahalanobis", "Entropy"]
+__all__ = ["MLS", "DKNN", "ODIN", "Energy", "VIM", "Mahalanobis", "Entropy", "Gram"]

{oodeel-0.1.1 → oodeel-0.2.0}/oodeel/methods/base.py

@@ -52,11 +52,13 @@ class OODBaseDetector(ABC):
         self,
         use_react: bool = False,
         react_quantile: float = 0.8,
+        postproc_fns: List[Callable] = None,
     ):
         self.feature_extractor: FeatureExtractor = None
         self.use_react = use_react
         self.react_quantile = react_quantile
         self.react_threshold = None
+        self.postproc_fns = self._sanitize_posproc_fns(postproc_fns)
 
     @abstractmethod
     def _score_tensor(self, inputs: TensorType) -> np.ndarray:
@@ -66,18 +68,43 @@ class OODBaseDetector(ABC):
 
         Args:
             inputs (TensorType): tensor to score
-
         Returns:
             Tuple[TensorType]: OOD scores, predicted logits
         """
         raise NotImplementedError()
 
+    def _sanitize_posproc_fns(
+        self,
+        postproc_fns: Union[List[Callable], None],
+    ) -> List[Callable]:
+        """Sanitize postproc fns used at each layer output of the feature extractor.
+
+        Args:
+            postproc_fns (Optional[List[Callable]], optional): List of postproc
+                functions, one per output layer. Defaults to None.
+
+        Returns:
+            List[Callable]: Sanitized postproc_fns list
+        """
+        if postproc_fns is not None:
+            assert len(postproc_fns) == len(
+                self.output_layers_id
+            ), "len of postproc_fns and output_layers_id must match"
+
+            def identity(x):
+                return x
+
+            postproc_fns = [identity if fn is None else fn for fn in postproc_fns]
+
+        return postproc_fns
+
     def fit(
         self,
         model: Callable,
         fit_dataset: Optional[Union[ItemType, DatasetType]] = None,
         feature_layers_id: List[Union[int, str]] = [],
         input_layer_id: Optional[Union[int, str]] = None,
+        **kwargs,
     ) -> None:
         """Prepare the detector for scoring:
         * Constructs the feature extractor based on the model
@@ -133,7 +160,7 @@ class OODBaseDetector(ABC):
         )
 
         if fit_dataset is not None:
-            self._fit_to_dataset(fit_dataset)
+            self._fit_to_dataset(fit_dataset, **kwargs)
 
     def _load_feature_extractor(
         self,
@@ -243,7 +270,9 @@ class OODBaseDetector(ABC):
     def compute_react_threshold(self, model: Callable, fit_dataset: DatasetType):
         penult_feat_extractor = self._load_feature_extractor(model, [-2])
         unclipped_features, _ = penult_feat_extractor.predict(fit_dataset)
-        self.react_threshold = self.op.quantile(unclipped_features, self.react_quantile)
+        self.react_threshold = self.op.quantile(
+            unclipped_features[0], self.react_quantile
+        )
 
     def __call__(self, inputs: Union[ItemType, DatasetType]) -> np.ndarray:
         """

{oodeel-0.1.1 → oodeel-0.2.0}/oodeel/methods/dknn.py

@@ -58,7 +58,7 @@ class DKNN(OODBaseDetector):
             fit_dataset: input dataset (ID) to construct the index with.
         """
         fit_projected, _ = self.feature_extractor.predict(fit_dataset)
-        fit_projected = self.op.convert_to_numpy(fit_projected)
+        fit_projected = self.op.convert_to_numpy(fit_projected[0])
         fit_projected = fit_projected.reshape(fit_projected.shape[0], -1)
         norm_fit_projected = self._l2_normalization(fit_projected)
         self.index = faiss.IndexFlatL2(norm_fit_projected.shape[1])
@@ -77,7 +77,7 @@ class DKNN(OODBaseDetector):
         """
 
         input_projected, _ = self.feature_extractor.predict_tensor(inputs)
-        input_projected = self.op.convert_to_numpy(input_projected)
+        input_projected = self.op.convert_to_numpy(input_projected[0])
         input_projected = input_projected.reshape(input_projected.shape[0], -1)
         norm_input_projected = self._l2_normalization(input_projected)
         scores, _ = self.index.search(norm_input_projected, self.nearest)