safetycage 0.0.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,19 @@
1
+ Copyright (c) 2026 [OUR AUTHORS ...]
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining a copy
4
+ of this software and associated documentation files (the "Software"), to deal
5
+ in the Software without restriction, including without limitation the rights
6
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
7
+ copies of the Software, and to permit persons to whom the Software is
8
+ furnished to do so, subject to the following conditions:
9
+
10
+ The above copyright notice and this permission notice shall be included in all
11
+ copies or substantial portions of the Software.
12
+
13
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
15
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
16
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
17
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
18
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
19
+ SOFTWARE.
@@ -0,0 +1,88 @@
1
+ Metadata-Version: 2.4
2
+ Name: safetycage
3
+ Version: 0.0.1
4
+ Summary: The official safety cage package for misclassification detection in predictive models.
5
+ Author: Pål Vegard Bun Johnsen, Joel Bjervig, Julia Qiu
6
+ Author-email: Pål Vegard Bun Johnsen <pal.johnsen@sintef.no>, Joel Bjervig <Joel.Bjervig@sintef.no>, Julia Qiu <Julia.Qiu@sintef.no>
7
+ License-Expression: MIT
8
+ License-File: LICENSE
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: Operating System :: OS Independent
11
+ Requires-Dist: joblib==1.3.2
12
+ Requires-Dist: matplotlib==3.8.2
13
+ Requires-Dist: numpy>=1.25.2
14
+ Requires-Dist: omegaconf==2.3.0
15
+ Requires-Dist: pyrootutils==1.0.4
16
+ Requires-Dist: scikit-learn==1.1.3
17
+ Requires-Dist: scipy==1.9.3
18
+ Requires-Dist: statsmodels==0.13.5
19
+ Requires-Dist: tqdm>=4.67.3
20
+ Requires-Python: ==3.11.7
21
+ Project-URL: Homepage, https://github.com/safety-cage/safetycage
22
+ Project-URL: Issues, https://github.com/safety-cage/safetycage/issues
23
+ Description-Content-Type: text/markdown
24
+
25
+ # safetycage(-pypi) (SafetyCage Test Package)
26
+
27
+ This is the git repo that hosts the development and testing of the safetycage python package (src & tests respectfully).
28
+
29
+ ## Background?
30
+
31
+ ## Description
32
+ Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
33
+
34
+ ## Visuals
35
+ - could put a gif on installing and using the package
36
+
37
+ ## Installation
38
+ TBA
39
+
40
+ ## Requirements
41
+ - Python 3.11.7
42
+
43
+ ## Usage
44
+ Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
45
+
46
+ ## Support
47
+ Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
48
+
49
+ ## Integrate with your tools
50
+
51
+ * [Set up project integrations](https://gitlab.sintef.no/aai-internal/safetycage-pypi/-/settings/integrations)
52
+
53
+ ## Collaborate with your team
54
+
55
+ * [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
56
+ * [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
57
+ * [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
58
+ * [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
59
+ * [Set auto-merge](https://docs.gitlab.com/user/project/merge_requests/auto_merge/)
60
+
61
+ ## Test and Deploy
62
+
63
+ Use the built-in continuous integration in GitLab.
64
+
65
+ * [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/)
66
+ * [Analyze your code for known vulnerabilities with Static Application Security Testing (SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
67
+ * [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
68
+ * [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
69
+ * [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
70
+
71
+ ## Roadmap
72
+ If you have ideas for releases in the future, it is a good idea to list them in the README.
73
+
74
+ ## Contributing
75
+ State if you are open to contributions and what your requirements are for accepting them.
76
+
77
+ For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
78
+
79
+ You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
80
+
81
+ ## Authors and acknowledgment
82
+ Show your appreciation to those who have contributed to the project.
83
+
84
+ ## License
85
+ MIT License
86
+
87
+ ## Project status
88
+ If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
@@ -0,0 +1,64 @@
1
+ # safetycage(-pypi) (SafetyCage Test Package)
2
+
3
+ This is the git repo that hosts the development and testing of the safetycage python package (src & tests respectfully).
4
+
5
+ ## Background?
6
+
7
+ ## Description
8
+ Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
9
+
10
+ ## Visuals
11
+ - could put a gif on installing and using the package
12
+
13
+ ## Installation
14
+ TBA
15
+
16
+ ## Requirements
17
+ - Python 3.11.7
18
+
19
+ ## Usage
20
+ Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
21
+
22
+ ## Support
23
+ Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
24
+
25
+ ## Integrate with your tools
26
+
27
+ * [Set up project integrations](https://gitlab.sintef.no/aai-internal/safetycage-pypi/-/settings/integrations)
28
+
29
+ ## Collaborate with your team
30
+
31
+ * [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
32
+ * [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
33
+ * [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
34
+ * [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
35
+ * [Set auto-merge](https://docs.gitlab.com/user/project/merge_requests/auto_merge/)
36
+
37
+ ## Test and Deploy
38
+
39
+ Use the built-in continuous integration in GitLab.
40
+
41
+ * [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/)
42
+ * [Analyze your code for known vulnerabilities with Static Application Security Testing (SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
43
+ * [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
44
+ * [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
45
+ * [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
46
+
47
+ ## Roadmap
48
+ If you have ideas for releases in the future, it is a good idea to list them in the README.
49
+
50
+ ## Contributing
51
+ State if you are open to contributions and what your requirements are for accepting them.
52
+
53
+ For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
54
+
55
+ You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
56
+
57
+ ## Authors and acknowledgment
58
+ Show your appreciation to those who have contributed to the project.
59
+
60
+ ## License
61
+ MIT License
62
+
63
+ ## Project status
64
+ If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
@@ -0,0 +1,47 @@
1
+ [build-system]
2
+ requires = ["uv_build >= 0.10.0, <0.11.0"]
3
+ build-backend = "uv_build"
4
+
5
+ [project]
6
+ name = "safetycage"
7
+ version = "0.0.1"
8
+ authors = [
9
+ { name="Pål Vegard Bun Johnsen", email="pal.johnsen@sintef.no" },
10
+ { name="Joel Bjervig", email="Joel.Bjervig@sintef.no" },
11
+ { name="Julia Qiu", email="Julia.Qiu@sintef.no" },
12
+ ]
13
+ description = "The official safety cage package for misclassification detection in predictive models."
14
+ readme = "README.md"
15
+ requires-python = "==3.11.7"
16
+ classifiers = [
17
+ "Programming Language :: Python :: 3",
18
+ "Operating System :: OS Independent",
19
+ ]
20
+ license = "MIT"
21
+ license-files = ["LICEN[CS]E*"]
22
+ dependencies = [
23
+ "joblib==1.3.2",
24
+ "matplotlib==3.8.2",
25
+ "numpy>=1.25.2",
26
+ "omegaconf==2.3.0",
27
+ "pyrootutils==1.0.4",
28
+ "scikit-learn==1.1.3",
29
+ "scipy==1.9.3",
30
+ "statsmodels==0.13.5",
31
+ "tqdm>=4.67.3",
32
+ ]
33
+
34
+ [project.urls]
35
+ Homepage = "https://github.com/safety-cage/safetycage"
36
+ Issues = "https://github.com/safety-cage/safetycage/issues"
37
+
38
+ [tool.uv]
39
+ package = true
40
+
41
+ [tool.uv.sources]
42
+ safetycage-testing = { workspace = true }
43
+
44
+ [dependency-groups]
45
+ dev = [
46
+ "safetycage-testing",
47
+ ]
@@ -0,0 +1,101 @@
1
+ from abc import ABC, abstractmethod
2
+ from typing import List, Any
3
+ from pathlib import Path
4
+
5
+ class DataModule(ABC):
6
+ """
7
+ Abstract base class for handling batched data regardless of source.
8
+
9
+ This class defines a common interface for loading, transforming, and splitting
10
+ data for the use by safety cage methods and model modules. It also provides
11
+ access to dataset properties such as the class labels and dataset size.
12
+
13
+ Subclasses must implement methods for data loading, preprocessing, and splitting.
14
+ """
15
+
16
+ def __init__(
17
+ self,
18
+ data_dir:str=None,
19
+ from_cache:bool = False,
20
+ batch_size: int = 32,
21
+ device:str="cpu"
22
+ ) -> None:
23
+ """
24
+ Initialize the data module.
25
+
26
+ Sets configuration parameters for loading and processing data, such as
27
+ caching behavior, batch size, and device placement.
28
+
29
+ The data directory is created if it does not already exist.
30
+
31
+ Args:
32
+ data_dir (str, optional): Path to the data directory. (default: None).
33
+ from_cache (bool, optional): Whether to load data from cached files. (default: False).
34
+ batch_size (int, optional): Number of samples per batch. (default: 32)
35
+ device (str, optional): Device used for data processing (e.g., "cpu", "cuda") (default: "cpu").
36
+ """
37
+
38
+ # Data parameters
39
+ self.from_cache = from_cache
40
+ self.batch_size = batch_size
41
+
42
+ # Device configuration
43
+ self.device = device
44
+
45
+ self.data_dir = Path(data_dir)
46
+ self.data_dir.mkdir(parents=True, exist_ok=True)
47
+
48
+
49
+ @property
50
+ @abstractmethod
51
+ def num_classes(self) -> int:
52
+ """Returns the number of classes in the dataset"""
53
+ raise NotImplementedError("Subclasses should implement this method to return the number of classes.")
54
+
55
+
56
+ @property
57
+ @abstractmethod
58
+ def classes(self) -> List[Any]:
59
+ """Returns the class names in the dataset"""
60
+ raise NotImplementedError("Subclasses should implement this method to return the class names.")
61
+
62
+
63
+ @property
64
+ @abstractmethod
65
+ def dataset_name(self) -> str:
66
+ """Get the name of the dataset."""
67
+ raise NotImplementedError("Subclasses should implement this method to return the dataset name.")
68
+
69
+
70
+ @abstractmethod
71
+ def setup(self) -> None:
72
+ """Setup the data module, should be implemented by subclasses"""
73
+ raise NotImplementedError("Subclasses should implement this method to setup the data module.")
74
+
75
+
76
+ @abstractmethod
77
+ def _load_data(self, filepath:str) -> None:
78
+ """Load data from the source or cache"""
79
+ raise NotImplementedError("Subclasses should implement this method to load data.")
80
+
81
+
82
+ @abstractmethod
83
+ def _transform(self,x,y):
84
+ """Transform the data, should be implemented by subclasses"""
85
+ raise NotImplementedError("Subclasses should implement this method to transform the data.")
86
+
87
+
88
+ @abstractmethod
89
+ def _split(self,x,y,split):
90
+ """Split the data into training and validation sets."""
91
+ raise NotImplementedError("Subclasses should implement this method to split the data.")
92
+
93
+ @abstractmethod
94
+ def to_joblib(self, path: str) -> None:
95
+ """Save the data module to a joblib file."""
96
+ raise NotImplementedError("Subclasses should implement this method to save the data module to a joblib file.")
97
+
98
+ @abstractmethod
99
+ def from_joblib(self, path: str) -> None:
100
+ """Load the data module from a joblib file."""
101
+ raise NotImplementedError("Subclasses should implement this method to load the data module from a joblib file.")
@@ -0,0 +1,83 @@
1
+ from abc import ABC, abstractmethod
2
+ from typing import Tuple, Union, List, Any, Dict
3
+ import numpy as np
4
+ import omegaconf
5
+
6
+ class ModelModule(ABC):
7
+ """
8
+ Abstract base class for handling models used by safety cage methods.
9
+
10
+ This class defines a common interface for generating predictions, probabilities,
11
+ and intermediate activations from a model. It enables safety cage methods to
12
+ interact with different model types in a consistent way.
13
+
14
+ Subclasses must implement methods for prediction, probability estimation, and
15
+ activation extraction.
16
+ """
17
+
18
+ def __init__(
19
+ self,
20
+ selected_layers:Union[str, List[str]],
21
+ use_onehot_encoder:bool,
22
+ model:Any,
23
+ **kwargs
24
+ ):
25
+ """
26
+ Initialize the model module.
27
+
28
+ Sets configuration parameters for interacting with the model, such as
29
+ which layers to extract activations from and how outputs are formatted.
30
+
31
+ Args:
32
+ model (Any, optional): Underlying model object. (default: None).
33
+ selected_layers (list[str], optional): List of layer names used for feature extraction. (default: None).
34
+ use_onehot_encoder (bool, optional): Whether model outputs are one-hot encoded. (default: False).
35
+ device (str, optional): Device used for model computation (e.g., "cpu", "cuda"). (default: "cpu").
36
+ """
37
+ super(ModelModule, self).__init__()
38
+
39
+ self.model = model
40
+ self.use_onehot_encoder = use_onehot_encoder
41
+
42
+ # Handle different types of selected_layers input
43
+ if isinstance(selected_layers, str):
44
+ self.selected_layers = [selected_layers]
45
+ elif isinstance(selected_layers, (list, omegaconf.listconfig.ListConfig)):
46
+ if all(isinstance(layer, str) for layer in selected_layers):
47
+ self.selected_layers = list(selected_layers) # Convert to regular list if it's ListConfig
48
+ else:
49
+ raise ValueError("All elements in selected_layers must be strings")
50
+ else:
51
+ raise ValueError(f"selected_layers must be a string or list of strings, got {type(selected_layers)}")
52
+
53
+
54
+ @abstractmethod
55
+ def _get_predictions(self, x: np.ndarray) -> np.ndarray:
56
+ """Get model predictions for input x."""
57
+ raise NotImplementedError("Implement based on your model architecture")
58
+
59
+
60
+ @abstractmethod
61
+ def _get_activations(self, x: np.ndarray) -> List[np.ndarray]:
62
+ """Calculate activations for each layer given input x."""
63
+ # This method should be implemented based on your specific model architecture
64
+ raise NotImplementedError("Implement based on your model architecture")
65
+
66
+
67
+ @abstractmethod
68
+ def _get_pre_activations(self, x: np.ndarray) -> List[np.ndarray]:
69
+ """Calculate pre-activation values for each layer given input x."""
70
+ raise NotImplementedError("Implement based on your model architecture")
71
+
72
+
73
+ @abstractmethod
74
+ def _calc_model_shape(self) -> Dict[str,int]:
75
+ """
76
+ Get the shape of each layer in the model.
77
+ Returns: List of integers representing the number of neurons in each layer
78
+ """
79
+ raise NotImplementedError("Implement based on your model architecture")
80
+
81
+ if __name__ == '__main__':
82
+ #Quick test to confirm it builds
83
+ model_module = ModelModule()
@@ -0,0 +1,271 @@
1
+ from typing import Any
2
+ import pyrootutils
3
+ from abc import ABC, abstractmethod
4
+ import numpy as np
5
+ import os
6
+ import json
7
+ import joblib
8
+ pyrootutils.setup_root(__file__, indicator=".project-root", pythonpath=True)
9
+
10
+ class SafetyCage(ABC):
11
+ """
12
+ Abstract base class for safety cage methods.
13
+
14
+ A safety cage detects misclassification in classification tasks. Safety cage methods
15
+ evaluate predictive models by computing statistics that indicate whether a sample is
16
+ likely to be misclassified and comparing to some optimal threshold statistic, called "alpha".
17
+ Concrete base classes define how these statistics are computed, how predictions are
18
+ evaluated, and how to find the optimal alpha value.
19
+
20
+ Subclasses must implement training, prediction, and the method to compute the statistic.
21
+ Implementations on how to flag misclassifications, find the best threshold, and save/load the
22
+ safetycage are provided for simplicity, but should be overridden if they do not meet the requirements
23
+ of the specific safety cage method.
24
+
25
+ Attributes:
26
+ model_module: Reference to model module object for making predictions.
27
+ data_module: Reference to data module object for handling data.
28
+ num_classes (int): Number of classes. Retrieved from the data module.
29
+ selected_classes (list): List of classes. Retrieved from the data module.
30
+ alpha (float): Threshold statistic value used for flagging misclassifications.
31
+ layer_params (dict, optional): Dictionary to store parameters for specific layers, if needed by
32
+ safety cage.
33
+ leq (bool, optional): If True, samples with statistic less than or equal to alpha are flagged as
34
+ misclassified. Only required if the default flag method is used.
35
+ """
36
+
37
+ def __init__(
38
+ self,
39
+ model_module,
40
+ data_module: Any,
41
+ **kwargs
42
+ ) -> None:
43
+ """
44
+ Initialize the safety cage.
45
+
46
+ Stores references to the model and data modules and initializes shared attributes
47
+ used across all safety cage methods.
48
+
49
+ Args:
50
+ model_module: Model module used for predictions and activations.
51
+ data_module (Any): Data module providing datasets and class information.
52
+ """
53
+
54
+ self.model_module = model_module
55
+ self.data_module = data_module
56
+
57
+ self.num_classes = data_module.num_classes
58
+ self.selected_classes = data_module.classes
59
+
60
+ self.alpha = None
61
+
62
+ #Train the parameters of the specified SafetyCage
63
+ @abstractmethod
64
+ def train_cage(self) -> None:
65
+ """
66
+ Train the safety cage.
67
+
68
+ Learns parameters from training data that are later used to evaluate
69
+ whether predictions are reliable.
70
+
71
+ Returns:
72
+ None
73
+ """
74
+ pass
75
+
76
+ #Apply the SafetyCage on unseen test samples
77
+ @abstractmethod
78
+ def predict(self, x, y) -> None:
79
+ """
80
+ Evaluate input samples using the trained safety cage.
81
+
82
+ Computes statistics that indicate how likely each prediction is a misclassification.
83
+
84
+ Args:
85
+ x: Input data samples
86
+ y: True labels
87
+
88
+ Returns:
89
+ numpy.ndarray: Computed statistics for each sample
90
+ """
91
+ pass
92
+
93
+ #Compute the statistics to evaluate whether each test sample is wrongly predicted
94
+ @abstractmethod
95
+ def _compute_statistics(self, x, y):
96
+ """
97
+ Compute per-sample statistics used to evaluate prediction reliability.
98
+
99
+ Args:
100
+ x: Input data samples
101
+ y: True labels
102
+
103
+ Returns:
104
+ numpy.ndarray: Per-sample statistics
105
+ """
106
+ pass
107
+
108
+ #Flag predictions as being correct (0) or wrong (1)
109
+ def flag(self, statistics: float | np.ndarray, alpha: float | None = None) -> float | np.ndarray:
110
+ """
111
+ Flag samples with probability less than or equal (safetycage.leq = True) to alpha as incorrect
112
+ or probability more than or equal (safetycage.leq = False) to alpha as incorrect.
113
+
114
+ This method identifies samples where the maximum/minimum probability is below/above a
115
+ specified threshold (alpha), marking them as potentially incorrect classifications.
116
+
117
+ *Requires safetycage.leq to be defined, not None.*
118
+
119
+ Args:
120
+ statistics (numpy.ndarray): Array of probability values to evaluate
121
+ alpha (float): Threshold value for flagging samples (0 to 1)
122
+ Returns:
123
+ numpy.ndarray: Boolean array where True indicates probabilities below/above the alpha threshold
124
+ depending on safetycage.leq
125
+ """
126
+
127
+ # Check priority of alpha parameter
128
+ if alpha is None:
129
+ # If not provided as input, try to use self.alpha
130
+ if hasattr(self, 'alpha') and self.alpha is not None:
131
+ alpha = self.alpha
132
+ else:
133
+ # If neither source is available, raise an error
134
+ raise ValueError("Missing alpha parameter: must be provided as input or set as class attribute")
135
+
136
+ if self.leq:
137
+ flags = statistics <= alpha
138
+ elif not self.leq and self.leq is not None:
139
+ flags = statistics >= alpha
140
+ else:
141
+ raise ValueError("Define safetycage.leq")
142
+
143
+ return flags
144
+
145
+ def find_best_threshold_flag(self, y_true, y_probs, metric_fn, greater_is_better=True) -> float | np.ndarray:
146
+ """
147
+ Find the optimal threshold for flagging samples by calling self.flag().
148
+
149
+ Evaluates thresholds t (from 1000 samples between min to max) and selects the one that maximizes
150
+ the given metric.
151
+
152
+ Args:
153
+ y_true (numpy.ndarray): Ground-truth misclassification labels
154
+ y_probs (numpy.ndarray): Computed statistics or probabilities
155
+ metric_fn (callable): Function to evaluate performance
156
+ greater_is_better (bool, optional): Whether greater metric values are better (default: True)
157
+
158
+ Returns:
159
+ dict: Dictionary containing the optimal threshold and the corresponding best metric value
160
+ """
161
+ if greater_is_better:
162
+ compare = lambda x, y: x > y
163
+ else:
164
+ compare = lambda x, y: x < y
165
+
166
+ best_metric = -np.inf
167
+ best_alpha = -np.inf
168
+ for t in np.linspace(min(y_probs), max(y_probs), num=1000):
169
+
170
+ flag = self.flag(y_probs, t)
171
+
172
+ # Flag is true when misclassification occurs
173
+ tps = np.sum(flag & y_true)
174
+ fps = np.sum(flag & (1 - y_true))
175
+
176
+ total_pos = y_true.sum()
177
+ total_neg = y_true.size - total_pos
178
+
179
+ fns = total_pos - tps
180
+ tns = total_neg - fps
181
+
182
+ metric = metric_fn(TP=tps, TN=tns, FP=fps, FN=fns)
183
+
184
+ if compare(metric, best_metric):
185
+ best_metric = metric
186
+ best_alpha = t
187
+
188
+ return {
189
+ "alpha_opt": best_alpha,
190
+ "metric_max": best_metric,
191
+ }
192
+
193
+ def save_cage(self, path):
194
+ """
195
+ Save the safety cage parameters, alpha and/or layer_params, to a specified folder path
196
+ in a file called "parameters.json" and/or "parameters.joblib".
197
+
198
+ Args:
199
+ path (str): Folder path where the safety cage parameters should be saved
200
+ """
201
+
202
+ os.makedirs(path, exist_ok=True)
203
+
204
+ if getattr(self, "alpha", None) is not None:
205
+ if getattr(self, "layer_params", None) is not None:
206
+
207
+ parameters_path = os.path.join(path, 'parameters.joblib')
208
+ parameters = {
209
+ 'alpha': self.alpha,
210
+ 'layer_params': self.layer_params
211
+ }
212
+ joblib.dump(parameters, parameters_path)
213
+
214
+ else:
215
+
216
+ parameters_path = os.path.join(path, 'parameters.json')
217
+
218
+ with open(parameters_path, 'w') as f:
219
+ json.dump({'alpha': str(self.alpha)}, f)
220
+
221
+ else:
222
+ raise ValueError("alpha is not set.")
223
+
224
+ def load_cage(self, path):
225
+ """
226
+ Load the safety cage parameters, alpha and/or layer_params, from a specified folder path.
227
+
228
+ Looks for a parameters.json or parameters.joblib file respectively.
229
+ Load cage assumes only one of these two files are present in the folder, and gives priority
230
+ to parameters.joblib if both are present.
231
+
232
+ Args:
233
+ path (str): Path to the parameters file or directory where the safety cage parameters should be loaded.
234
+ """
235
+ joblib_path = os.path.join(path, "parameters.joblib")
236
+ json_path = os.path.join(path, "parameters.json")
237
+
238
+ if os.path.isfile(path):
239
+ if path.endswith(".joblib"):
240
+
241
+ parameters = joblib.load(path)
242
+ self.alpha = parameters["alpha"]
243
+ self.layer_params = parameters["layer_params"]
244
+
245
+ elif path.endswith(".json"):
246
+
247
+ with open(path, "r") as f:
248
+ data = json.load(f)
249
+ self.alpha = float(data["alpha"])
250
+
251
+ else:
252
+ raise ValueError(f"Unsupported file type: {path}")
253
+
254
+ elif os.path.exists(joblib_path):
255
+
256
+ parameters = joblib.load(joblib_path)
257
+
258
+ # Restore parameters
259
+ self.alpha = parameters['alpha']
260
+ self.layer_params = parameters['layer_params']
261
+
262
+ elif os.path.exists(json_path):
263
+
264
+ with open(json_path, 'r') as f:
265
+ data = json.load(f)
266
+ self.alpha = float(data['alpha'])
267
+ else:
268
+ raise FileNotFoundError(f"Safety cage parameters file not found at {path}")
269
+
270
+ if __name__ == "__main__":
271
+ SafetyCage(None, None, None)
File without changes