robustmargin 0.2.1__tar.gz → 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- robustmargin-0.3.0/PKG-INFO +537 -0
- robustmargin-0.3.0/README.md +500 -0
- {robustmargin-0.2.1 → robustmargin-0.3.0}/pyproject.toml +12 -1
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/__init__.py +18 -6
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/censoring.py +188 -36
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/empirical.py +2 -2
- robustmargin-0.3.0/src/robustmargin/intervals.py +2338 -0
- robustmargin-0.3.0/src/robustmargin/models.py +4498 -0
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/ratio.py +75 -13
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/report.py +140 -30
- robustmargin-0.3.0/src/robustmargin/schema/scorecard-v2.json +1100 -0
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/testing.py +291 -21
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/threshold.py +329 -95
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/types.py +460 -21
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/validation.py +487 -54
- robustmargin-0.3.0/src/robustmargin.egg-info/PKG-INFO +537 -0
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin.egg-info/SOURCES.txt +2 -1
- robustmargin-0.2.1/PKG-INFO +0 -285
- robustmargin-0.2.1/README.md +0 -250
- robustmargin-0.2.1/src/robustmargin/intervals.py +0 -861
- robustmargin-0.2.1/src/robustmargin/models.py +0 -2585
- robustmargin-0.2.1/src/robustmargin.egg-info/PKG-INFO +0 -285
- {robustmargin-0.2.1 → robustmargin-0.3.0}/LICENSE +0 -0
- {robustmargin-0.2.1 → robustmargin-0.3.0}/setup.cfg +0 -0
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/py.typed +0 -0
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/schema/scorecard-v1.json +0 -0
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin.egg-info/dependency_links.txt +0 -0
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin.egg-info/requires.txt +0 -0
- {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin.egg-info/top_level.txt +0 -0
|
@@ -0,0 +1,537 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: robustmargin
|
|
3
|
+
Version: 0.3.0
|
|
4
|
+
Summary: Robustness margins for AI safety evaluation: how much stress an AI system takes before it fails, with a confidence interval.
|
|
5
|
+
Author: robustmargin contributors
|
|
6
|
+
License-Expression: Apache-2.0
|
|
7
|
+
Project-URL: Homepage, https://github.com/robustmargin/robustmargin
|
|
8
|
+
Project-URL: Source, https://github.com/robustmargin/robustmargin
|
|
9
|
+
Project-URL: Changelog, https://github.com/robustmargin/robustmargin/blob/main/CHANGELOG.md
|
|
10
|
+
Project-URL: Issues, https://github.com/robustmargin/robustmargin/issues
|
|
11
|
+
Keywords: ai-safety,ai-evaluation,robustness,stress-testing,autonomous-systems,dose-response,confidence-intervals,statistics
|
|
12
|
+
Classifier: Development Status :: 3 - Alpha
|
|
13
|
+
Classifier: Intended Audience :: Science/Research
|
|
14
|
+
Classifier: Operating System :: OS Independent
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
18
|
+
Classifier: Topic :: Scientific/Engineering
|
|
19
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
20
|
+
Classifier: Topic :: Scientific/Engineering :: Mathematics
|
|
21
|
+
Classifier: Typing :: Typed
|
|
22
|
+
Requires-Python: >=3.11
|
|
23
|
+
Description-Content-Type: text/markdown
|
|
24
|
+
License-File: LICENSE
|
|
25
|
+
Requires-Dist: numpy>=1.26
|
|
26
|
+
Requires-Dist: scipy>=1.11.1
|
|
27
|
+
Requires-Dist: statsmodels>=0.14
|
|
28
|
+
Provides-Extra: dev
|
|
29
|
+
Requires-Dist: pytest>=8.0; extra == "dev"
|
|
30
|
+
Requires-Dist: pytest-cov>=5.0; extra == "dev"
|
|
31
|
+
Requires-Dist: hypothesis>=6.100; extra == "dev"
|
|
32
|
+
Requires-Dist: ruff>=0.5; extra == "dev"
|
|
33
|
+
Requires-Dist: mypy>=1.10; extra == "dev"
|
|
34
|
+
Requires-Dist: pre-commit>=3.5; extra == "dev"
|
|
35
|
+
Requires-Dist: jsonschema>=4.20; extra == "dev"
|
|
36
|
+
Dynamic: license-file
|
|
37
|
+
|
|
38
|
+
# robustmargin
|
|
39
|
+
[](https://doi.org/10.5281/zenodo.23585829)
|
|
40
|
+
|
|
41
|
+
**Robustness margins for AI safety evaluation: how much stress an AI system takes before it
|
|
42
|
+
fails, with a confidence interval.**
|
|
43
|
+
|
|
44
|
+
robustmargin is a robustness calculation engine. AI systems are usually evaluated by a success
|
|
45
|
+
rate under nominal conditions; a safety case needs a different number: how far conditions can
|
|
46
|
+
degrade before the system starts to fail, and how sure we are of that point. robustmargin
|
|
47
|
+
estimates that robustness margin, with a confidence interval, from stress tests at increasing
|
|
48
|
+
severity, scored pass/fail or on a continuous scale.
|
|
49
|
+
|
|
50
|
+
It is model-agnostic and framework-free: it sees only severities and outcomes (success counts or
|
|
51
|
+
real-valued scores), and needs no GPU and no ML framework. That makes it usable wherever
|
|
52
|
+
robustness and safety carry weight:
|
|
53
|
+
|
|
54
|
+
- **AI safety research**: measure how much perturbation a model or policy tolerates, and whether
|
|
55
|
+
a training intervention moved that margin.
|
|
56
|
+
- **Physical AI**: learned policies evaluated under graded sensing noise, latency or
|
|
57
|
+
disturbances.
|
|
58
|
+
- **Autonomous and safety-critical systems**: margins reported with honest uncertainty, and
|
|
59
|
+
censored bounds instead of invented numbers when the data cannot support an estimate.
|
|
60
|
+
|
|
61
|
+
> **Status: latest release `0.3.0`.** Dose-response fits for pass/fail outcomes that fall or rise
|
|
62
|
+
> with severity and for real-valued scores (Gaussian least squares), thresholds with
|
|
63
|
+
> profile-likelihood or delta confidence intervals, censoring, ratios of two thresholds with
|
|
64
|
+
> Fieller or log-delta intervals, fit diagnostics, and the grid break-point rule. Clustered and
|
|
65
|
+
> paired inference, model-free fits and design helpers are planned for `0.4`. While the version
|
|
66
|
+
> is below 1.0, a minor release may change the public API.
|
|
67
|
+
|
|
68
|
+
## What it does
|
|
69
|
+
|
|
70
|
+
You test a system at increasing levels of some stress (the *severity*), such as input noise, a
|
|
71
|
+
perturbation, a disturbance or a distribution shift, and count successes at each level (or record
|
|
72
|
+
a score for each run). robustmargin fits a dose-response curve to those outcomes and reports the
|
|
73
|
+
severity at which performance crosses a level you choose, with a confidence interval. When the
|
|
74
|
+
data cannot support a number, it says so: a threshold beyond the tested range is censored and
|
|
75
|
+
reported as a bound, and a failed fit carries a status flag and no estimate.
|
|
76
|
+
|
|
77
|
+
The statistics are dose-response threshold estimation, the method toxicology and psychophysics
|
|
78
|
+
use to find the dose at which a response crosses a given level. That is why the second worked
|
|
79
|
+
example below uses a classic toxicology dataset: the code is the same whatever the system under
|
|
80
|
+
test.
|
|
81
|
+
|
|
82
|
+
For control engineers, the threshold is an empirical analogue of the probabilistic robustness
|
|
83
|
+
margin of randomized robust control (Barmish, Lagoa & Tempo, ACC 1997; Chen, Zhou & Aravena,
|
|
84
|
+
SIAM J. Control Optim. 2008). That margin is the largest radius at which a known model still
|
|
85
|
+
succeeds with probability 1 − ε, averaged over every perturbation up to that radius.
|
|
86
|
+
robustmargin needs no model: it estimates the corresponding point from success counts at held
|
|
87
|
+
severities, with a confidence interval.
|
|
88
|
+
|
|
89
|
+
## How it fits into an evaluation stack
|
|
90
|
+
|
|
91
|
+
robustmargin does not run models. Your policy or model evaluator does, and a thin adapter turns
|
|
92
|
+
its results into robustmargin's input:
|
|
93
|
+
|
|
94
|
+
1. The evaluator applies a graded degradation (input noise, a perturbation, a disturbance, a
|
|
95
|
+
distribution shift) at a set of severity levels.
|
|
96
|
+
2. It records the outcome of each run: pass/fail, or a real-valued score.
|
|
97
|
+
3. The adapter hands robustmargin the severities and the success counts or scores; robustmargin
|
|
98
|
+
returns the threshold, its confidence interval, its censoring status and fit diagnostics.
|
|
99
|
+
|
|
100
|
+
Because the interface is only severities and outcomes, the same adapter pattern works for any
|
|
101
|
+
evaluator. A typical use is comparing checkpoints, for example before and after RL
|
|
102
|
+
post-training: `ratio_interval` gives a confidence interval for the ratio of their margins, so
|
|
103
|
+
"the new checkpoint is more robust" becomes a claim with an interval attached.
|
|
104
|
+
|
|
105
|
+
## Install
|
|
106
|
+
|
|
107
|
+
Requires Python 3.11 or later.
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
pip install robustmargin
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Runtime dependencies are `numpy`, `scipy`, and `statsmodels`, with loose minimum versions.
|
|
114
|
+
|
|
115
|
+
## Worked example: robustness before and after RL post-training
|
|
116
|
+
|
|
117
|
+
Two checkpoints of the same policy, before and after RL post-training, are evaluated on a task at
|
|
118
|
+
increasing levels of Gaussian input noise: 200 runs per level, each scored pass or fail. The counts
|
|
119
|
+
are illustrative. The question a safety case asks is not "what is the success rate?" but "how much
|
|
120
|
+
noise can each checkpoint take before success falls below 90%, and did post-training move that
|
|
121
|
+
point?"
|
|
122
|
+
|
|
123
|
+
```python
|
|
124
|
+
from robustmargin import (
|
|
125
|
+
Axis,
|
|
126
|
+
Censoring,
|
|
127
|
+
Definition,
|
|
128
|
+
Observations,
|
|
129
|
+
Status,
|
|
130
|
+
fit_dose_response,
|
|
131
|
+
ratio_interval,
|
|
132
|
+
threshold,
|
|
133
|
+
)
|
|
134
|
+
|
|
135
|
+
axis = Axis(name="input_noise", unit="std", scale="log")
|
|
136
|
+
noise = [0.0, 0.05, 0.1, 0.2, 0.4, 0.8]
|
|
137
|
+
runs = [200] * len(noise)
|
|
138
|
+
passed = {
|
|
139
|
+
"before_rl": [196, 193, 181, 142, 61, 9],
|
|
140
|
+
"after_rl": [197, 196, 192, 177, 119, 38],
|
|
141
|
+
}
|
|
142
|
+
|
|
143
|
+
margins = {}
|
|
144
|
+
for name, successes in passed.items():
|
|
145
|
+
obs = Observations.from_counts(
|
|
146
|
+
axis,
|
|
147
|
+
severity=noise,
|
|
148
|
+
successes=successes,
|
|
149
|
+
trials=runs,
|
|
150
|
+
outcome="task_success",
|
|
151
|
+
direction="decreasing",
|
|
152
|
+
)
|
|
153
|
+
# Clean-input success is below 100%, so the upper asymptote is estimated, not fixed at 1.
|
|
154
|
+
fit = fit_dose_response(obs, model="binomial", link="probit", upper="estimate", lower=0.0)
|
|
155
|
+
assert fit.status is Status.OK
|
|
156
|
+
margins[name] = threshold(
|
|
157
|
+
fit, definition=Definition.absolute(0.9), interval_method="profile", level=0.95
|
|
158
|
+
)
|
|
159
|
+
|
|
160
|
+
# The noise level at which success falls to 90%, with a 95% profile-likelihood interval.
|
|
161
|
+
before, after = margins["before_rl"], margins["after_rl"]
|
|
162
|
+
assert before.censoring is Censoring.NONE and after.censoring is Censoring.NONE
|
|
163
|
+
assert 0.096 < before.lo < before.value < before.hi < 0.130 # about 0.11
|
|
164
|
+
assert 0.159 < after.lo < after.value < after.hi < 0.217 # about 0.19
|
|
165
|
+
|
|
166
|
+
# Did RL post-training move the margin? The ratio of the two margins, with its own interval.
|
|
167
|
+
ratio = ratio_interval(after, before, method="log_delta", dependence="independent")
|
|
168
|
+
assert 1.35 < ratio.lo < ratio.estimate < ratio.hi < 2.06 # about 1.67x, interval excludes 1
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
When the data cannot support a number, robustmargin does not invent one. A checkpoint that never
|
|
172
|
+
drops below the criterion within the tested range gets a censored bound, not an estimate:
|
|
173
|
+
|
|
174
|
+
```python
|
|
175
|
+
from robustmargin import (
|
|
176
|
+
Axis,
|
|
177
|
+
Censoring,
|
|
178
|
+
Definition,
|
|
179
|
+
Observations,
|
|
180
|
+
fit_dose_response,
|
|
181
|
+
threshold,
|
|
182
|
+
)
|
|
183
|
+
|
|
184
|
+
obs = Observations.from_counts(
|
|
185
|
+
Axis(name="input_noise", unit="std", scale="log"),
|
|
186
|
+
severity=[0.0, 0.05, 0.1, 0.2, 0.4, 0.8],
|
|
187
|
+
successes=[200, 200, 199, 200, 199, 198],
|
|
188
|
+
trials=[200] * 6,
|
|
189
|
+
outcome="task_success",
|
|
190
|
+
direction="decreasing",
|
|
191
|
+
)
|
|
192
|
+
fit = fit_dose_response(obs, model="binomial", link="probit", upper="estimate", lower=0.0)
|
|
193
|
+
t = threshold(fit, definition=Definition.absolute(0.9), interval_method="profile", level=0.95)
|
|
194
|
+
|
|
195
|
+
# "The margin exceeds 0.8" (exact one-sided bound), never "the margin is 0.8".
|
|
196
|
+
assert t.censoring is Censoring.RIGHT
|
|
197
|
+
assert t.value is None and t.lo == 0.8
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
## Worked example: a classic dose-response dataset
|
|
201
|
+
|
|
202
|
+
Finney's (1971) toxicity data: groups of insects exposed to increasing doses, with the number
|
|
203
|
+
affected in each group. The outcome here is survival, which decreases with dose. The dose-0 group
|
|
204
|
+
is an untreated control.
|
|
205
|
+
|
|
206
|
+
```python
|
|
207
|
+
from robustmargin import (
|
|
208
|
+
Axis,
|
|
209
|
+
Censoring,
|
|
210
|
+
Definition,
|
|
211
|
+
Observations,
|
|
212
|
+
Status,
|
|
213
|
+
fit_dose_response,
|
|
214
|
+
threshold,
|
|
215
|
+
)
|
|
216
|
+
|
|
217
|
+
dose = [0.0, 2.6, 3.8, 5.1, 7.7, 10.2]
|
|
218
|
+
affected = [0, 6, 16, 24, 42, 44]
|
|
219
|
+
total = [49, 50, 48, 46, 49, 50]
|
|
220
|
+
|
|
221
|
+
obs = Observations.from_counts(
|
|
222
|
+
Axis(name="dose", unit="mg", scale="log"),
|
|
223
|
+
severity=dose,
|
|
224
|
+
successes=[n - a for n, a in zip(total, affected, strict=True)],
|
|
225
|
+
trials=total,
|
|
226
|
+
outcome="survival",
|
|
227
|
+
direction="decreasing",
|
|
228
|
+
)
|
|
229
|
+
|
|
230
|
+
# The link and the asymptotes are chosen before looking at the data; nothing is auto-selected.
|
|
231
|
+
fit = fit_dose_response(obs, model="binomial", link="probit", upper=1.0, lower=0.0)
|
|
232
|
+
assert fit.status is Status.OK
|
|
233
|
+
|
|
234
|
+
# The dose at which survival falls to 50%, with a 95% profile-likelihood interval.
|
|
235
|
+
t50 = threshold(fit, definition=Definition.absolute(0.5), interval_method="profile", level=0.95)
|
|
236
|
+
assert t50.censoring is Censoring.NONE
|
|
237
|
+
assert abs(t50.value - 4.85) < 0.01
|
|
238
|
+
assert 4.3 < t50.lo < t50.value < t50.hi < 5.4
|
|
239
|
+
|
|
240
|
+
# A successful fit carries goodness-of-fit diagnostics, and every fit, failed or not, carries a
|
|
241
|
+
# monotonicity check on the raw counts. Both are reported and never acted on: nothing is
|
|
242
|
+
# rescaled, and the link you chose is never swapped for a better-scoring one.
|
|
243
|
+
diag = fit.diagnostics
|
|
244
|
+
assert diag.df == 3 # five dosed groups, two estimated parameters
|
|
245
|
+
assert diag.dispersion < 1.5 # no sign of overdispersion
|
|
246
|
+
assert set(diag.link_aic) == {"probit", "logit", "cloglog"}
|
|
247
|
+
assert not fit.monotonicity.any_flagged # survival never rises between adjacent doses
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
### Comparing two curves
|
|
251
|
+
|
|
252
|
+
Two curves from the `selenium` data: survival against concentration for two forms of a
|
|
253
|
+
compound. `baseline_fraction(0.5)` asks where survival falls to half of what the control group
|
|
254
|
+
achieves, and the upper asymptote is estimated rather than fixed at 1. The second form shows
|
|
255
|
+
what the diagnostics are for.
|
|
256
|
+
|
|
257
|
+
```python
|
|
258
|
+
from robustmargin import (
|
|
259
|
+
Axis,
|
|
260
|
+
Definition,
|
|
261
|
+
Observations,
|
|
262
|
+
fit_dose_response,
|
|
263
|
+
ratio_interval,
|
|
264
|
+
threshold,
|
|
265
|
+
)
|
|
266
|
+
|
|
267
|
+
conc = [0.0, 100.0, 200.0, 300.0, 400.0, 500.0]
|
|
268
|
+
curves = {
|
|
269
|
+
"form_a": ([151, 146, 116, 159, 150, 140], [3, 40, 31, 85, 102, 112]),
|
|
270
|
+
"form_b": ([141, 153, 142, 139, 154, 155], [2, 30, 59, 82, 62, 85]),
|
|
271
|
+
}
|
|
272
|
+
|
|
273
|
+
thresholds = {}
|
|
274
|
+
fits = {}
|
|
275
|
+
for name, (total, dead) in curves.items():
|
|
276
|
+
obs = Observations.from_counts(
|
|
277
|
+
Axis(name="concentration", unit="ug/L", scale="log"),
|
|
278
|
+
severity=conc,
|
|
279
|
+
successes=[n - d for n, d in zip(total, dead, strict=True)],
|
|
280
|
+
trials=total,
|
|
281
|
+
outcome="survival",
|
|
282
|
+
direction="decreasing",
|
|
283
|
+
)
|
|
284
|
+
fits[name] = fit_dose_response(
|
|
285
|
+
obs, model="binomial", link="probit", upper="estimate", lower=0.0
|
|
286
|
+
)
|
|
287
|
+
thresholds[name] = threshold(
|
|
288
|
+
fits[name],
|
|
289
|
+
definition=Definition.baseline_fraction(0.5),
|
|
290
|
+
interval_method="profile",
|
|
291
|
+
level=0.95,
|
|
292
|
+
)
|
|
293
|
+
|
|
294
|
+
ratio = ratio_interval(
|
|
295
|
+
thresholds["form_a"], thresholds["form_b"], method="fieller", dependence="independent"
|
|
296
|
+
)
|
|
297
|
+
assert 0.5 < ratio.lo < ratio.estimate < ratio.hi < 0.9 # form_a fails at a lower concentration
|
|
298
|
+
|
|
299
|
+
# form_b is where diagnostics earn their place: survival rises from 300 to 400, more than chance
|
|
300
|
+
# allows, and the groups vary far more than a binomial model expects. Both are reported on the
|
|
301
|
+
# fit. The reversal is also repeated as a warning on every threshold and ratio built from it.
|
|
302
|
+
# Neither changes a number.
|
|
303
|
+
form_b = fits["form_b"]
|
|
304
|
+
flagged = [(p.severity_low, p.severity_high) for p in form_b.monotonicity.pairs if p.flagged]
|
|
305
|
+
assert flagged == [(300.0, 400.0)]
|
|
306
|
+
assert form_b.diagnostics.dispersion > 1.5
|
|
307
|
+
assert any(w.startswith("OVERDISPERSED: ") for w in form_b.diagnostics.warnings)
|
|
308
|
+
assert any(w.startswith("NON_MONOTONE_DATA: ") for w in ratio.warnings)
|
|
309
|
+
```
|
|
310
|
+
|
|
311
|
+
Treat an overdispersed fit's intervals as too narrow. Clustered inference, which addresses this,
|
|
312
|
+
is planned for `0.4`.
|
|
313
|
+
|
|
314
|
+
### An outcome that rises with severity
|
|
315
|
+
|
|
316
|
+
Often what you count gets *worse* as severity grows: events per trial, not successes. Declare it
|
|
317
|
+
with `direction="increasing"`, and `successes` counts the outcome that rises. The curve runs from
|
|
318
|
+
`lower`, the rate at zero severity, up to `upper`. An increasing fit is the decreasing fit of the
|
|
319
|
+
complementary outcome (`trials - successes`) mapped back, so every status, interval and censoring
|
|
320
|
+
rule is the one the decreasing case uses, and every number and message is on the scale of the
|
|
321
|
+
outcome you counted.
|
|
322
|
+
|
|
323
|
+
`Definition.baseline_multiple(k)` asks where the rate reaches `k` times its value at zero
|
|
324
|
+
severity. It is defined on log axes only, and `threshold` raises `ValueError` on a linear axis. A
|
|
325
|
+
multiple of zero is not a threshold, so `lower` must be estimated (or fixed above zero): a fixed
|
|
326
|
+
`lower=0.0` over a control that has events is reported as `CONTROL_INCOMPATIBLE`, and
|
|
327
|
+
`baseline_multiple` raises for a fixed `lower` of zero. It also raises `ValueError` on any failed
|
|
328
|
+
fit (`SEPARATION` or `NOT_CONVERGED`), because a multiple of the control's observed rate would be
|
|
329
|
+
an uncovered target; `baseline_fraction(p)` raises on a failed fit for the same reason. Use
|
|
330
|
+
`Definition.absolute(a)` then, or a converged refit (for example with `lower` fixed above zero),
|
|
331
|
+
and catch the `ValueError` per job in a sweep.
|
|
332
|
+
|
|
333
|
+
```python
|
|
334
|
+
from robustmargin import (
|
|
335
|
+
Axis,
|
|
336
|
+
Definition,
|
|
337
|
+
Observations,
|
|
338
|
+
Status,
|
|
339
|
+
fit_dose_response,
|
|
340
|
+
threshold,
|
|
341
|
+
)
|
|
342
|
+
|
|
343
|
+
severity = [0.0, 0.5, 1.0, 2.0, 4.0, 8.0, 16.0]
|
|
344
|
+
events = [20, 22, 35, 90, 220, 340, 380]
|
|
345
|
+
|
|
346
|
+
obs = Observations.from_counts(
|
|
347
|
+
Axis(name="severity", unit="unit", scale="log"),
|
|
348
|
+
severity=severity,
|
|
349
|
+
successes=events,
|
|
350
|
+
trials=[400] * len(severity),
|
|
351
|
+
outcome="event",
|
|
352
|
+
direction="increasing",
|
|
353
|
+
)
|
|
354
|
+
fit = fit_dose_response(obs, model="binomial", link="probit", upper=1.0, lower="estimate")
|
|
355
|
+
assert fit.status is Status.OK
|
|
356
|
+
assert 0.03 < fit.params["lower"].value < 0.07 # about 5% of trials have an event at severity 0
|
|
357
|
+
|
|
358
|
+
# The severity at which the event rate doubles, with a 95% profile-likelihood interval.
|
|
359
|
+
doubling = threshold(
|
|
360
|
+
fit, definition=Definition.baseline_multiple(2.0), interval_method="profile", level=0.95
|
|
361
|
+
)
|
|
362
|
+
assert doubling.status is Status.OK
|
|
363
|
+
assert 0.7 < doubling.lo < doubling.value < doubling.hi < 1.5
|
|
364
|
+
assert abs(fit.predict(doubling.value).estimate - 2 * fit.params["lower"].value) < 1e-6
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
`Definition.absolute(a)` and `Definition.relative(p)` mean the same in both directions, with the
|
|
368
|
+
crossing read upward. `Definition.baseline_fraction(p)` belongs to a decreasing outcome, and
|
|
369
|
+
`threshold` raises if you pair it with an increasing fit (and `baseline_multiple` with a
|
|
370
|
+
decreasing one). For an increasing outcome, `RIGHT` reads "still passes at the highest level
|
|
371
|
+
tested": the rate is confidently below the target there.
|
|
372
|
+
|
|
373
|
+
### A continuous response
|
|
374
|
+
|
|
375
|
+
When each run gives a real-valued score instead of a pass or fail, build
|
|
376
|
+
`ContinuousObservations` and fit with `model="gaussian"`. The curve is the same, now with
|
|
377
|
+
independent normal errors fitted by least squares, and `link` still names the shape of the
|
|
378
|
+
transition (`"logit"`, `"probit"` and `"cloglog"` are R `drc`'s `LL.4`, `LN.4` and `W1.4`).
|
|
379
|
+
`ContinuousObservations.from_observations` takes one response per observation and `from_summaries` takes
|
|
380
|
+
a mean, a sample SD and a count per level; both describe the same data and give the same fit.
|
|
381
|
+
|
|
382
|
+
`upper` and `lower` are in the units of the response, so a threshold target such as
|
|
383
|
+
`Definition.absolute(0.7)` is too. Intervals are calibrated on the fit's residual degrees of
|
|
384
|
+
freedom: the delta interval uses a Student-t quantile and the profile uses the F distribution on
|
|
385
|
+
the change in the residual sum of squares. A continuous response has no exact bounds, so a
|
|
386
|
+
threshold is `OK`, `UNREACHABLE` (the target lies outside the fitted range of the response) or
|
|
387
|
+
`NOT_CONVERGED`, and a value beyond the tested range carries an `EXTRAPOLATED:` warning. A fit that
|
|
388
|
+
cannot be identified (a transition wider than the tested range, a step between two levels, no
|
|
389
|
+
residual degrees of freedom, a response that does not move the way `direction` says) is
|
|
390
|
+
`NOT_CONVERGED` with the reason in `warnings` and no number.
|
|
391
|
+
|
|
392
|
+
```python
|
|
393
|
+
from robustmargin import (
|
|
394
|
+
Axis,
|
|
395
|
+
ContinuousObservations,
|
|
396
|
+
Definition,
|
|
397
|
+
Status,
|
|
398
|
+
fit_dose_response,
|
|
399
|
+
threshold,
|
|
400
|
+
)
|
|
401
|
+
|
|
402
|
+
severity = [x for x in (0.0, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0) for _ in range(3)]
|
|
403
|
+
score = [
|
|
404
|
+
0.95, 0.96, 0.94, 0.87, 0.88, 0.87, 0.80, 0.84, 0.78, 0.59, 0.63, 0.62,
|
|
405
|
+
0.39, 0.36, 0.39, 0.27, 0.20, 0.23, 0.13, 0.14, 0.13,
|
|
406
|
+
] # fmt: skip
|
|
407
|
+
|
|
408
|
+
obs = ContinuousObservations.from_observations(
|
|
409
|
+
Axis(name="severity", unit="unit", scale="log"),
|
|
410
|
+
severity=severity,
|
|
411
|
+
value=score,
|
|
412
|
+
outcome="score",
|
|
413
|
+
direction="decreasing",
|
|
414
|
+
)
|
|
415
|
+
fit = fit_dose_response(obs, model="gaussian", link="logit", upper="estimate", lower="estimate")
|
|
416
|
+
assert fit.status is Status.OK
|
|
417
|
+
assert fit.residual.df == 17 # 21 observations, 4 estimated parameters
|
|
418
|
+
|
|
419
|
+
# The severity at which the score has fallen to 0.7, with a 95% profile interval.
|
|
420
|
+
margin = threshold(fit, definition=Definition.absolute(0.7), interval_method="profile", level=0.95)
|
|
421
|
+
assert margin.status is Status.OK
|
|
422
|
+
assert 0.6 < margin.lo < margin.value < margin.hi < 0.9
|
|
423
|
+
assert abs(fit.predict(margin.value).estimate - 0.7) < 1e-6
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
A ratio of two continuous thresholds uses a Student-t quantile on Welch-Satterthwaite degrees of
|
|
427
|
+
freedom, and `ratio_interval` raises `ValueError` for a count threshold against a continuous one.
|
|
428
|
+
|
|
429
|
+
## Grid break-point rule
|
|
430
|
+
|
|
431
|
+
`grid_break_point` finds the smallest tested severity at which observed performance falls
|
|
432
|
+
strictly below a criterion. It reports an **observed grid statistic, not an estimated
|
|
433
|
+
threshold**. The reported value is always one of the severities you tested. It is never
|
|
434
|
+
interpolated, and it has no confidence interval.
|
|
435
|
+
|
|
436
|
+
|
|
437
|
+
```python
|
|
438
|
+
from robustmargin import Censoring, grid_break_point
|
|
439
|
+
|
|
440
|
+
severity = [0.0, 0.5, 1.0, 2.0, 4.0]
|
|
441
|
+
performance = [1.0, 0.98, 0.96, 0.91, 0.40] # e.g. success rate at each level
|
|
442
|
+
|
|
443
|
+
result = grid_break_point(severity, performance, criterion=0.95)
|
|
444
|
+
assert result.value == 2.0 # first tested level strictly below 0.95
|
|
445
|
+
assert result.max_tested == 4.0
|
|
446
|
+
assert result.censoring is Censoring.NONE
|
|
447
|
+
```
|
|
448
|
+
|
|
449
|
+
When no tested level fails, there is no value. The result is right-censored and records how far
|
|
450
|
+
testing went, so it reads as "held up to 2.0", never as "cannot fail":
|
|
451
|
+
|
|
452
|
+
```python
|
|
453
|
+
from robustmargin import Censoring, grid_break_point
|
|
454
|
+
|
|
455
|
+
held = grid_break_point([0.0, 1.0, 2.0], [1.0, 0.99, 0.97], criterion=0.95)
|
|
456
|
+
assert held.value is None
|
|
457
|
+
assert held.max_tested == 2.0
|
|
458
|
+
assert held.censoring is Censoring.RIGHT
|
|
459
|
+
```
|
|
460
|
+
|
|
461
|
+
On a signed axis, pass `signed=True`. The rule then breaks on magnitude, on whichever side fails
|
|
462
|
+
first:
|
|
463
|
+
|
|
464
|
+
```python
|
|
465
|
+
from robustmargin import grid_break_point
|
|
466
|
+
|
|
467
|
+
signed = grid_break_point(
|
|
468
|
+
[-0.2, -0.1, 0.0, 0.1, 0.2],
|
|
469
|
+
[0.05, 0.77, 0.99, 0.33, 0.0],
|
|
470
|
+
criterion=0.95,
|
|
471
|
+
signed=True,
|
|
472
|
+
)
|
|
473
|
+
assert signed.value == 0.1
|
|
474
|
+
assert signed.max_tested == 0.2
|
|
475
|
+
```
|
|
476
|
+
|
|
477
|
+
Some points about the rule:
|
|
478
|
+
|
|
479
|
+
- A level sitting exactly on the criterion passes. Only `performance < criterion` fails.
|
|
480
|
+
- Monotonicity is neither assumed nor checked, and nothing is pooled or fitted.
|
|
481
|
+
- Invalid input raises `ValueError` rather than returning a placeholder. That covers empty or
|
|
482
|
+
mismatched arrays, non-finite values, and negative severities without `signed=True`.
|
|
483
|
+
|
|
484
|
+
## Results and JSON
|
|
485
|
+
|
|
486
|
+
Every result is a frozen dataclass carrying a `schema_version` and a `provenance` mapping.
|
|
487
|
+
`provenance` records where the inputs came from; robustmargin stores it and never interprets it.
|
|
488
|
+
|
|
489
|
+
- `GridBreakPoint` is what `grid_break_point` returns.
|
|
490
|
+
- `Fit`, `Threshold` and `Ratio` are what `fit_dose_response`, `threshold` and `ratio_interval`
|
|
491
|
+
return. They check their own consistency when constructed. For example, a failed fit cannot
|
|
492
|
+
carry parameters, and a censored threshold cannot carry a point estimate.
|
|
493
|
+
|
|
494
|
+
Results are collected in a `Scorecard` and written to JSON with `robustmargin.report`:
|
|
495
|
+
|
|
496
|
+
```python
|
|
497
|
+
import json
|
|
498
|
+
|
|
499
|
+
from robustmargin import Scorecard, grid_break_point
|
|
500
|
+
from robustmargin.report import from_dict, load_schema, to_dict
|
|
501
|
+
|
|
502
|
+
held = grid_break_point([0.0, 1.0, 2.0], [1.0, 0.99, 0.97], criterion=0.95)
|
|
503
|
+
card = Scorecard(results=(held,), provenance={"run": "example"})
|
|
504
|
+
|
|
505
|
+
text = json.dumps(to_dict(card), allow_nan=False)
|
|
506
|
+
assert from_dict(json.loads(text)) == card
|
|
507
|
+
assert load_schema()["$schema"] == "https://json-schema.org/draft/2020-12/schema"
|
|
508
|
+
```
|
|
509
|
+
|
|
510
|
+
The JSON follows `schema/scorecard-v2.json`, a JSON Schema shipped inside the package, and
|
|
511
|
+
`schema/scorecard-v1.json` is kept so that cards written by `0.1.1` and `0.2.0` still read.
|
|
512
|
+
Schema `"2"` adds increasing outcomes and `baseline_multiple`. It covers binary outcomes only,
|
|
513
|
+
and reading is strict: a card with fields this version does not know is rejected, not silently
|
|
514
|
+
trimmed. To read a stored card, use a robustmargin at least as new as the
|
|
515
|
+
one that wrote it. The card's `library_version` field records which version that was.
|
|
516
|
+
|
|
517
|
+
For your own tests, `robustmargin.testing` provides `fake_fit`, `fake_threshold` and `fake_ratio`.
|
|
518
|
+
They return schema-valid results in every status and censoring state, and each is labelled as a
|
|
519
|
+
fake rather than an estimate.
|
|
520
|
+
|
|
521
|
+
## Development
|
|
522
|
+
|
|
523
|
+
```bash
|
|
524
|
+
pip install -e ".[dev]"
|
|
525
|
+
ruff check . && ruff format --check .
|
|
526
|
+
mypy
|
|
527
|
+
pytest # enforces 90% branch coverage
|
|
528
|
+
```
|
|
529
|
+
|
|
530
|
+
CI runs the tests on Python 3.11, 3.12, and 3.13 against current dependencies. It also runs them
|
|
531
|
+
on Python 3.11 against the oldest supported versions, pinned in
|
|
532
|
+
[`ci/constraints-min.txt`](https://github.com/robustmargin/robustmargin/blob/main/ci/constraints-min.txt).
|
|
533
|
+
Changes are recorded in [`CHANGELOG.md`](https://github.com/robustmargin/robustmargin/blob/main/CHANGELOG.md).
|
|
534
|
+
|
|
535
|
+
## License
|
|
536
|
+
|
|
537
|
+
Apache-2.0. See [`LICENSE`](https://github.com/robustmargin/robustmargin/blob/main/LICENSE).
|