robustmargin 0.2.1__tar.gz → 0.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (29) hide show
  1. robustmargin-0.3.0/PKG-INFO +537 -0
  2. robustmargin-0.3.0/README.md +500 -0
  3. {robustmargin-0.2.1 → robustmargin-0.3.0}/pyproject.toml +12 -1
  4. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/__init__.py +18 -6
  5. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/censoring.py +188 -36
  6. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/empirical.py +2 -2
  7. robustmargin-0.3.0/src/robustmargin/intervals.py +2338 -0
  8. robustmargin-0.3.0/src/robustmargin/models.py +4498 -0
  9. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/ratio.py +75 -13
  10. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/report.py +140 -30
  11. robustmargin-0.3.0/src/robustmargin/schema/scorecard-v2.json +1100 -0
  12. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/testing.py +291 -21
  13. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/threshold.py +329 -95
  14. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/types.py +460 -21
  15. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/validation.py +487 -54
  16. robustmargin-0.3.0/src/robustmargin.egg-info/PKG-INFO +537 -0
  17. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin.egg-info/SOURCES.txt +2 -1
  18. robustmargin-0.2.1/PKG-INFO +0 -285
  19. robustmargin-0.2.1/README.md +0 -250
  20. robustmargin-0.2.1/src/robustmargin/intervals.py +0 -861
  21. robustmargin-0.2.1/src/robustmargin/models.py +0 -2585
  22. robustmargin-0.2.1/src/robustmargin.egg-info/PKG-INFO +0 -285
  23. {robustmargin-0.2.1 → robustmargin-0.3.0}/LICENSE +0 -0
  24. {robustmargin-0.2.1 → robustmargin-0.3.0}/setup.cfg +0 -0
  25. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/py.typed +0 -0
  26. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin/schema/scorecard-v1.json +0 -0
  27. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin.egg-info/dependency_links.txt +0 -0
  28. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin.egg-info/requires.txt +0 -0
  29. {robustmargin-0.2.1 → robustmargin-0.3.0}/src/robustmargin.egg-info/top_level.txt +0 -0
@@ -0,0 +1,537 @@
1
+ Metadata-Version: 2.4
2
+ Name: robustmargin
3
+ Version: 0.3.0
4
+ Summary: Robustness margins for AI safety evaluation: how much stress an AI system takes before it fails, with a confidence interval.
5
+ Author: robustmargin contributors
6
+ License-Expression: Apache-2.0
7
+ Project-URL: Homepage, https://github.com/robustmargin/robustmargin
8
+ Project-URL: Source, https://github.com/robustmargin/robustmargin
9
+ Project-URL: Changelog, https://github.com/robustmargin/robustmargin/blob/main/CHANGELOG.md
10
+ Project-URL: Issues, https://github.com/robustmargin/robustmargin/issues
11
+ Keywords: ai-safety,ai-evaluation,robustness,stress-testing,autonomous-systems,dose-response,confidence-intervals,statistics
12
+ Classifier: Development Status :: 3 - Alpha
13
+ Classifier: Intended Audience :: Science/Research
14
+ Classifier: Operating System :: OS Independent
15
+ Classifier: Programming Language :: Python :: 3.11
16
+ Classifier: Programming Language :: Python :: 3.12
17
+ Classifier: Programming Language :: Python :: 3.13
18
+ Classifier: Topic :: Scientific/Engineering
19
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
20
+ Classifier: Topic :: Scientific/Engineering :: Mathematics
21
+ Classifier: Typing :: Typed
22
+ Requires-Python: >=3.11
23
+ Description-Content-Type: text/markdown
24
+ License-File: LICENSE
25
+ Requires-Dist: numpy>=1.26
26
+ Requires-Dist: scipy>=1.11.1
27
+ Requires-Dist: statsmodels>=0.14
28
+ Provides-Extra: dev
29
+ Requires-Dist: pytest>=8.0; extra == "dev"
30
+ Requires-Dist: pytest-cov>=5.0; extra == "dev"
31
+ Requires-Dist: hypothesis>=6.100; extra == "dev"
32
+ Requires-Dist: ruff>=0.5; extra == "dev"
33
+ Requires-Dist: mypy>=1.10; extra == "dev"
34
+ Requires-Dist: pre-commit>=3.5; extra == "dev"
35
+ Requires-Dist: jsonschema>=4.20; extra == "dev"
36
+ Dynamic: license-file
37
+
38
+ # robustmargin
39
+ [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.23585829.svg)](https://doi.org/10.5281/zenodo.23585829)
40
+
41
+ **Robustness margins for AI safety evaluation: how much stress an AI system takes before it
42
+ fails, with a confidence interval.**
43
+
44
+ robustmargin is a robustness calculation engine. AI systems are usually evaluated by a success
45
+ rate under nominal conditions; a safety case needs a different number: how far conditions can
46
+ degrade before the system starts to fail, and how sure we are of that point. robustmargin
47
+ estimates that robustness margin, with a confidence interval, from stress tests at increasing
48
+ severity, scored pass/fail or on a continuous scale.
49
+
50
+ It is model-agnostic and framework-free: it sees only severities and outcomes (success counts or
51
+ real-valued scores), and needs no GPU and no ML framework. That makes it usable wherever
52
+ robustness and safety carry weight:
53
+
54
+ - **AI safety research**: measure how much perturbation a model or policy tolerates, and whether
55
+ a training intervention moved that margin.
56
+ - **Physical AI**: learned policies evaluated under graded sensing noise, latency or
57
+ disturbances.
58
+ - **Autonomous and safety-critical systems**: margins reported with honest uncertainty, and
59
+ censored bounds instead of invented numbers when the data cannot support an estimate.
60
+
61
+ > **Status: latest release `0.3.0`.** Dose-response fits for pass/fail outcomes that fall or rise
62
+ > with severity and for real-valued scores (Gaussian least squares), thresholds with
63
+ > profile-likelihood or delta confidence intervals, censoring, ratios of two thresholds with
64
+ > Fieller or log-delta intervals, fit diagnostics, and the grid break-point rule. Clustered and
65
+ > paired inference, model-free fits and design helpers are planned for `0.4`. While the version
66
+ > is below 1.0, a minor release may change the public API.
67
+
68
+ ## What it does
69
+
70
+ You test a system at increasing levels of some stress (the *severity*), such as input noise, a
71
+ perturbation, a disturbance or a distribution shift, and count successes at each level (or record
72
+ a score for each run). robustmargin fits a dose-response curve to those outcomes and reports the
73
+ severity at which performance crosses a level you choose, with a confidence interval. When the
74
+ data cannot support a number, it says so: a threshold beyond the tested range is censored and
75
+ reported as a bound, and a failed fit carries a status flag and no estimate.
76
+
77
+ The statistics are dose-response threshold estimation, the method toxicology and psychophysics
78
+ use to find the dose at which a response crosses a given level. That is why the second worked
79
+ example below uses a classic toxicology dataset: the code is the same whatever the system under
80
+ test.
81
+
82
+ For control engineers, the threshold is an empirical analogue of the probabilistic robustness
83
+ margin of randomized robust control (Barmish, Lagoa & Tempo, ACC 1997; Chen, Zhou & Aravena,
84
+ SIAM J. Control Optim. 2008). That margin is the largest radius at which a known model still
85
+ succeeds with probability 1 − ε, averaged over every perturbation up to that radius.
86
+ robustmargin needs no model: it estimates the corresponding point from success counts at held
87
+ severities, with a confidence interval.
88
+
89
+ ## How it fits into an evaluation stack
90
+
91
+ robustmargin does not run models. Your policy or model evaluator does, and a thin adapter turns
92
+ its results into robustmargin's input:
93
+
94
+ 1. The evaluator applies a graded degradation (input noise, a perturbation, a disturbance, a
95
+ distribution shift) at a set of severity levels.
96
+ 2. It records the outcome of each run: pass/fail, or a real-valued score.
97
+ 3. The adapter hands robustmargin the severities and the success counts or scores; robustmargin
98
+ returns the threshold, its confidence interval, its censoring status and fit diagnostics.
99
+
100
+ Because the interface is only severities and outcomes, the same adapter pattern works for any
101
+ evaluator. A typical use is comparing checkpoints, for example before and after RL
102
+ post-training: `ratio_interval` gives a confidence interval for the ratio of their margins, so
103
+ "the new checkpoint is more robust" becomes a claim with an interval attached.
104
+
105
+ ## Install
106
+
107
+ Requires Python 3.11 or later.
108
+
109
+ ```bash
110
+ pip install robustmargin
111
+ ```
112
+
113
+ Runtime dependencies are `numpy`, `scipy`, and `statsmodels`, with loose minimum versions.
114
+
115
+ ## Worked example: robustness before and after RL post-training
116
+
117
+ Two checkpoints of the same policy, before and after RL post-training, are evaluated on a task at
118
+ increasing levels of Gaussian input noise: 200 runs per level, each scored pass or fail. The counts
119
+ are illustrative. The question a safety case asks is not "what is the success rate?" but "how much
120
+ noise can each checkpoint take before success falls below 90%, and did post-training move that
121
+ point?"
122
+
123
+ ```python
124
+ from robustmargin import (
125
+ Axis,
126
+ Censoring,
127
+ Definition,
128
+ Observations,
129
+ Status,
130
+ fit_dose_response,
131
+ ratio_interval,
132
+ threshold,
133
+ )
134
+
135
+ axis = Axis(name="input_noise", unit="std", scale="log")
136
+ noise = [0.0, 0.05, 0.1, 0.2, 0.4, 0.8]
137
+ runs = [200] * len(noise)
138
+ passed = {
139
+ "before_rl": [196, 193, 181, 142, 61, 9],
140
+ "after_rl": [197, 196, 192, 177, 119, 38],
141
+ }
142
+
143
+ margins = {}
144
+ for name, successes in passed.items():
145
+ obs = Observations.from_counts(
146
+ axis,
147
+ severity=noise,
148
+ successes=successes,
149
+ trials=runs,
150
+ outcome="task_success",
151
+ direction="decreasing",
152
+ )
153
+ # Clean-input success is below 100%, so the upper asymptote is estimated, not fixed at 1.
154
+ fit = fit_dose_response(obs, model="binomial", link="probit", upper="estimate", lower=0.0)
155
+ assert fit.status is Status.OK
156
+ margins[name] = threshold(
157
+ fit, definition=Definition.absolute(0.9), interval_method="profile", level=0.95
158
+ )
159
+
160
+ # The noise level at which success falls to 90%, with a 95% profile-likelihood interval.
161
+ before, after = margins["before_rl"], margins["after_rl"]
162
+ assert before.censoring is Censoring.NONE and after.censoring is Censoring.NONE
163
+ assert 0.096 < before.lo < before.value < before.hi < 0.130 # about 0.11
164
+ assert 0.159 < after.lo < after.value < after.hi < 0.217 # about 0.19
165
+
166
+ # Did RL post-training move the margin? The ratio of the two margins, with its own interval.
167
+ ratio = ratio_interval(after, before, method="log_delta", dependence="independent")
168
+ assert 1.35 < ratio.lo < ratio.estimate < ratio.hi < 2.06 # about 1.67x, interval excludes 1
169
+ ```
170
+
171
+ When the data cannot support a number, robustmargin does not invent one. A checkpoint that never
172
+ drops below the criterion within the tested range gets a censored bound, not an estimate:
173
+
174
+ ```python
175
+ from robustmargin import (
176
+ Axis,
177
+ Censoring,
178
+ Definition,
179
+ Observations,
180
+ fit_dose_response,
181
+ threshold,
182
+ )
183
+
184
+ obs = Observations.from_counts(
185
+ Axis(name="input_noise", unit="std", scale="log"),
186
+ severity=[0.0, 0.05, 0.1, 0.2, 0.4, 0.8],
187
+ successes=[200, 200, 199, 200, 199, 198],
188
+ trials=[200] * 6,
189
+ outcome="task_success",
190
+ direction="decreasing",
191
+ )
192
+ fit = fit_dose_response(obs, model="binomial", link="probit", upper="estimate", lower=0.0)
193
+ t = threshold(fit, definition=Definition.absolute(0.9), interval_method="profile", level=0.95)
194
+
195
+ # "The margin exceeds 0.8" (exact one-sided bound), never "the margin is 0.8".
196
+ assert t.censoring is Censoring.RIGHT
197
+ assert t.value is None and t.lo == 0.8
198
+ ```
199
+
200
+ ## Worked example: a classic dose-response dataset
201
+
202
+ Finney's (1971) toxicity data: groups of insects exposed to increasing doses, with the number
203
+ affected in each group. The outcome here is survival, which decreases with dose. The dose-0 group
204
+ is an untreated control.
205
+
206
+ ```python
207
+ from robustmargin import (
208
+ Axis,
209
+ Censoring,
210
+ Definition,
211
+ Observations,
212
+ Status,
213
+ fit_dose_response,
214
+ threshold,
215
+ )
216
+
217
+ dose = [0.0, 2.6, 3.8, 5.1, 7.7, 10.2]
218
+ affected = [0, 6, 16, 24, 42, 44]
219
+ total = [49, 50, 48, 46, 49, 50]
220
+
221
+ obs = Observations.from_counts(
222
+ Axis(name="dose", unit="mg", scale="log"),
223
+ severity=dose,
224
+ successes=[n - a for n, a in zip(total, affected, strict=True)],
225
+ trials=total,
226
+ outcome="survival",
227
+ direction="decreasing",
228
+ )
229
+
230
+ # The link and the asymptotes are chosen before looking at the data; nothing is auto-selected.
231
+ fit = fit_dose_response(obs, model="binomial", link="probit", upper=1.0, lower=0.0)
232
+ assert fit.status is Status.OK
233
+
234
+ # The dose at which survival falls to 50%, with a 95% profile-likelihood interval.
235
+ t50 = threshold(fit, definition=Definition.absolute(0.5), interval_method="profile", level=0.95)
236
+ assert t50.censoring is Censoring.NONE
237
+ assert abs(t50.value - 4.85) < 0.01
238
+ assert 4.3 < t50.lo < t50.value < t50.hi < 5.4
239
+
240
+ # A successful fit carries goodness-of-fit diagnostics, and every fit, failed or not, carries a
241
+ # monotonicity check on the raw counts. Both are reported and never acted on: nothing is
242
+ # rescaled, and the link you chose is never swapped for a better-scoring one.
243
+ diag = fit.diagnostics
244
+ assert diag.df == 3 # five dosed groups, two estimated parameters
245
+ assert diag.dispersion < 1.5 # no sign of overdispersion
246
+ assert set(diag.link_aic) == {"probit", "logit", "cloglog"}
247
+ assert not fit.monotonicity.any_flagged # survival never rises between adjacent doses
248
+ ```
249
+
250
+ ### Comparing two curves
251
+
252
+ Two curves from the `selenium` data: survival against concentration for two forms of a
253
+ compound. `baseline_fraction(0.5)` asks where survival falls to half of what the control group
254
+ achieves, and the upper asymptote is estimated rather than fixed at 1. The second form shows
255
+ what the diagnostics are for.
256
+
257
+ ```python
258
+ from robustmargin import (
259
+ Axis,
260
+ Definition,
261
+ Observations,
262
+ fit_dose_response,
263
+ ratio_interval,
264
+ threshold,
265
+ )
266
+
267
+ conc = [0.0, 100.0, 200.0, 300.0, 400.0, 500.0]
268
+ curves = {
269
+ "form_a": ([151, 146, 116, 159, 150, 140], [3, 40, 31, 85, 102, 112]),
270
+ "form_b": ([141, 153, 142, 139, 154, 155], [2, 30, 59, 82, 62, 85]),
271
+ }
272
+
273
+ thresholds = {}
274
+ fits = {}
275
+ for name, (total, dead) in curves.items():
276
+ obs = Observations.from_counts(
277
+ Axis(name="concentration", unit="ug/L", scale="log"),
278
+ severity=conc,
279
+ successes=[n - d for n, d in zip(total, dead, strict=True)],
280
+ trials=total,
281
+ outcome="survival",
282
+ direction="decreasing",
283
+ )
284
+ fits[name] = fit_dose_response(
285
+ obs, model="binomial", link="probit", upper="estimate", lower=0.0
286
+ )
287
+ thresholds[name] = threshold(
288
+ fits[name],
289
+ definition=Definition.baseline_fraction(0.5),
290
+ interval_method="profile",
291
+ level=0.95,
292
+ )
293
+
294
+ ratio = ratio_interval(
295
+ thresholds["form_a"], thresholds["form_b"], method="fieller", dependence="independent"
296
+ )
297
+ assert 0.5 < ratio.lo < ratio.estimate < ratio.hi < 0.9 # form_a fails at a lower concentration
298
+
299
+ # form_b is where diagnostics earn their place: survival rises from 300 to 400, more than chance
300
+ # allows, and the groups vary far more than a binomial model expects. Both are reported on the
301
+ # fit. The reversal is also repeated as a warning on every threshold and ratio built from it.
302
+ # Neither changes a number.
303
+ form_b = fits["form_b"]
304
+ flagged = [(p.severity_low, p.severity_high) for p in form_b.monotonicity.pairs if p.flagged]
305
+ assert flagged == [(300.0, 400.0)]
306
+ assert form_b.diagnostics.dispersion > 1.5
307
+ assert any(w.startswith("OVERDISPERSED: ") for w in form_b.diagnostics.warnings)
308
+ assert any(w.startswith("NON_MONOTONE_DATA: ") for w in ratio.warnings)
309
+ ```
310
+
311
+ Treat an overdispersed fit's intervals as too narrow. Clustered inference, which addresses this,
312
+ is planned for `0.4`.
313
+
314
+ ### An outcome that rises with severity
315
+
316
+ Often what you count gets *worse* as severity grows: events per trial, not successes. Declare it
317
+ with `direction="increasing"`, and `successes` counts the outcome that rises. The curve runs from
318
+ `lower`, the rate at zero severity, up to `upper`. An increasing fit is the decreasing fit of the
319
+ complementary outcome (`trials - successes`) mapped back, so every status, interval and censoring
320
+ rule is the one the decreasing case uses, and every number and message is on the scale of the
321
+ outcome you counted.
322
+
323
+ `Definition.baseline_multiple(k)` asks where the rate reaches `k` times its value at zero
324
+ severity. It is defined on log axes only, and `threshold` raises `ValueError` on a linear axis. A
325
+ multiple of zero is not a threshold, so `lower` must be estimated (or fixed above zero): a fixed
326
+ `lower=0.0` over a control that has events is reported as `CONTROL_INCOMPATIBLE`, and
327
+ `baseline_multiple` raises for a fixed `lower` of zero. It also raises `ValueError` on any failed
328
+ fit (`SEPARATION` or `NOT_CONVERGED`), because a multiple of the control's observed rate would be
329
+ an uncovered target; `baseline_fraction(p)` raises on a failed fit for the same reason. Use
330
+ `Definition.absolute(a)` then, or a converged refit (for example with `lower` fixed above zero),
331
+ and catch the `ValueError` per job in a sweep.
332
+
333
+ ```python
334
+ from robustmargin import (
335
+ Axis,
336
+ Definition,
337
+ Observations,
338
+ Status,
339
+ fit_dose_response,
340
+ threshold,
341
+ )
342
+
343
+ severity = [0.0, 0.5, 1.0, 2.0, 4.0, 8.0, 16.0]
344
+ events = [20, 22, 35, 90, 220, 340, 380]
345
+
346
+ obs = Observations.from_counts(
347
+ Axis(name="severity", unit="unit", scale="log"),
348
+ severity=severity,
349
+ successes=events,
350
+ trials=[400] * len(severity),
351
+ outcome="event",
352
+ direction="increasing",
353
+ )
354
+ fit = fit_dose_response(obs, model="binomial", link="probit", upper=1.0, lower="estimate")
355
+ assert fit.status is Status.OK
356
+ assert 0.03 < fit.params["lower"].value < 0.07 # about 5% of trials have an event at severity 0
357
+
358
+ # The severity at which the event rate doubles, with a 95% profile-likelihood interval.
359
+ doubling = threshold(
360
+ fit, definition=Definition.baseline_multiple(2.0), interval_method="profile", level=0.95
361
+ )
362
+ assert doubling.status is Status.OK
363
+ assert 0.7 < doubling.lo < doubling.value < doubling.hi < 1.5
364
+ assert abs(fit.predict(doubling.value).estimate - 2 * fit.params["lower"].value) < 1e-6
365
+ ```
366
+
367
+ `Definition.absolute(a)` and `Definition.relative(p)` mean the same in both directions, with the
368
+ crossing read upward. `Definition.baseline_fraction(p)` belongs to a decreasing outcome, and
369
+ `threshold` raises if you pair it with an increasing fit (and `baseline_multiple` with a
370
+ decreasing one). For an increasing outcome, `RIGHT` reads "still passes at the highest level
371
+ tested": the rate is confidently below the target there.
372
+
373
+ ### A continuous response
374
+
375
+ When each run gives a real-valued score instead of a pass or fail, build
376
+ `ContinuousObservations` and fit with `model="gaussian"`. The curve is the same, now with
377
+ independent normal errors fitted by least squares, and `link` still names the shape of the
378
+ transition (`"logit"`, `"probit"` and `"cloglog"` are R `drc`'s `LL.4`, `LN.4` and `W1.4`).
379
+ `ContinuousObservations.from_observations` takes one response per observation and `from_summaries` takes
380
+ a mean, a sample SD and a count per level; both describe the same data and give the same fit.
381
+
382
+ `upper` and `lower` are in the units of the response, so a threshold target such as
383
+ `Definition.absolute(0.7)` is too. Intervals are calibrated on the fit's residual degrees of
384
+ freedom: the delta interval uses a Student-t quantile and the profile uses the F distribution on
385
+ the change in the residual sum of squares. A continuous response has no exact bounds, so a
386
+ threshold is `OK`, `UNREACHABLE` (the target lies outside the fitted range of the response) or
387
+ `NOT_CONVERGED`, and a value beyond the tested range carries an `EXTRAPOLATED:` warning. A fit that
388
+ cannot be identified (a transition wider than the tested range, a step between two levels, no
389
+ residual degrees of freedom, a response that does not move the way `direction` says) is
390
+ `NOT_CONVERGED` with the reason in `warnings` and no number.
391
+
392
+ ```python
393
+ from robustmargin import (
394
+ Axis,
395
+ ContinuousObservations,
396
+ Definition,
397
+ Status,
398
+ fit_dose_response,
399
+ threshold,
400
+ )
401
+
402
+ severity = [x for x in (0.0, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0) for _ in range(3)]
403
+ score = [
404
+ 0.95, 0.96, 0.94, 0.87, 0.88, 0.87, 0.80, 0.84, 0.78, 0.59, 0.63, 0.62,
405
+ 0.39, 0.36, 0.39, 0.27, 0.20, 0.23, 0.13, 0.14, 0.13,
406
+ ] # fmt: skip
407
+
408
+ obs = ContinuousObservations.from_observations(
409
+ Axis(name="severity", unit="unit", scale="log"),
410
+ severity=severity,
411
+ value=score,
412
+ outcome="score",
413
+ direction="decreasing",
414
+ )
415
+ fit = fit_dose_response(obs, model="gaussian", link="logit", upper="estimate", lower="estimate")
416
+ assert fit.status is Status.OK
417
+ assert fit.residual.df == 17 # 21 observations, 4 estimated parameters
418
+
419
+ # The severity at which the score has fallen to 0.7, with a 95% profile interval.
420
+ margin = threshold(fit, definition=Definition.absolute(0.7), interval_method="profile", level=0.95)
421
+ assert margin.status is Status.OK
422
+ assert 0.6 < margin.lo < margin.value < margin.hi < 0.9
423
+ assert abs(fit.predict(margin.value).estimate - 0.7) < 1e-6
424
+ ```
425
+
426
+ A ratio of two continuous thresholds uses a Student-t quantile on Welch-Satterthwaite degrees of
427
+ freedom, and `ratio_interval` raises `ValueError` for a count threshold against a continuous one.
428
+
429
+ ## Grid break-point rule
430
+
431
+ `grid_break_point` finds the smallest tested severity at which observed performance falls
432
+ strictly below a criterion. It reports an **observed grid statistic, not an estimated
433
+ threshold**. The reported value is always one of the severities you tested. It is never
434
+ interpolated, and it has no confidence interval.
435
+
436
+
437
+ ```python
438
+ from robustmargin import Censoring, grid_break_point
439
+
440
+ severity = [0.0, 0.5, 1.0, 2.0, 4.0]
441
+ performance = [1.0, 0.98, 0.96, 0.91, 0.40] # e.g. success rate at each level
442
+
443
+ result = grid_break_point(severity, performance, criterion=0.95)
444
+ assert result.value == 2.0 # first tested level strictly below 0.95
445
+ assert result.max_tested == 4.0
446
+ assert result.censoring is Censoring.NONE
447
+ ```
448
+
449
+ When no tested level fails, there is no value. The result is right-censored and records how far
450
+ testing went, so it reads as "held up to 2.0", never as "cannot fail":
451
+
452
+ ```python
453
+ from robustmargin import Censoring, grid_break_point
454
+
455
+ held = grid_break_point([0.0, 1.0, 2.0], [1.0, 0.99, 0.97], criterion=0.95)
456
+ assert held.value is None
457
+ assert held.max_tested == 2.0
458
+ assert held.censoring is Censoring.RIGHT
459
+ ```
460
+
461
+ On a signed axis, pass `signed=True`. The rule then breaks on magnitude, on whichever side fails
462
+ first:
463
+
464
+ ```python
465
+ from robustmargin import grid_break_point
466
+
467
+ signed = grid_break_point(
468
+ [-0.2, -0.1, 0.0, 0.1, 0.2],
469
+ [0.05, 0.77, 0.99, 0.33, 0.0],
470
+ criterion=0.95,
471
+ signed=True,
472
+ )
473
+ assert signed.value == 0.1
474
+ assert signed.max_tested == 0.2
475
+ ```
476
+
477
+ Some points about the rule:
478
+
479
+ - A level sitting exactly on the criterion passes. Only `performance < criterion` fails.
480
+ - Monotonicity is neither assumed nor checked, and nothing is pooled or fitted.
481
+ - Invalid input raises `ValueError` rather than returning a placeholder. That covers empty or
482
+ mismatched arrays, non-finite values, and negative severities without `signed=True`.
483
+
484
+ ## Results and JSON
485
+
486
+ Every result is a frozen dataclass carrying a `schema_version` and a `provenance` mapping.
487
+ `provenance` records where the inputs came from; robustmargin stores it and never interprets it.
488
+
489
+ - `GridBreakPoint` is what `grid_break_point` returns.
490
+ - `Fit`, `Threshold` and `Ratio` are what `fit_dose_response`, `threshold` and `ratio_interval`
491
+ return. They check their own consistency when constructed. For example, a failed fit cannot
492
+ carry parameters, and a censored threshold cannot carry a point estimate.
493
+
494
+ Results are collected in a `Scorecard` and written to JSON with `robustmargin.report`:
495
+
496
+ ```python
497
+ import json
498
+
499
+ from robustmargin import Scorecard, grid_break_point
500
+ from robustmargin.report import from_dict, load_schema, to_dict
501
+
502
+ held = grid_break_point([0.0, 1.0, 2.0], [1.0, 0.99, 0.97], criterion=0.95)
503
+ card = Scorecard(results=(held,), provenance={"run": "example"})
504
+
505
+ text = json.dumps(to_dict(card), allow_nan=False)
506
+ assert from_dict(json.loads(text)) == card
507
+ assert load_schema()["$schema"] == "https://json-schema.org/draft/2020-12/schema"
508
+ ```
509
+
510
+ The JSON follows `schema/scorecard-v2.json`, a JSON Schema shipped inside the package, and
511
+ `schema/scorecard-v1.json` is kept so that cards written by `0.1.1` and `0.2.0` still read.
512
+ Schema `"2"` adds increasing outcomes and `baseline_multiple`. It covers binary outcomes only,
513
+ and reading is strict: a card with fields this version does not know is rejected, not silently
514
+ trimmed. To read a stored card, use a robustmargin at least as new as the
515
+ one that wrote it. The card's `library_version` field records which version that was.
516
+
517
+ For your own tests, `robustmargin.testing` provides `fake_fit`, `fake_threshold` and `fake_ratio`.
518
+ They return schema-valid results in every status and censoring state, and each is labelled as a
519
+ fake rather than an estimate.
520
+
521
+ ## Development
522
+
523
+ ```bash
524
+ pip install -e ".[dev]"
525
+ ruff check . && ruff format --check .
526
+ mypy
527
+ pytest # enforces 90% branch coverage
528
+ ```
529
+
530
+ CI runs the tests on Python 3.11, 3.12, and 3.13 against current dependencies. It also runs them
531
+ on Python 3.11 against the oldest supported versions, pinned in
532
+ [`ci/constraints-min.txt`](https://github.com/robustmargin/robustmargin/blob/main/ci/constraints-min.txt).
533
+ Changes are recorded in [`CHANGELOG.md`](https://github.com/robustmargin/robustmargin/blob/main/CHANGELOG.md).
534
+
535
+ ## License
536
+
537
+ Apache-2.0. See [`LICENSE`](https://github.com/robustmargin/robustmargin/blob/main/LICENSE).