robustmargin 0.3.0__tar.gz → 0.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- robustmargin-0.4.0/NOTICE +4 -0
- {robustmargin-0.3.0/src/robustmargin.egg-info → robustmargin-0.4.0}/PKG-INFO +256 -17
- {robustmargin-0.3.0 → robustmargin-0.4.0}/README.md +250 -14
- {robustmargin-0.3.0 → robustmargin-0.4.0}/pyproject.toml +7 -5
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/__init__.py +21 -7
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/censoring.py +101 -6
- robustmargin-0.4.0/src/robustmargin/design.py +1093 -0
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/empirical.py +2 -2
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/intervals.py +951 -63
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/models.py +824 -47
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/ratio.py +443 -95
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/report.py +233 -48
- robustmargin-0.4.0/src/robustmargin/schema/scorecard-v3.json +1372 -0
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/testing.py +406 -73
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/threshold.py +638 -81
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/types.py +245 -8
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/validation.py +214 -30
- {robustmargin-0.3.0 → robustmargin-0.4.0/src/robustmargin.egg-info}/PKG-INFO +256 -17
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin.egg-info/SOURCES.txt +4 -1
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin.egg-info/requires.txt +1 -1
- {robustmargin-0.3.0 → robustmargin-0.4.0}/LICENSE +0 -0
- {robustmargin-0.3.0 → robustmargin-0.4.0}/setup.cfg +0 -0
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/py.typed +0 -0
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/schema/scorecard-v1.json +0 -0
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/schema/scorecard-v2.json +0 -0
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin.egg-info/dependency_links.txt +0 -0
- {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin.egg-info/top_level.txt +0 -0
|
@@ -1,13 +1,15 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: robustmargin
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.4.0
|
|
4
4
|
Summary: Robustness margins for AI safety evaluation: how much stress an AI system takes before it fails, with a confidence interval.
|
|
5
|
-
Author:
|
|
5
|
+
Author: Babak Badkoubeh
|
|
6
6
|
License-Expression: Apache-2.0
|
|
7
7
|
Project-URL: Homepage, https://github.com/robustmargin/robustmargin
|
|
8
8
|
Project-URL: Source, https://github.com/robustmargin/robustmargin
|
|
9
9
|
Project-URL: Changelog, https://github.com/robustmargin/robustmargin/blob/main/CHANGELOG.md
|
|
10
10
|
Project-URL: Issues, https://github.com/robustmargin/robustmargin/issues
|
|
11
|
+
Project-URL: DOI, https://doi.org/10.5281/zenodo.23585829
|
|
12
|
+
Project-URL: Author ORCID, https://orcid.org/0009-0005-3586-7901
|
|
11
13
|
Keywords: ai-safety,ai-evaluation,robustness,stress-testing,autonomous-systems,dose-response,confidence-intervals,statistics
|
|
12
14
|
Classifier: Development Status :: 3 - Alpha
|
|
13
15
|
Classifier: Intended Audience :: Science/Research
|
|
@@ -22,8 +24,9 @@ Classifier: Typing :: Typed
|
|
|
22
24
|
Requires-Python: >=3.11
|
|
23
25
|
Description-Content-Type: text/markdown
|
|
24
26
|
License-File: LICENSE
|
|
27
|
+
License-File: NOTICE
|
|
25
28
|
Requires-Dist: numpy>=1.26
|
|
26
|
-
Requires-Dist: scipy>=1.
|
|
29
|
+
Requires-Dist: scipy>=1.12
|
|
27
30
|
Requires-Dist: statsmodels>=0.14
|
|
28
31
|
Provides-Extra: dev
|
|
29
32
|
Requires-Dist: pytest>=8.0; extra == "dev"
|
|
@@ -58,12 +61,12 @@ robustness and safety carry weight:
|
|
|
58
61
|
- **Autonomous and safety-critical systems**: margins reported with honest uncertainty, and
|
|
59
62
|
censored bounds instead of invented numbers when the data cannot support an estimate.
|
|
60
63
|
|
|
61
|
-
> **Status: latest release `0.
|
|
62
|
-
> with severity and for real-valued scores (Gaussian least squares),
|
|
63
|
-
> profile-likelihood or
|
|
64
|
-
> Fieller or log-delta intervals,
|
|
65
|
-
>
|
|
66
|
-
> is below 1.0, a minor release may change the public API.
|
|
64
|
+
> **Status: latest release `0.4.0`.** Dose-response fits for pass/fail outcomes that fall or rise
|
|
65
|
+
> with severity and for real-valued scores (Gaussian least squares), model-free isotonic fits,
|
|
66
|
+
> thresholds with profile-likelihood, delta or bootstrap confidence intervals, censoring, ratios
|
|
67
|
+
> of two thresholds with Fieller or log-delta intervals, clustered and paired inference for
|
|
68
|
+
> outcomes that share scenarios, simulation-based design planning, fit diagnostics, and the grid
|
|
69
|
+
> break-point rule. While the version is below 1.0, a minor release may change the public API.
|
|
67
70
|
|
|
68
71
|
## What it does
|
|
69
72
|
|
|
@@ -100,7 +103,10 @@ its results into robustmargin's input:
|
|
|
100
103
|
Because the interface is only severities and outcomes, the same adapter pattern works for any
|
|
101
104
|
evaluator. A typical use is comparing checkpoints, for example before and after RL
|
|
102
105
|
post-training: `ratio_interval` gives a confidence interval for the ratio of their margins, so
|
|
103
|
-
"the new checkpoint is more robust" becomes a claim with an interval attached.
|
|
106
|
+
"the new checkpoint is more robust" becomes a claim with an interval attached. Before the sweep,
|
|
107
|
+
`robustmargin.design` simulates candidate designs from a pilot fit, so the number of runs per
|
|
108
|
+
level (and of shared scenarios, for a paired comparison) is chosen for the interval width you
|
|
109
|
+
need.
|
|
104
110
|
|
|
105
111
|
## Install
|
|
106
112
|
|
|
@@ -308,8 +314,8 @@ assert any(w.startswith("OVERDISPERSED: ") for w in form_b.diagnostics.warnings)
|
|
|
308
314
|
assert any(w.startswith("NON_MONOTONE_DATA: ") for w in ratio.warnings)
|
|
309
315
|
```
|
|
310
316
|
|
|
311
|
-
Treat an overdispersed fit's intervals as too narrow.
|
|
312
|
-
|
|
317
|
+
Treat an overdispersed fit's intervals as too narrow. When the extra variation comes from trials
|
|
318
|
+
that share a cluster, see "Repeated scenarios: clustered and paired inference" below.
|
|
313
319
|
|
|
314
320
|
### An outcome that rises with severity
|
|
315
321
|
|
|
@@ -426,6 +432,226 @@ assert abs(fit.predict(margin.value).estimate - 0.7) < 1e-6
|
|
|
426
432
|
A ratio of two continuous thresholds uses a Student-t quantile on Welch-Satterthwaite degrees of
|
|
427
433
|
freedom, and `ratio_interval` raises `ValueError` for a count threshold against a continuous one.
|
|
428
434
|
|
|
435
|
+
### A model-free fit and bootstrap intervals
|
|
436
|
+
|
|
437
|
+
When you would rather assume only that performance falls (or rises) with severity, fit
|
|
438
|
+
`model="isotonic"`: it takes counts or a continuous response, needs no `link`, `upper` or `lower`,
|
|
439
|
+
and returns the monotone least-squares value of every tested level in `Fit.cell_estimates` (found
|
|
440
|
+
by pool-adjacent-violators). There is no curve between tested levels, so `Fit.predict` and
|
|
441
|
+
`diagnose` raise. A threshold on an isotonic fit is where the straight line drawn through those
|
|
442
|
+
values (on `log` severity for a log axis) first reaches the target. Nothing is extrapolated: a
|
|
443
|
+
target the line never reaches inside the tested levels is `UNREACHABLE` with no number. For counts
|
|
444
|
+
the exact bounds (`RIGHT`, `LEFT`, `FAILS_AT_BASELINE`) still come first. `relative` raises,
|
|
445
|
+
because there are no asymptotes, and `baseline_fraction` and `baseline_multiple` use the isotonic
|
|
446
|
+
value at the zero-severity control as the baseline.
|
|
447
|
+
|
|
448
|
+
An isotonic fit has no covariance, so its interval is a bootstrap: `interval_method="cell_bootstrap"`
|
|
449
|
+
resamples each tested level from its own data, refits, and reports percentiles of the resampled
|
|
450
|
+
thresholds. `n_resamples` defaults to 999 and `seed` makes the run reproducible; when you give no
|
|
451
|
+
seed one is drawn and recorded in `Threshold.bootstrap`, so any result can be rerun. A resampled
|
|
452
|
+
curve that never reaches the target counts toward an open side of the interval, so the interval
|
|
453
|
+
can be one-sided, or carry no bounds when the data cannot say either way.
|
|
454
|
+
|
|
455
|
+
```python
|
|
456
|
+
from robustmargin import Axis, Definition, Observations, Status, fit_dose_response, threshold
|
|
457
|
+
|
|
458
|
+
obs = Observations.from_counts(
|
|
459
|
+
Axis(name="severity", unit="unit", scale="log"),
|
|
460
|
+
severity=[0.0, 0.5, 1.0, 2.0, 4.0, 8.0],
|
|
461
|
+
successes=[100, 98, 90, 60, 20, 5],
|
|
462
|
+
trials=[100] * 6,
|
|
463
|
+
outcome="success",
|
|
464
|
+
direction="decreasing",
|
|
465
|
+
)
|
|
466
|
+
fit = fit_dose_response(obs, model="isotonic")
|
|
467
|
+
assert fit.cell_estimates == (1.0, 0.98, 0.9, 0.6, 0.2, 0.05) # already monotone
|
|
468
|
+
|
|
469
|
+
# The severity at which success falls to 50%, with a 95% bootstrap interval.
|
|
470
|
+
margin = threshold(
|
|
471
|
+
fit,
|
|
472
|
+
definition=Definition.absolute(0.5),
|
|
473
|
+
interval_method="cell_bootstrap",
|
|
474
|
+
level=0.95,
|
|
475
|
+
n_resamples=499,
|
|
476
|
+
seed=7,
|
|
477
|
+
)
|
|
478
|
+
assert margin.status is Status.OK
|
|
479
|
+
assert 2.0 < margin.value < 4.0
|
|
480
|
+
assert margin.lo <= margin.hi
|
|
481
|
+
assert margin.bootstrap.seed == 7 and margin.bootstrap.failed == 0
|
|
482
|
+
```
|
|
483
|
+
|
|
484
|
+
`interval_method="parametric_bootstrap"` does the same for a binomial or Gaussian fit: it simulates
|
|
485
|
+
new data from the fitted curve at the tested levels and refits the same model. On a fit with an
|
|
486
|
+
estimated asymptote each refit is far slower than with fixed asymptotes, so give it a smaller
|
|
487
|
+
`n_resamples`. A replicate whose refit fails is counted in `Threshold.bootstrap` and enters the
|
|
488
|
+
percentiles as the worst case for each end (so failures can only widen the interval), with a warning
|
|
489
|
+
above 1% and no interval above 10% (or above `alpha/2`, if that is smaller).
|
|
490
|
+
|
|
491
|
+
### Repeated scenarios: clustered and paired inference
|
|
492
|
+
|
|
493
|
+
When many trials come from the same scenario, they are not independent, and intervals that assume
|
|
494
|
+
they are come out too narrow. Give `Observations` a `cluster` id for every row and ask for
|
|
495
|
+
`dependence="clustered"`. The point estimate is the usual maximum-likelihood fit, unchanged; its
|
|
496
|
+
covariance becomes a cluster-robust sandwich that uses only the variation between clusters, and
|
|
497
|
+
`Fit.cluster_cells` keeps the per-cluster counts so that whole clusters can be resampled. This is
|
|
498
|
+
for count outcomes (`model="binomial"`) only. A clustered fit needs more clusters than it has
|
|
499
|
+
estimated parameters (`ValueError` otherwise), and a result built on few clusters carries a
|
|
500
|
+
`FEW_CLUSTERS:` warning stating the count it considers few. Monotonicity is checked by an exact sign
|
|
501
|
+
test across clusters (`check_monotonicity(obs, dependence="clustered")`).
|
|
502
|
+
|
|
503
|
+
A clustered threshold makes no exact claim, because the exact bounds treat trials as independent.
|
|
504
|
+
Its recommended interval is `"delta"`: the sandwich covariance with a Student-t quantile on
|
|
505
|
+
`G - 1` degrees of freedom. `"cluster_bootstrap"` (resample whole clusters with replacement and
|
|
506
|
+
refit) is an alternative that needs about 40 or more clusters, because a percentile interval from
|
|
507
|
+
few clusters covers less than its stated level; the other methods raise. A beyond-range crossing is
|
|
508
|
+
an `EXTRAPOLATED` value, not a `RIGHT` bound. When two outcomes are measured on the same clusters,
|
|
509
|
+
the ratio of their thresholds takes `dependence="paired"`, with `method="log_delta"` (or
|
|
510
|
+
`"fieller"`), which keeps the correlation between the two. The example below uses 12 clusters, so
|
|
511
|
+
its results carry the `FEW_CLUSTERS:` warning, which is raised below 20 clusters (below 40 for
|
|
512
|
+
`"cluster_bootstrap"`).
|
|
513
|
+
|
|
514
|
+
```python
|
|
515
|
+
import math
|
|
516
|
+
|
|
517
|
+
import numpy as np
|
|
518
|
+
|
|
519
|
+
from robustmargin import (
|
|
520
|
+
Axis,
|
|
521
|
+
Definition,
|
|
522
|
+
Observations,
|
|
523
|
+
Status,
|
|
524
|
+
fit_dose_response,
|
|
525
|
+
ratio_interval,
|
|
526
|
+
threshold,
|
|
527
|
+
)
|
|
528
|
+
|
|
529
|
+
axis = Axis(name="severity", unit="unit", scale="log")
|
|
530
|
+
levels = [0.0, 1.0, 2.0, 4.0, 8.0, 16.0, 32.0]
|
|
531
|
+
n_clusters, trials = 12, 20
|
|
532
|
+
rng = np.random.default_rng(3)
|
|
533
|
+
scenario_shift = rng.normal(0.0, 0.4, n_clusters) # each scenario is easier or harder
|
|
534
|
+
|
|
535
|
+
|
|
536
|
+
def observe(log_midpoint_shift: float) -> Observations:
|
|
537
|
+
severity, successes, cluster = [], [], []
|
|
538
|
+
for g in range(n_clusters):
|
|
539
|
+
for x in levels:
|
|
540
|
+
z = (math.log(x) - (math.log(6.0) + log_midpoint_shift)) / 0.6 if x > 0 else -4.0
|
|
541
|
+
p = 1.0 - 0.5 * (1.0 + math.erf((z + scenario_shift[g]) / math.sqrt(2.0)))
|
|
542
|
+
severity.append(x)
|
|
543
|
+
successes.append(int(rng.binomial(trials, p)))
|
|
544
|
+
cluster.append(f"scenario-{g}")
|
|
545
|
+
return Observations.from_counts(
|
|
546
|
+
axis,
|
|
547
|
+
severity=severity,
|
|
548
|
+
successes=successes,
|
|
549
|
+
trials=[trials] * len(severity),
|
|
550
|
+
outcome="success",
|
|
551
|
+
direction="decreasing",
|
|
552
|
+
cluster=cluster,
|
|
553
|
+
)
|
|
554
|
+
|
|
555
|
+
|
|
556
|
+
margins = {}
|
|
557
|
+
for name, shift in {"a": 0.0, "b": 0.5}.items():
|
|
558
|
+
fit = fit_dose_response(
|
|
559
|
+
observe(shift),
|
|
560
|
+
model="binomial",
|
|
561
|
+
link="probit",
|
|
562
|
+
upper=1.0,
|
|
563
|
+
lower=0.0,
|
|
564
|
+
dependence="clustered",
|
|
565
|
+
)
|
|
566
|
+
assert fit.status is Status.OK and fit.dependence == "clustered"
|
|
567
|
+
margins[name] = threshold(
|
|
568
|
+
fit,
|
|
569
|
+
definition=Definition.absolute(0.5),
|
|
570
|
+
interval_method="delta",
|
|
571
|
+
dependence="clustered",
|
|
572
|
+
level=0.95,
|
|
573
|
+
)
|
|
574
|
+
assert margins[name].status is Status.OK
|
|
575
|
+
assert margins[name].lo < margins[name].value < margins[name].hi
|
|
576
|
+
|
|
577
|
+
# Outcome "b" is measured on the same twelve scenarios as "a": a paired ratio of the two margins.
|
|
578
|
+
ratio = ratio_interval(
|
|
579
|
+
margins["b"],
|
|
580
|
+
margins["a"],
|
|
581
|
+
method="log_delta",
|
|
582
|
+
dependence="paired",
|
|
583
|
+
level=0.95,
|
|
584
|
+
)
|
|
585
|
+
assert ratio.status is Status.OK and ratio.dependence == "paired"
|
|
586
|
+
assert 1.0 < ratio.lo < ratio.estimate < ratio.hi
|
|
587
|
+
```
|
|
588
|
+
|
|
589
|
+
`method="fieller"` and `"log_delta"` with `dependence="paired"` use the covariance between the two
|
|
590
|
+
fits' cluster-robust estimates and Student-t quantiles on `G - 1` degrees of freedom. Both
|
|
591
|
+
thresholds must come from clustered fits over the same set of cluster ids. `method="cluster_bootstrap"`
|
|
592
|
+
with `dependence="paired"` resamples the same clusters for both and is the alternative; it also
|
|
593
|
+
computes the sandwich standard error alongside, and a `CLUSTER_CROSS_CHECK:` warning says when the
|
|
594
|
+
two disagree by more than a stated tolerance. A ratio of clustered thresholds on disjoint sets of
|
|
595
|
+
clusters (`dependence="independent"`) uses a Student-t quantile with Welch-Satterthwaite degrees of
|
|
596
|
+
freedom.
|
|
597
|
+
|
|
598
|
+
### Planning a design
|
|
599
|
+
|
|
600
|
+
Before running a sweep, `robustmargin.design` estimates how wide the interval will be. Give it a
|
|
601
|
+
pilot `Fit` (say, from a coarse sweep) and a candidate design: it treats the pilot's point
|
|
602
|
+
estimates as the truth, simulates whole experiments at that design, refits the same model on each,
|
|
603
|
+
and calls the real `threshold` with your interval method. `severity_grid` builds the levels. Each
|
|
604
|
+
replicate gets its own random stream from `seed`, so the result is the same for any `workers`. With
|
|
605
|
+
`n_clusters` and `icc` the trials come in clusters, with the pilot curve as every level's marginal
|
|
606
|
+
success rate; `icc` is the latent correlation of the underlying normal variables, smaller on the
|
|
607
|
+
binary outcomes (a latent 0.3 is about 0.19 at a success rate of 0.5). `plan_ratio_width` does the
|
|
608
|
+
same for a ratio, independent or paired. A bootstrap interval method takes `n_resamples`, and costs
|
|
609
|
+
about 13 s per replicate at the default 999 resamples.
|
|
610
|
+
|
|
611
|
+
```python
|
|
612
|
+
import numpy as np
|
|
613
|
+
from scipy.stats import norm
|
|
614
|
+
|
|
615
|
+
from robustmargin import Axis, Definition, Observations, fit_dose_response
|
|
616
|
+
from robustmargin.design import plan_threshold_width, severity_grid
|
|
617
|
+
|
|
618
|
+
axis = Axis(name="severity", unit="unit", scale="log")
|
|
619
|
+
pilot_levels = severity_grid(axis, lowest=0.1, highest=12.8, n_levels=8, control=False)
|
|
620
|
+
rate = 1.0 - norm.cdf(np.log(pilot_levels) / 0.5)
|
|
621
|
+
pilot = fit_dose_response(
|
|
622
|
+
Observations.from_counts(
|
|
623
|
+
axis,
|
|
624
|
+
severity=pilot_levels,
|
|
625
|
+
successes=[round(100 * p) for p in rate],
|
|
626
|
+
trials=[100] * len(pilot_levels),
|
|
627
|
+
outcome="success",
|
|
628
|
+
direction="decreasing",
|
|
629
|
+
),
|
|
630
|
+
model="binomial",
|
|
631
|
+
link="probit",
|
|
632
|
+
upper=1.0,
|
|
633
|
+
lower=0.0,
|
|
634
|
+
)
|
|
635
|
+
|
|
636
|
+
widths = plan_threshold_width(
|
|
637
|
+
pilot,
|
|
638
|
+
severity=severity_grid(axis, lowest=0.25, highest=4.0, n_levels=5, control=False),
|
|
639
|
+
trials_per_level=100,
|
|
640
|
+
definition=Definition.absolute(0.5),
|
|
641
|
+
interval_method="delta",
|
|
642
|
+
level=0.95,
|
|
643
|
+
n_replicates=20,
|
|
644
|
+
seed=1,
|
|
645
|
+
)
|
|
646
|
+
share, se = widths.share_at_most(1.0) # share of experiments with log(hi / lo) <= 1
|
|
647
|
+
print(widths.quantile(0.5), share, se, widths.outcome_counts())
|
|
648
|
+
```
|
|
649
|
+
|
|
650
|
+
A replicate with no finite width (a censored threshold, a failed fit) is counted as an infinite
|
|
651
|
+
width and tallied by reason in `widths.outcome_counts()`, never dropped. The reasons are `"OK"`, a
|
|
652
|
+
`Status`, `Censoring` or `IntervalShape` value, `"BOTH_SIDES_OPEN"` and `"LOWER_AT_ZERO"`. A `WidthPlan` serialises with
|
|
653
|
+
`robustmargin.report.to_dict` as its own document, separate from the score-card schema.
|
|
654
|
+
|
|
429
655
|
## Grid break-point rule
|
|
430
656
|
|
|
431
657
|
`grid_break_point` finds the smallest tested severity at which observed performance falls
|
|
@@ -507,10 +733,11 @@ assert from_dict(json.loads(text)) == card
|
|
|
507
733
|
assert load_schema()["$schema"] == "https://json-schema.org/draft/2020-12/schema"
|
|
508
734
|
```
|
|
509
735
|
|
|
510
|
-
The JSON follows `schema/scorecard-
|
|
511
|
-
`schema/scorecard-v1.json`
|
|
512
|
-
|
|
513
|
-
and
|
|
736
|
+
The JSON follows `schema/scorecard-v3.json`, a JSON Schema shipped inside the package, and
|
|
737
|
+
`schema/scorecard-v1.json` and `schema/scorecard-v2.json` are kept so that cards written by
|
|
738
|
+
earlier releases still read, and are written back as `"3"`. Schema `"2"` added increasing outcomes,
|
|
739
|
+
`baseline_multiple` and the Gaussian model; schema `"3"` adds the isotonic model, the two bootstrap
|
|
740
|
+
interval methods, `Fit.cell_estimates` and `Threshold.bootstrap`. Reading is strict: a card with fields this version does not know is rejected, not silently
|
|
514
741
|
trimmed. To read a stored card, use a robustmargin at least as new as the
|
|
515
742
|
one that wrote it. The card's `library_version` field records which version that was.
|
|
516
743
|
|
|
@@ -532,6 +759,18 @@ on Python 3.11 against the oldest supported versions, pinned in
|
|
|
532
759
|
[`ci/constraints-min.txt`](https://github.com/robustmargin/robustmargin/blob/main/ci/constraints-min.txt).
|
|
533
760
|
Changes are recorded in [`CHANGELOG.md`](https://github.com/robustmargin/robustmargin/blob/main/CHANGELOG.md).
|
|
534
761
|
|
|
762
|
+
## Citing
|
|
763
|
+
|
|
764
|
+
robustmargin is written and maintained by Babak Badkoubeh
|
|
765
|
+
([ORCID profile](https://orcid.org/0009-0005-3586-7901)). If you use it in published work, please
|
|
766
|
+
cite it. Each release is archived on Zenodo
|
|
767
|
+
([doi:10.5281/zenodo.23585829](https://doi.org/10.5281/zenodo.23585829)), and the citation
|
|
768
|
+
metadata is in
|
|
769
|
+
[`CITATION.cff`](https://github.com/robustmargin/robustmargin/blob/main/CITATION.cff); GitHub's
|
|
770
|
+
"Cite this repository" button renders it as BibTeX or APA.
|
|
771
|
+
|
|
535
772
|
## License
|
|
536
773
|
|
|
537
|
-
Apache-2.0. See
|
|
774
|
+
Apache-2.0, copyright 2026 Babak Badkoubeh. See
|
|
775
|
+
[`LICENSE`](https://github.com/robustmargin/robustmargin/blob/main/LICENSE) and
|
|
776
|
+
[`NOTICE`](https://github.com/robustmargin/robustmargin/blob/main/NOTICE).
|
|
@@ -21,12 +21,12 @@ robustness and safety carry weight:
|
|
|
21
21
|
- **Autonomous and safety-critical systems**: margins reported with honest uncertainty, and
|
|
22
22
|
censored bounds instead of invented numbers when the data cannot support an estimate.
|
|
23
23
|
|
|
24
|
-
> **Status: latest release `0.
|
|
25
|
-
> with severity and for real-valued scores (Gaussian least squares),
|
|
26
|
-
> profile-likelihood or
|
|
27
|
-
> Fieller or log-delta intervals,
|
|
28
|
-
>
|
|
29
|
-
> is below 1.0, a minor release may change the public API.
|
|
24
|
+
> **Status: latest release `0.4.0`.** Dose-response fits for pass/fail outcomes that fall or rise
|
|
25
|
+
> with severity and for real-valued scores (Gaussian least squares), model-free isotonic fits,
|
|
26
|
+
> thresholds with profile-likelihood, delta or bootstrap confidence intervals, censoring, ratios
|
|
27
|
+
> of two thresholds with Fieller or log-delta intervals, clustered and paired inference for
|
|
28
|
+
> outcomes that share scenarios, simulation-based design planning, fit diagnostics, and the grid
|
|
29
|
+
> break-point rule. While the version is below 1.0, a minor release may change the public API.
|
|
30
30
|
|
|
31
31
|
## What it does
|
|
32
32
|
|
|
@@ -63,7 +63,10 @@ its results into robustmargin's input:
|
|
|
63
63
|
Because the interface is only severities and outcomes, the same adapter pattern works for any
|
|
64
64
|
evaluator. A typical use is comparing checkpoints, for example before and after RL
|
|
65
65
|
post-training: `ratio_interval` gives a confidence interval for the ratio of their margins, so
|
|
66
|
-
"the new checkpoint is more robust" becomes a claim with an interval attached.
|
|
66
|
+
"the new checkpoint is more robust" becomes a claim with an interval attached. Before the sweep,
|
|
67
|
+
`robustmargin.design` simulates candidate designs from a pilot fit, so the number of runs per
|
|
68
|
+
level (and of shared scenarios, for a paired comparison) is chosen for the interval width you
|
|
69
|
+
need.
|
|
67
70
|
|
|
68
71
|
## Install
|
|
69
72
|
|
|
@@ -271,8 +274,8 @@ assert any(w.startswith("OVERDISPERSED: ") for w in form_b.diagnostics.warnings)
|
|
|
271
274
|
assert any(w.startswith("NON_MONOTONE_DATA: ") for w in ratio.warnings)
|
|
272
275
|
```
|
|
273
276
|
|
|
274
|
-
Treat an overdispersed fit's intervals as too narrow.
|
|
275
|
-
|
|
277
|
+
Treat an overdispersed fit's intervals as too narrow. When the extra variation comes from trials
|
|
278
|
+
that share a cluster, see "Repeated scenarios: clustered and paired inference" below.
|
|
276
279
|
|
|
277
280
|
### An outcome that rises with severity
|
|
278
281
|
|
|
@@ -389,6 +392,226 @@ assert abs(fit.predict(margin.value).estimate - 0.7) < 1e-6
|
|
|
389
392
|
A ratio of two continuous thresholds uses a Student-t quantile on Welch-Satterthwaite degrees of
|
|
390
393
|
freedom, and `ratio_interval` raises `ValueError` for a count threshold against a continuous one.
|
|
391
394
|
|
|
395
|
+
### A model-free fit and bootstrap intervals
|
|
396
|
+
|
|
397
|
+
When you would rather assume only that performance falls (or rises) with severity, fit
|
|
398
|
+
`model="isotonic"`: it takes counts or a continuous response, needs no `link`, `upper` or `lower`,
|
|
399
|
+
and returns the monotone least-squares value of every tested level in `Fit.cell_estimates` (found
|
|
400
|
+
by pool-adjacent-violators). There is no curve between tested levels, so `Fit.predict` and
|
|
401
|
+
`diagnose` raise. A threshold on an isotonic fit is where the straight line drawn through those
|
|
402
|
+
values (on `log` severity for a log axis) first reaches the target. Nothing is extrapolated: a
|
|
403
|
+
target the line never reaches inside the tested levels is `UNREACHABLE` with no number. For counts
|
|
404
|
+
the exact bounds (`RIGHT`, `LEFT`, `FAILS_AT_BASELINE`) still come first. `relative` raises,
|
|
405
|
+
because there are no asymptotes, and `baseline_fraction` and `baseline_multiple` use the isotonic
|
|
406
|
+
value at the zero-severity control as the baseline.
|
|
407
|
+
|
|
408
|
+
An isotonic fit has no covariance, so its interval is a bootstrap: `interval_method="cell_bootstrap"`
|
|
409
|
+
resamples each tested level from its own data, refits, and reports percentiles of the resampled
|
|
410
|
+
thresholds. `n_resamples` defaults to 999 and `seed` makes the run reproducible; when you give no
|
|
411
|
+
seed one is drawn and recorded in `Threshold.bootstrap`, so any result can be rerun. A resampled
|
|
412
|
+
curve that never reaches the target counts toward an open side of the interval, so the interval
|
|
413
|
+
can be one-sided, or carry no bounds when the data cannot say either way.
|
|
414
|
+
|
|
415
|
+
```python
|
|
416
|
+
from robustmargin import Axis, Definition, Observations, Status, fit_dose_response, threshold
|
|
417
|
+
|
|
418
|
+
obs = Observations.from_counts(
|
|
419
|
+
Axis(name="severity", unit="unit", scale="log"),
|
|
420
|
+
severity=[0.0, 0.5, 1.0, 2.0, 4.0, 8.0],
|
|
421
|
+
successes=[100, 98, 90, 60, 20, 5],
|
|
422
|
+
trials=[100] * 6,
|
|
423
|
+
outcome="success",
|
|
424
|
+
direction="decreasing",
|
|
425
|
+
)
|
|
426
|
+
fit = fit_dose_response(obs, model="isotonic")
|
|
427
|
+
assert fit.cell_estimates == (1.0, 0.98, 0.9, 0.6, 0.2, 0.05) # already monotone
|
|
428
|
+
|
|
429
|
+
# The severity at which success falls to 50%, with a 95% bootstrap interval.
|
|
430
|
+
margin = threshold(
|
|
431
|
+
fit,
|
|
432
|
+
definition=Definition.absolute(0.5),
|
|
433
|
+
interval_method="cell_bootstrap",
|
|
434
|
+
level=0.95,
|
|
435
|
+
n_resamples=499,
|
|
436
|
+
seed=7,
|
|
437
|
+
)
|
|
438
|
+
assert margin.status is Status.OK
|
|
439
|
+
assert 2.0 < margin.value < 4.0
|
|
440
|
+
assert margin.lo <= margin.hi
|
|
441
|
+
assert margin.bootstrap.seed == 7 and margin.bootstrap.failed == 0
|
|
442
|
+
```
|
|
443
|
+
|
|
444
|
+
`interval_method="parametric_bootstrap"` does the same for a binomial or Gaussian fit: it simulates
|
|
445
|
+
new data from the fitted curve at the tested levels and refits the same model. On a fit with an
|
|
446
|
+
estimated asymptote each refit is far slower than with fixed asymptotes, so give it a smaller
|
|
447
|
+
`n_resamples`. A replicate whose refit fails is counted in `Threshold.bootstrap` and enters the
|
|
448
|
+
percentiles as the worst case for each end (so failures can only widen the interval), with a warning
|
|
449
|
+
above 1% and no interval above 10% (or above `alpha/2`, if that is smaller).
|
|
450
|
+
|
|
451
|
+
### Repeated scenarios: clustered and paired inference
|
|
452
|
+
|
|
453
|
+
When many trials come from the same scenario, they are not independent, and intervals that assume
|
|
454
|
+
they are come out too narrow. Give `Observations` a `cluster` id for every row and ask for
|
|
455
|
+
`dependence="clustered"`. The point estimate is the usual maximum-likelihood fit, unchanged; its
|
|
456
|
+
covariance becomes a cluster-robust sandwich that uses only the variation between clusters, and
|
|
457
|
+
`Fit.cluster_cells` keeps the per-cluster counts so that whole clusters can be resampled. This is
|
|
458
|
+
for count outcomes (`model="binomial"`) only. A clustered fit needs more clusters than it has
|
|
459
|
+
estimated parameters (`ValueError` otherwise), and a result built on few clusters carries a
|
|
460
|
+
`FEW_CLUSTERS:` warning stating the count it considers few. Monotonicity is checked by an exact sign
|
|
461
|
+
test across clusters (`check_monotonicity(obs, dependence="clustered")`).
|
|
462
|
+
|
|
463
|
+
A clustered threshold makes no exact claim, because the exact bounds treat trials as independent.
|
|
464
|
+
Its recommended interval is `"delta"`: the sandwich covariance with a Student-t quantile on
|
|
465
|
+
`G - 1` degrees of freedom. `"cluster_bootstrap"` (resample whole clusters with replacement and
|
|
466
|
+
refit) is an alternative that needs about 40 or more clusters, because a percentile interval from
|
|
467
|
+
few clusters covers less than its stated level; the other methods raise. A beyond-range crossing is
|
|
468
|
+
an `EXTRAPOLATED` value, not a `RIGHT` bound. When two outcomes are measured on the same clusters,
|
|
469
|
+
the ratio of their thresholds takes `dependence="paired"`, with `method="log_delta"` (or
|
|
470
|
+
`"fieller"`), which keeps the correlation between the two. The example below uses 12 clusters, so
|
|
471
|
+
its results carry the `FEW_CLUSTERS:` warning, which is raised below 20 clusters (below 40 for
|
|
472
|
+
`"cluster_bootstrap"`).
|
|
473
|
+
|
|
474
|
+
```python
|
|
475
|
+
import math
|
|
476
|
+
|
|
477
|
+
import numpy as np
|
|
478
|
+
|
|
479
|
+
from robustmargin import (
|
|
480
|
+
Axis,
|
|
481
|
+
Definition,
|
|
482
|
+
Observations,
|
|
483
|
+
Status,
|
|
484
|
+
fit_dose_response,
|
|
485
|
+
ratio_interval,
|
|
486
|
+
threshold,
|
|
487
|
+
)
|
|
488
|
+
|
|
489
|
+
axis = Axis(name="severity", unit="unit", scale="log")
|
|
490
|
+
levels = [0.0, 1.0, 2.0, 4.0, 8.0, 16.0, 32.0]
|
|
491
|
+
n_clusters, trials = 12, 20
|
|
492
|
+
rng = np.random.default_rng(3)
|
|
493
|
+
scenario_shift = rng.normal(0.0, 0.4, n_clusters) # each scenario is easier or harder
|
|
494
|
+
|
|
495
|
+
|
|
496
|
+
def observe(log_midpoint_shift: float) -> Observations:
|
|
497
|
+
severity, successes, cluster = [], [], []
|
|
498
|
+
for g in range(n_clusters):
|
|
499
|
+
for x in levels:
|
|
500
|
+
z = (math.log(x) - (math.log(6.0) + log_midpoint_shift)) / 0.6 if x > 0 else -4.0
|
|
501
|
+
p = 1.0 - 0.5 * (1.0 + math.erf((z + scenario_shift[g]) / math.sqrt(2.0)))
|
|
502
|
+
severity.append(x)
|
|
503
|
+
successes.append(int(rng.binomial(trials, p)))
|
|
504
|
+
cluster.append(f"scenario-{g}")
|
|
505
|
+
return Observations.from_counts(
|
|
506
|
+
axis,
|
|
507
|
+
severity=severity,
|
|
508
|
+
successes=successes,
|
|
509
|
+
trials=[trials] * len(severity),
|
|
510
|
+
outcome="success",
|
|
511
|
+
direction="decreasing",
|
|
512
|
+
cluster=cluster,
|
|
513
|
+
)
|
|
514
|
+
|
|
515
|
+
|
|
516
|
+
margins = {}
|
|
517
|
+
for name, shift in {"a": 0.0, "b": 0.5}.items():
|
|
518
|
+
fit = fit_dose_response(
|
|
519
|
+
observe(shift),
|
|
520
|
+
model="binomial",
|
|
521
|
+
link="probit",
|
|
522
|
+
upper=1.0,
|
|
523
|
+
lower=0.0,
|
|
524
|
+
dependence="clustered",
|
|
525
|
+
)
|
|
526
|
+
assert fit.status is Status.OK and fit.dependence == "clustered"
|
|
527
|
+
margins[name] = threshold(
|
|
528
|
+
fit,
|
|
529
|
+
definition=Definition.absolute(0.5),
|
|
530
|
+
interval_method="delta",
|
|
531
|
+
dependence="clustered",
|
|
532
|
+
level=0.95,
|
|
533
|
+
)
|
|
534
|
+
assert margins[name].status is Status.OK
|
|
535
|
+
assert margins[name].lo < margins[name].value < margins[name].hi
|
|
536
|
+
|
|
537
|
+
# Outcome "b" is measured on the same twelve scenarios as "a": a paired ratio of the two margins.
|
|
538
|
+
ratio = ratio_interval(
|
|
539
|
+
margins["b"],
|
|
540
|
+
margins["a"],
|
|
541
|
+
method="log_delta",
|
|
542
|
+
dependence="paired",
|
|
543
|
+
level=0.95,
|
|
544
|
+
)
|
|
545
|
+
assert ratio.status is Status.OK and ratio.dependence == "paired"
|
|
546
|
+
assert 1.0 < ratio.lo < ratio.estimate < ratio.hi
|
|
547
|
+
```
|
|
548
|
+
|
|
549
|
+
`method="fieller"` and `"log_delta"` with `dependence="paired"` use the covariance between the two
|
|
550
|
+
fits' cluster-robust estimates and Student-t quantiles on `G - 1` degrees of freedom. Both
|
|
551
|
+
thresholds must come from clustered fits over the same set of cluster ids. `method="cluster_bootstrap"`
|
|
552
|
+
with `dependence="paired"` resamples the same clusters for both and is the alternative; it also
|
|
553
|
+
computes the sandwich standard error alongside, and a `CLUSTER_CROSS_CHECK:` warning says when the
|
|
554
|
+
two disagree by more than a stated tolerance. A ratio of clustered thresholds on disjoint sets of
|
|
555
|
+
clusters (`dependence="independent"`) uses a Student-t quantile with Welch-Satterthwaite degrees of
|
|
556
|
+
freedom.
|
|
557
|
+
|
|
558
|
+
### Planning a design
|
|
559
|
+
|
|
560
|
+
Before running a sweep, `robustmargin.design` estimates how wide the interval will be. Give it a
|
|
561
|
+
pilot `Fit` (say, from a coarse sweep) and a candidate design: it treats the pilot's point
|
|
562
|
+
estimates as the truth, simulates whole experiments at that design, refits the same model on each,
|
|
563
|
+
and calls the real `threshold` with your interval method. `severity_grid` builds the levels. Each
|
|
564
|
+
replicate gets its own random stream from `seed`, so the result is the same for any `workers`. With
|
|
565
|
+
`n_clusters` and `icc` the trials come in clusters, with the pilot curve as every level's marginal
|
|
566
|
+
success rate; `icc` is the latent correlation of the underlying normal variables, smaller on the
|
|
567
|
+
binary outcomes (a latent 0.3 is about 0.19 at a success rate of 0.5). `plan_ratio_width` does the
|
|
568
|
+
same for a ratio, independent or paired. A bootstrap interval method takes `n_resamples`, and costs
|
|
569
|
+
about 13 s per replicate at the default 999 resamples.
|
|
570
|
+
|
|
571
|
+
```python
|
|
572
|
+
import numpy as np
|
|
573
|
+
from scipy.stats import norm
|
|
574
|
+
|
|
575
|
+
from robustmargin import Axis, Definition, Observations, fit_dose_response
|
|
576
|
+
from robustmargin.design import plan_threshold_width, severity_grid
|
|
577
|
+
|
|
578
|
+
axis = Axis(name="severity", unit="unit", scale="log")
|
|
579
|
+
pilot_levels = severity_grid(axis, lowest=0.1, highest=12.8, n_levels=8, control=False)
|
|
580
|
+
rate = 1.0 - norm.cdf(np.log(pilot_levels) / 0.5)
|
|
581
|
+
pilot = fit_dose_response(
|
|
582
|
+
Observations.from_counts(
|
|
583
|
+
axis,
|
|
584
|
+
severity=pilot_levels,
|
|
585
|
+
successes=[round(100 * p) for p in rate],
|
|
586
|
+
trials=[100] * len(pilot_levels),
|
|
587
|
+
outcome="success",
|
|
588
|
+
direction="decreasing",
|
|
589
|
+
),
|
|
590
|
+
model="binomial",
|
|
591
|
+
link="probit",
|
|
592
|
+
upper=1.0,
|
|
593
|
+
lower=0.0,
|
|
594
|
+
)
|
|
595
|
+
|
|
596
|
+
widths = plan_threshold_width(
|
|
597
|
+
pilot,
|
|
598
|
+
severity=severity_grid(axis, lowest=0.25, highest=4.0, n_levels=5, control=False),
|
|
599
|
+
trials_per_level=100,
|
|
600
|
+
definition=Definition.absolute(0.5),
|
|
601
|
+
interval_method="delta",
|
|
602
|
+
level=0.95,
|
|
603
|
+
n_replicates=20,
|
|
604
|
+
seed=1,
|
|
605
|
+
)
|
|
606
|
+
share, se = widths.share_at_most(1.0) # share of experiments with log(hi / lo) <= 1
|
|
607
|
+
print(widths.quantile(0.5), share, se, widths.outcome_counts())
|
|
608
|
+
```
|
|
609
|
+
|
|
610
|
+
A replicate with no finite width (a censored threshold, a failed fit) is counted as an infinite
|
|
611
|
+
width and tallied by reason in `widths.outcome_counts()`, never dropped. The reasons are `"OK"`, a
|
|
612
|
+
`Status`, `Censoring` or `IntervalShape` value, `"BOTH_SIDES_OPEN"` and `"LOWER_AT_ZERO"`. A `WidthPlan` serialises with
|
|
613
|
+
`robustmargin.report.to_dict` as its own document, separate from the score-card schema.
|
|
614
|
+
|
|
392
615
|
## Grid break-point rule
|
|
393
616
|
|
|
394
617
|
`grid_break_point` finds the smallest tested severity at which observed performance falls
|
|
@@ -470,10 +693,11 @@ assert from_dict(json.loads(text)) == card
|
|
|
470
693
|
assert load_schema()["$schema"] == "https://json-schema.org/draft/2020-12/schema"
|
|
471
694
|
```
|
|
472
695
|
|
|
473
|
-
The JSON follows `schema/scorecard-
|
|
474
|
-
`schema/scorecard-v1.json`
|
|
475
|
-
|
|
476
|
-
and
|
|
696
|
+
The JSON follows `schema/scorecard-v3.json`, a JSON Schema shipped inside the package, and
|
|
697
|
+
`schema/scorecard-v1.json` and `schema/scorecard-v2.json` are kept so that cards written by
|
|
698
|
+
earlier releases still read, and are written back as `"3"`. Schema `"2"` added increasing outcomes,
|
|
699
|
+
`baseline_multiple` and the Gaussian model; schema `"3"` adds the isotonic model, the two bootstrap
|
|
700
|
+
interval methods, `Fit.cell_estimates` and `Threshold.bootstrap`. Reading is strict: a card with fields this version does not know is rejected, not silently
|
|
477
701
|
trimmed. To read a stored card, use a robustmargin at least as new as the
|
|
478
702
|
one that wrote it. The card's `library_version` field records which version that was.
|
|
479
703
|
|
|
@@ -495,6 +719,18 @@ on Python 3.11 against the oldest supported versions, pinned in
|
|
|
495
719
|
[`ci/constraints-min.txt`](https://github.com/robustmargin/robustmargin/blob/main/ci/constraints-min.txt).
|
|
496
720
|
Changes are recorded in [`CHANGELOG.md`](https://github.com/robustmargin/robustmargin/blob/main/CHANGELOG.md).
|
|
497
721
|
|
|
722
|
+
## Citing
|
|
723
|
+
|
|
724
|
+
robustmargin is written and maintained by Babak Badkoubeh
|
|
725
|
+
([ORCID profile](https://orcid.org/0009-0005-3586-7901)). If you use it in published work, please
|
|
726
|
+
cite it. Each release is archived on Zenodo
|
|
727
|
+
([doi:10.5281/zenodo.23585829](https://doi.org/10.5281/zenodo.23585829)), and the citation
|
|
728
|
+
metadata is in
|
|
729
|
+
[`CITATION.cff`](https://github.com/robustmargin/robustmargin/blob/main/CITATION.cff); GitHub's
|
|
730
|
+
"Cite this repository" button renders it as BibTeX or APA.
|
|
731
|
+
|
|
498
732
|
## License
|
|
499
733
|
|
|
500
|
-
Apache-2.0. See
|
|
734
|
+
Apache-2.0, copyright 2026 Babak Badkoubeh. See
|
|
735
|
+
[`LICENSE`](https://github.com/robustmargin/robustmargin/blob/main/LICENSE) and
|
|
736
|
+
[`NOTICE`](https://github.com/robustmargin/robustmargin/blob/main/NOTICE).
|