robustmargin 0.3.0__tar.gz → 0.4.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (27) hide show
  1. robustmargin-0.4.0/NOTICE +4 -0
  2. {robustmargin-0.3.0/src/robustmargin.egg-info → robustmargin-0.4.0}/PKG-INFO +256 -17
  3. {robustmargin-0.3.0 → robustmargin-0.4.0}/README.md +250 -14
  4. {robustmargin-0.3.0 → robustmargin-0.4.0}/pyproject.toml +7 -5
  5. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/__init__.py +21 -7
  6. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/censoring.py +101 -6
  7. robustmargin-0.4.0/src/robustmargin/design.py +1093 -0
  8. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/empirical.py +2 -2
  9. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/intervals.py +951 -63
  10. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/models.py +824 -47
  11. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/ratio.py +443 -95
  12. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/report.py +233 -48
  13. robustmargin-0.4.0/src/robustmargin/schema/scorecard-v3.json +1372 -0
  14. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/testing.py +406 -73
  15. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/threshold.py +638 -81
  16. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/types.py +245 -8
  17. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/validation.py +214 -30
  18. {robustmargin-0.3.0 → robustmargin-0.4.0/src/robustmargin.egg-info}/PKG-INFO +256 -17
  19. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin.egg-info/SOURCES.txt +4 -1
  20. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin.egg-info/requires.txt +1 -1
  21. {robustmargin-0.3.0 → robustmargin-0.4.0}/LICENSE +0 -0
  22. {robustmargin-0.3.0 → robustmargin-0.4.0}/setup.cfg +0 -0
  23. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/py.typed +0 -0
  24. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/schema/scorecard-v1.json +0 -0
  25. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin/schema/scorecard-v2.json +0 -0
  26. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin.egg-info/dependency_links.txt +0 -0
  27. {robustmargin-0.3.0 → robustmargin-0.4.0}/src/robustmargin.egg-info/top_level.txt +0 -0
@@ -0,0 +1,4 @@
1
+ robustmargin
2
+ Copyright 2026 Babak Badkoubeh
3
+
4
+ This product is licensed under the Apache License, Version 2.0 (see LICENSE).
@@ -1,13 +1,15 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: robustmargin
3
- Version: 0.3.0
3
+ Version: 0.4.0
4
4
  Summary: Robustness margins for AI safety evaluation: how much stress an AI system takes before it fails, with a confidence interval.
5
- Author: robustmargin contributors
5
+ Author: Babak Badkoubeh
6
6
  License-Expression: Apache-2.0
7
7
  Project-URL: Homepage, https://github.com/robustmargin/robustmargin
8
8
  Project-URL: Source, https://github.com/robustmargin/robustmargin
9
9
  Project-URL: Changelog, https://github.com/robustmargin/robustmargin/blob/main/CHANGELOG.md
10
10
  Project-URL: Issues, https://github.com/robustmargin/robustmargin/issues
11
+ Project-URL: DOI, https://doi.org/10.5281/zenodo.23585829
12
+ Project-URL: Author ORCID, https://orcid.org/0009-0005-3586-7901
11
13
  Keywords: ai-safety,ai-evaluation,robustness,stress-testing,autonomous-systems,dose-response,confidence-intervals,statistics
12
14
  Classifier: Development Status :: 3 - Alpha
13
15
  Classifier: Intended Audience :: Science/Research
@@ -22,8 +24,9 @@ Classifier: Typing :: Typed
22
24
  Requires-Python: >=3.11
23
25
  Description-Content-Type: text/markdown
24
26
  License-File: LICENSE
27
+ License-File: NOTICE
25
28
  Requires-Dist: numpy>=1.26
26
- Requires-Dist: scipy>=1.11.1
29
+ Requires-Dist: scipy>=1.12
27
30
  Requires-Dist: statsmodels>=0.14
28
31
  Provides-Extra: dev
29
32
  Requires-Dist: pytest>=8.0; extra == "dev"
@@ -58,12 +61,12 @@ robustness and safety carry weight:
58
61
  - **Autonomous and safety-critical systems**: margins reported with honest uncertainty, and
59
62
  censored bounds instead of invented numbers when the data cannot support an estimate.
60
63
 
61
- > **Status: latest release `0.3.0`.** Dose-response fits for pass/fail outcomes that fall or rise
62
- > with severity and for real-valued scores (Gaussian least squares), thresholds with
63
- > profile-likelihood or delta confidence intervals, censoring, ratios of two thresholds with
64
- > Fieller or log-delta intervals, fit diagnostics, and the grid break-point rule. Clustered and
65
- > paired inference, model-free fits and design helpers are planned for `0.4`. While the version
66
- > is below 1.0, a minor release may change the public API.
64
+ > **Status: latest release `0.4.0`.** Dose-response fits for pass/fail outcomes that fall or rise
65
+ > with severity and for real-valued scores (Gaussian least squares), model-free isotonic fits,
66
+ > thresholds with profile-likelihood, delta or bootstrap confidence intervals, censoring, ratios
67
+ > of two thresholds with Fieller or log-delta intervals, clustered and paired inference for
68
+ > outcomes that share scenarios, simulation-based design planning, fit diagnostics, and the grid
69
+ > break-point rule. While the version is below 1.0, a minor release may change the public API.
67
70
 
68
71
  ## What it does
69
72
 
@@ -100,7 +103,10 @@ its results into robustmargin's input:
100
103
  Because the interface is only severities and outcomes, the same adapter pattern works for any
101
104
  evaluator. A typical use is comparing checkpoints, for example before and after RL
102
105
  post-training: `ratio_interval` gives a confidence interval for the ratio of their margins, so
103
- "the new checkpoint is more robust" becomes a claim with an interval attached.
106
+ "the new checkpoint is more robust" becomes a claim with an interval attached. Before the sweep,
107
+ `robustmargin.design` simulates candidate designs from a pilot fit, so the number of runs per
108
+ level (and of shared scenarios, for a paired comparison) is chosen for the interval width you
109
+ need.
104
110
 
105
111
  ## Install
106
112
 
@@ -308,8 +314,8 @@ assert any(w.startswith("OVERDISPERSED: ") for w in form_b.diagnostics.warnings)
308
314
  assert any(w.startswith("NON_MONOTONE_DATA: ") for w in ratio.warnings)
309
315
  ```
310
316
 
311
- Treat an overdispersed fit's intervals as too narrow. Clustered inference, which addresses this,
312
- is planned for `0.4`.
317
+ Treat an overdispersed fit's intervals as too narrow. When the extra variation comes from trials
318
+ that share a cluster, see "Repeated scenarios: clustered and paired inference" below.
313
319
 
314
320
  ### An outcome that rises with severity
315
321
 
@@ -426,6 +432,226 @@ assert abs(fit.predict(margin.value).estimate - 0.7) < 1e-6
426
432
  A ratio of two continuous thresholds uses a Student-t quantile on Welch-Satterthwaite degrees of
427
433
  freedom, and `ratio_interval` raises `ValueError` for a count threshold against a continuous one.
428
434
 
435
+ ### A model-free fit and bootstrap intervals
436
+
437
+ When you would rather assume only that performance falls (or rises) with severity, fit
438
+ `model="isotonic"`: it takes counts or a continuous response, needs no `link`, `upper` or `lower`,
439
+ and returns the monotone least-squares value of every tested level in `Fit.cell_estimates` (found
440
+ by pool-adjacent-violators). There is no curve between tested levels, so `Fit.predict` and
441
+ `diagnose` raise. A threshold on an isotonic fit is where the straight line drawn through those
442
+ values (on `log` severity for a log axis) first reaches the target. Nothing is extrapolated: a
443
+ target the line never reaches inside the tested levels is `UNREACHABLE` with no number. For counts
444
+ the exact bounds (`RIGHT`, `LEFT`, `FAILS_AT_BASELINE`) still come first. `relative` raises,
445
+ because there are no asymptotes, and `baseline_fraction` and `baseline_multiple` use the isotonic
446
+ value at the zero-severity control as the baseline.
447
+
448
+ An isotonic fit has no covariance, so its interval is a bootstrap: `interval_method="cell_bootstrap"`
449
+ resamples each tested level from its own data, refits, and reports percentiles of the resampled
450
+ thresholds. `n_resamples` defaults to 999 and `seed` makes the run reproducible; when you give no
451
+ seed one is drawn and recorded in `Threshold.bootstrap`, so any result can be rerun. A resampled
452
+ curve that never reaches the target counts toward an open side of the interval, so the interval
453
+ can be one-sided, or carry no bounds when the data cannot say either way.
454
+
455
+ ```python
456
+ from robustmargin import Axis, Definition, Observations, Status, fit_dose_response, threshold
457
+
458
+ obs = Observations.from_counts(
459
+ Axis(name="severity", unit="unit", scale="log"),
460
+ severity=[0.0, 0.5, 1.0, 2.0, 4.0, 8.0],
461
+ successes=[100, 98, 90, 60, 20, 5],
462
+ trials=[100] * 6,
463
+ outcome="success",
464
+ direction="decreasing",
465
+ )
466
+ fit = fit_dose_response(obs, model="isotonic")
467
+ assert fit.cell_estimates == (1.0, 0.98, 0.9, 0.6, 0.2, 0.05) # already monotone
468
+
469
+ # The severity at which success falls to 50%, with a 95% bootstrap interval.
470
+ margin = threshold(
471
+ fit,
472
+ definition=Definition.absolute(0.5),
473
+ interval_method="cell_bootstrap",
474
+ level=0.95,
475
+ n_resamples=499,
476
+ seed=7,
477
+ )
478
+ assert margin.status is Status.OK
479
+ assert 2.0 < margin.value < 4.0
480
+ assert margin.lo <= margin.hi
481
+ assert margin.bootstrap.seed == 7 and margin.bootstrap.failed == 0
482
+ ```
483
+
484
+ `interval_method="parametric_bootstrap"` does the same for a binomial or Gaussian fit: it simulates
485
+ new data from the fitted curve at the tested levels and refits the same model. On a fit with an
486
+ estimated asymptote each refit is far slower than with fixed asymptotes, so give it a smaller
487
+ `n_resamples`. A replicate whose refit fails is counted in `Threshold.bootstrap` and enters the
488
+ percentiles as the worst case for each end (so failures can only widen the interval), with a warning
489
+ above 1% and no interval above 10% (or above `alpha/2`, if that is smaller).
490
+
491
+ ### Repeated scenarios: clustered and paired inference
492
+
493
+ When many trials come from the same scenario, they are not independent, and intervals that assume
494
+ they are come out too narrow. Give `Observations` a `cluster` id for every row and ask for
495
+ `dependence="clustered"`. The point estimate is the usual maximum-likelihood fit, unchanged; its
496
+ covariance becomes a cluster-robust sandwich that uses only the variation between clusters, and
497
+ `Fit.cluster_cells` keeps the per-cluster counts so that whole clusters can be resampled. This is
498
+ for count outcomes (`model="binomial"`) only. A clustered fit needs more clusters than it has
499
+ estimated parameters (`ValueError` otherwise), and a result built on few clusters carries a
500
+ `FEW_CLUSTERS:` warning stating the count it considers few. Monotonicity is checked by an exact sign
501
+ test across clusters (`check_monotonicity(obs, dependence="clustered")`).
502
+
503
+ A clustered threshold makes no exact claim, because the exact bounds treat trials as independent.
504
+ Its recommended interval is `"delta"`: the sandwich covariance with a Student-t quantile on
505
+ `G - 1` degrees of freedom. `"cluster_bootstrap"` (resample whole clusters with replacement and
506
+ refit) is an alternative that needs about 40 or more clusters, because a percentile interval from
507
+ few clusters covers less than its stated level; the other methods raise. A beyond-range crossing is
508
+ an `EXTRAPOLATED` value, not a `RIGHT` bound. When two outcomes are measured on the same clusters,
509
+ the ratio of their thresholds takes `dependence="paired"`, with `method="log_delta"` (or
510
+ `"fieller"`), which keeps the correlation between the two. The example below uses 12 clusters, so
511
+ its results carry the `FEW_CLUSTERS:` warning, which is raised below 20 clusters (below 40 for
512
+ `"cluster_bootstrap"`).
513
+
514
+ ```python
515
+ import math
516
+
517
+ import numpy as np
518
+
519
+ from robustmargin import (
520
+ Axis,
521
+ Definition,
522
+ Observations,
523
+ Status,
524
+ fit_dose_response,
525
+ ratio_interval,
526
+ threshold,
527
+ )
528
+
529
+ axis = Axis(name="severity", unit="unit", scale="log")
530
+ levels = [0.0, 1.0, 2.0, 4.0, 8.0, 16.0, 32.0]
531
+ n_clusters, trials = 12, 20
532
+ rng = np.random.default_rng(3)
533
+ scenario_shift = rng.normal(0.0, 0.4, n_clusters) # each scenario is easier or harder
534
+
535
+
536
+ def observe(log_midpoint_shift: float) -> Observations:
537
+ severity, successes, cluster = [], [], []
538
+ for g in range(n_clusters):
539
+ for x in levels:
540
+ z = (math.log(x) - (math.log(6.0) + log_midpoint_shift)) / 0.6 if x > 0 else -4.0
541
+ p = 1.0 - 0.5 * (1.0 + math.erf((z + scenario_shift[g]) / math.sqrt(2.0)))
542
+ severity.append(x)
543
+ successes.append(int(rng.binomial(trials, p)))
544
+ cluster.append(f"scenario-{g}")
545
+ return Observations.from_counts(
546
+ axis,
547
+ severity=severity,
548
+ successes=successes,
549
+ trials=[trials] * len(severity),
550
+ outcome="success",
551
+ direction="decreasing",
552
+ cluster=cluster,
553
+ )
554
+
555
+
556
+ margins = {}
557
+ for name, shift in {"a": 0.0, "b": 0.5}.items():
558
+ fit = fit_dose_response(
559
+ observe(shift),
560
+ model="binomial",
561
+ link="probit",
562
+ upper=1.0,
563
+ lower=0.0,
564
+ dependence="clustered",
565
+ )
566
+ assert fit.status is Status.OK and fit.dependence == "clustered"
567
+ margins[name] = threshold(
568
+ fit,
569
+ definition=Definition.absolute(0.5),
570
+ interval_method="delta",
571
+ dependence="clustered",
572
+ level=0.95,
573
+ )
574
+ assert margins[name].status is Status.OK
575
+ assert margins[name].lo < margins[name].value < margins[name].hi
576
+
577
+ # Outcome "b" is measured on the same twelve scenarios as "a": a paired ratio of the two margins.
578
+ ratio = ratio_interval(
579
+ margins["b"],
580
+ margins["a"],
581
+ method="log_delta",
582
+ dependence="paired",
583
+ level=0.95,
584
+ )
585
+ assert ratio.status is Status.OK and ratio.dependence == "paired"
586
+ assert 1.0 < ratio.lo < ratio.estimate < ratio.hi
587
+ ```
588
+
589
+ `method="fieller"` and `"log_delta"` with `dependence="paired"` use the covariance between the two
590
+ fits' cluster-robust estimates and Student-t quantiles on `G - 1` degrees of freedom. Both
591
+ thresholds must come from clustered fits over the same set of cluster ids. `method="cluster_bootstrap"`
592
+ with `dependence="paired"` resamples the same clusters for both and is the alternative; it also
593
+ computes the sandwich standard error alongside, and a `CLUSTER_CROSS_CHECK:` warning says when the
594
+ two disagree by more than a stated tolerance. A ratio of clustered thresholds on disjoint sets of
595
+ clusters (`dependence="independent"`) uses a Student-t quantile with Welch-Satterthwaite degrees of
596
+ freedom.
597
+
598
+ ### Planning a design
599
+
600
+ Before running a sweep, `robustmargin.design` estimates how wide the interval will be. Give it a
601
+ pilot `Fit` (say, from a coarse sweep) and a candidate design: it treats the pilot's point
602
+ estimates as the truth, simulates whole experiments at that design, refits the same model on each,
603
+ and calls the real `threshold` with your interval method. `severity_grid` builds the levels. Each
604
+ replicate gets its own random stream from `seed`, so the result is the same for any `workers`. With
605
+ `n_clusters` and `icc` the trials come in clusters, with the pilot curve as every level's marginal
606
+ success rate; `icc` is the latent correlation of the underlying normal variables, smaller on the
607
+ binary outcomes (a latent 0.3 is about 0.19 at a success rate of 0.5). `plan_ratio_width` does the
608
+ same for a ratio, independent or paired. A bootstrap interval method takes `n_resamples`, and costs
609
+ about 13 s per replicate at the default 999 resamples.
610
+
611
+ ```python
612
+ import numpy as np
613
+ from scipy.stats import norm
614
+
615
+ from robustmargin import Axis, Definition, Observations, fit_dose_response
616
+ from robustmargin.design import plan_threshold_width, severity_grid
617
+
618
+ axis = Axis(name="severity", unit="unit", scale="log")
619
+ pilot_levels = severity_grid(axis, lowest=0.1, highest=12.8, n_levels=8, control=False)
620
+ rate = 1.0 - norm.cdf(np.log(pilot_levels) / 0.5)
621
+ pilot = fit_dose_response(
622
+ Observations.from_counts(
623
+ axis,
624
+ severity=pilot_levels,
625
+ successes=[round(100 * p) for p in rate],
626
+ trials=[100] * len(pilot_levels),
627
+ outcome="success",
628
+ direction="decreasing",
629
+ ),
630
+ model="binomial",
631
+ link="probit",
632
+ upper=1.0,
633
+ lower=0.0,
634
+ )
635
+
636
+ widths = plan_threshold_width(
637
+ pilot,
638
+ severity=severity_grid(axis, lowest=0.25, highest=4.0, n_levels=5, control=False),
639
+ trials_per_level=100,
640
+ definition=Definition.absolute(0.5),
641
+ interval_method="delta",
642
+ level=0.95,
643
+ n_replicates=20,
644
+ seed=1,
645
+ )
646
+ share, se = widths.share_at_most(1.0) # share of experiments with log(hi / lo) <= 1
647
+ print(widths.quantile(0.5), share, se, widths.outcome_counts())
648
+ ```
649
+
650
+ A replicate with no finite width (a censored threshold, a failed fit) is counted as an infinite
651
+ width and tallied by reason in `widths.outcome_counts()`, never dropped. The reasons are `"OK"`, a
652
+ `Status`, `Censoring` or `IntervalShape` value, `"BOTH_SIDES_OPEN"` and `"LOWER_AT_ZERO"`. A `WidthPlan` serialises with
653
+ `robustmargin.report.to_dict` as its own document, separate from the score-card schema.
654
+
429
655
  ## Grid break-point rule
430
656
 
431
657
  `grid_break_point` finds the smallest tested severity at which observed performance falls
@@ -507,10 +733,11 @@ assert from_dict(json.loads(text)) == card
507
733
  assert load_schema()["$schema"] == "https://json-schema.org/draft/2020-12/schema"
508
734
  ```
509
735
 
510
- The JSON follows `schema/scorecard-v2.json`, a JSON Schema shipped inside the package, and
511
- `schema/scorecard-v1.json` is kept so that cards written by `0.1.1` and `0.2.0` still read.
512
- Schema `"2"` adds increasing outcomes and `baseline_multiple`. It covers binary outcomes only,
513
- and reading is strict: a card with fields this version does not know is rejected, not silently
736
+ The JSON follows `schema/scorecard-v3.json`, a JSON Schema shipped inside the package, and
737
+ `schema/scorecard-v1.json` and `schema/scorecard-v2.json` are kept so that cards written by
738
+ earlier releases still read, and are written back as `"3"`. Schema `"2"` added increasing outcomes,
739
+ `baseline_multiple` and the Gaussian model; schema `"3"` adds the isotonic model, the two bootstrap
740
+ interval methods, `Fit.cell_estimates` and `Threshold.bootstrap`. Reading is strict: a card with fields this version does not know is rejected, not silently
514
741
  trimmed. To read a stored card, use a robustmargin at least as new as the
515
742
  one that wrote it. The card's `library_version` field records which version that was.
516
743
 
@@ -532,6 +759,18 @@ on Python 3.11 against the oldest supported versions, pinned in
532
759
  [`ci/constraints-min.txt`](https://github.com/robustmargin/robustmargin/blob/main/ci/constraints-min.txt).
533
760
  Changes are recorded in [`CHANGELOG.md`](https://github.com/robustmargin/robustmargin/blob/main/CHANGELOG.md).
534
761
 
762
+ ## Citing
763
+
764
+ robustmargin is written and maintained by Babak Badkoubeh
765
+ ([ORCID profile](https://orcid.org/0009-0005-3586-7901)). If you use it in published work, please
766
+ cite it. Each release is archived on Zenodo
767
+ ([doi:10.5281/zenodo.23585829](https://doi.org/10.5281/zenodo.23585829)), and the citation
768
+ metadata is in
769
+ [`CITATION.cff`](https://github.com/robustmargin/robustmargin/blob/main/CITATION.cff); GitHub's
770
+ "Cite this repository" button renders it as BibTeX or APA.
771
+
535
772
  ## License
536
773
 
537
- Apache-2.0. See [`LICENSE`](https://github.com/robustmargin/robustmargin/blob/main/LICENSE).
774
+ Apache-2.0, copyright 2026 Babak Badkoubeh. See
775
+ [`LICENSE`](https://github.com/robustmargin/robustmargin/blob/main/LICENSE) and
776
+ [`NOTICE`](https://github.com/robustmargin/robustmargin/blob/main/NOTICE).
@@ -21,12 +21,12 @@ robustness and safety carry weight:
21
21
  - **Autonomous and safety-critical systems**: margins reported with honest uncertainty, and
22
22
  censored bounds instead of invented numbers when the data cannot support an estimate.
23
23
 
24
- > **Status: latest release `0.3.0`.** Dose-response fits for pass/fail outcomes that fall or rise
25
- > with severity and for real-valued scores (Gaussian least squares), thresholds with
26
- > profile-likelihood or delta confidence intervals, censoring, ratios of two thresholds with
27
- > Fieller or log-delta intervals, fit diagnostics, and the grid break-point rule. Clustered and
28
- > paired inference, model-free fits and design helpers are planned for `0.4`. While the version
29
- > is below 1.0, a minor release may change the public API.
24
+ > **Status: latest release `0.4.0`.** Dose-response fits for pass/fail outcomes that fall or rise
25
+ > with severity and for real-valued scores (Gaussian least squares), model-free isotonic fits,
26
+ > thresholds with profile-likelihood, delta or bootstrap confidence intervals, censoring, ratios
27
+ > of two thresholds with Fieller or log-delta intervals, clustered and paired inference for
28
+ > outcomes that share scenarios, simulation-based design planning, fit diagnostics, and the grid
29
+ > break-point rule. While the version is below 1.0, a minor release may change the public API.
30
30
 
31
31
  ## What it does
32
32
 
@@ -63,7 +63,10 @@ its results into robustmargin's input:
63
63
  Because the interface is only severities and outcomes, the same adapter pattern works for any
64
64
  evaluator. A typical use is comparing checkpoints, for example before and after RL
65
65
  post-training: `ratio_interval` gives a confidence interval for the ratio of their margins, so
66
- "the new checkpoint is more robust" becomes a claim with an interval attached.
66
+ "the new checkpoint is more robust" becomes a claim with an interval attached. Before the sweep,
67
+ `robustmargin.design` simulates candidate designs from a pilot fit, so the number of runs per
68
+ level (and of shared scenarios, for a paired comparison) is chosen for the interval width you
69
+ need.
67
70
 
68
71
  ## Install
69
72
 
@@ -271,8 +274,8 @@ assert any(w.startswith("OVERDISPERSED: ") for w in form_b.diagnostics.warnings)
271
274
  assert any(w.startswith("NON_MONOTONE_DATA: ") for w in ratio.warnings)
272
275
  ```
273
276
 
274
- Treat an overdispersed fit's intervals as too narrow. Clustered inference, which addresses this,
275
- is planned for `0.4`.
277
+ Treat an overdispersed fit's intervals as too narrow. When the extra variation comes from trials
278
+ that share a cluster, see "Repeated scenarios: clustered and paired inference" below.
276
279
 
277
280
  ### An outcome that rises with severity
278
281
 
@@ -389,6 +392,226 @@ assert abs(fit.predict(margin.value).estimate - 0.7) < 1e-6
389
392
  A ratio of two continuous thresholds uses a Student-t quantile on Welch-Satterthwaite degrees of
390
393
  freedom, and `ratio_interval` raises `ValueError` for a count threshold against a continuous one.
391
394
 
395
+ ### A model-free fit and bootstrap intervals
396
+
397
+ When you would rather assume only that performance falls (or rises) with severity, fit
398
+ `model="isotonic"`: it takes counts or a continuous response, needs no `link`, `upper` or `lower`,
399
+ and returns the monotone least-squares value of every tested level in `Fit.cell_estimates` (found
400
+ by pool-adjacent-violators). There is no curve between tested levels, so `Fit.predict` and
401
+ `diagnose` raise. A threshold on an isotonic fit is where the straight line drawn through those
402
+ values (on `log` severity for a log axis) first reaches the target. Nothing is extrapolated: a
403
+ target the line never reaches inside the tested levels is `UNREACHABLE` with no number. For counts
404
+ the exact bounds (`RIGHT`, `LEFT`, `FAILS_AT_BASELINE`) still come first. `relative` raises,
405
+ because there are no asymptotes, and `baseline_fraction` and `baseline_multiple` use the isotonic
406
+ value at the zero-severity control as the baseline.
407
+
408
+ An isotonic fit has no covariance, so its interval is a bootstrap: `interval_method="cell_bootstrap"`
409
+ resamples each tested level from its own data, refits, and reports percentiles of the resampled
410
+ thresholds. `n_resamples` defaults to 999 and `seed` makes the run reproducible; when you give no
411
+ seed one is drawn and recorded in `Threshold.bootstrap`, so any result can be rerun. A resampled
412
+ curve that never reaches the target counts toward an open side of the interval, so the interval
413
+ can be one-sided, or carry no bounds when the data cannot say either way.
414
+
415
+ ```python
416
+ from robustmargin import Axis, Definition, Observations, Status, fit_dose_response, threshold
417
+
418
+ obs = Observations.from_counts(
419
+ Axis(name="severity", unit="unit", scale="log"),
420
+ severity=[0.0, 0.5, 1.0, 2.0, 4.0, 8.0],
421
+ successes=[100, 98, 90, 60, 20, 5],
422
+ trials=[100] * 6,
423
+ outcome="success",
424
+ direction="decreasing",
425
+ )
426
+ fit = fit_dose_response(obs, model="isotonic")
427
+ assert fit.cell_estimates == (1.0, 0.98, 0.9, 0.6, 0.2, 0.05) # already monotone
428
+
429
+ # The severity at which success falls to 50%, with a 95% bootstrap interval.
430
+ margin = threshold(
431
+ fit,
432
+ definition=Definition.absolute(0.5),
433
+ interval_method="cell_bootstrap",
434
+ level=0.95,
435
+ n_resamples=499,
436
+ seed=7,
437
+ )
438
+ assert margin.status is Status.OK
439
+ assert 2.0 < margin.value < 4.0
440
+ assert margin.lo <= margin.hi
441
+ assert margin.bootstrap.seed == 7 and margin.bootstrap.failed == 0
442
+ ```
443
+
444
+ `interval_method="parametric_bootstrap"` does the same for a binomial or Gaussian fit: it simulates
445
+ new data from the fitted curve at the tested levels and refits the same model. On a fit with an
446
+ estimated asymptote each refit is far slower than with fixed asymptotes, so give it a smaller
447
+ `n_resamples`. A replicate whose refit fails is counted in `Threshold.bootstrap` and enters the
448
+ percentiles as the worst case for each end (so failures can only widen the interval), with a warning
449
+ above 1% and no interval above 10% (or above `alpha/2`, if that is smaller).
450
+
451
+ ### Repeated scenarios: clustered and paired inference
452
+
453
+ When many trials come from the same scenario, they are not independent, and intervals that assume
454
+ they are come out too narrow. Give `Observations` a `cluster` id for every row and ask for
455
+ `dependence="clustered"`. The point estimate is the usual maximum-likelihood fit, unchanged; its
456
+ covariance becomes a cluster-robust sandwich that uses only the variation between clusters, and
457
+ `Fit.cluster_cells` keeps the per-cluster counts so that whole clusters can be resampled. This is
458
+ for count outcomes (`model="binomial"`) only. A clustered fit needs more clusters than it has
459
+ estimated parameters (`ValueError` otherwise), and a result built on few clusters carries a
460
+ `FEW_CLUSTERS:` warning stating the count it considers few. Monotonicity is checked by an exact sign
461
+ test across clusters (`check_monotonicity(obs, dependence="clustered")`).
462
+
463
+ A clustered threshold makes no exact claim, because the exact bounds treat trials as independent.
464
+ Its recommended interval is `"delta"`: the sandwich covariance with a Student-t quantile on
465
+ `G - 1` degrees of freedom. `"cluster_bootstrap"` (resample whole clusters with replacement and
466
+ refit) is an alternative that needs about 40 or more clusters, because a percentile interval from
467
+ few clusters covers less than its stated level; the other methods raise. A beyond-range crossing is
468
+ an `EXTRAPOLATED` value, not a `RIGHT` bound. When two outcomes are measured on the same clusters,
469
+ the ratio of their thresholds takes `dependence="paired"`, with `method="log_delta"` (or
470
+ `"fieller"`), which keeps the correlation between the two. The example below uses 12 clusters, so
471
+ its results carry the `FEW_CLUSTERS:` warning, which is raised below 20 clusters (below 40 for
472
+ `"cluster_bootstrap"`).
473
+
474
+ ```python
475
+ import math
476
+
477
+ import numpy as np
478
+
479
+ from robustmargin import (
480
+ Axis,
481
+ Definition,
482
+ Observations,
483
+ Status,
484
+ fit_dose_response,
485
+ ratio_interval,
486
+ threshold,
487
+ )
488
+
489
+ axis = Axis(name="severity", unit="unit", scale="log")
490
+ levels = [0.0, 1.0, 2.0, 4.0, 8.0, 16.0, 32.0]
491
+ n_clusters, trials = 12, 20
492
+ rng = np.random.default_rng(3)
493
+ scenario_shift = rng.normal(0.0, 0.4, n_clusters) # each scenario is easier or harder
494
+
495
+
496
+ def observe(log_midpoint_shift: float) -> Observations:
497
+ severity, successes, cluster = [], [], []
498
+ for g in range(n_clusters):
499
+ for x in levels:
500
+ z = (math.log(x) - (math.log(6.0) + log_midpoint_shift)) / 0.6 if x > 0 else -4.0
501
+ p = 1.0 - 0.5 * (1.0 + math.erf((z + scenario_shift[g]) / math.sqrt(2.0)))
502
+ severity.append(x)
503
+ successes.append(int(rng.binomial(trials, p)))
504
+ cluster.append(f"scenario-{g}")
505
+ return Observations.from_counts(
506
+ axis,
507
+ severity=severity,
508
+ successes=successes,
509
+ trials=[trials] * len(severity),
510
+ outcome="success",
511
+ direction="decreasing",
512
+ cluster=cluster,
513
+ )
514
+
515
+
516
+ margins = {}
517
+ for name, shift in {"a": 0.0, "b": 0.5}.items():
518
+ fit = fit_dose_response(
519
+ observe(shift),
520
+ model="binomial",
521
+ link="probit",
522
+ upper=1.0,
523
+ lower=0.0,
524
+ dependence="clustered",
525
+ )
526
+ assert fit.status is Status.OK and fit.dependence == "clustered"
527
+ margins[name] = threshold(
528
+ fit,
529
+ definition=Definition.absolute(0.5),
530
+ interval_method="delta",
531
+ dependence="clustered",
532
+ level=0.95,
533
+ )
534
+ assert margins[name].status is Status.OK
535
+ assert margins[name].lo < margins[name].value < margins[name].hi
536
+
537
+ # Outcome "b" is measured on the same twelve scenarios as "a": a paired ratio of the two margins.
538
+ ratio = ratio_interval(
539
+ margins["b"],
540
+ margins["a"],
541
+ method="log_delta",
542
+ dependence="paired",
543
+ level=0.95,
544
+ )
545
+ assert ratio.status is Status.OK and ratio.dependence == "paired"
546
+ assert 1.0 < ratio.lo < ratio.estimate < ratio.hi
547
+ ```
548
+
549
+ `method="fieller"` and `"log_delta"` with `dependence="paired"` use the covariance between the two
550
+ fits' cluster-robust estimates and Student-t quantiles on `G - 1` degrees of freedom. Both
551
+ thresholds must come from clustered fits over the same set of cluster ids. `method="cluster_bootstrap"`
552
+ with `dependence="paired"` resamples the same clusters for both and is the alternative; it also
553
+ computes the sandwich standard error alongside, and a `CLUSTER_CROSS_CHECK:` warning says when the
554
+ two disagree by more than a stated tolerance. A ratio of clustered thresholds on disjoint sets of
555
+ clusters (`dependence="independent"`) uses a Student-t quantile with Welch-Satterthwaite degrees of
556
+ freedom.
557
+
558
+ ### Planning a design
559
+
560
+ Before running a sweep, `robustmargin.design` estimates how wide the interval will be. Give it a
561
+ pilot `Fit` (say, from a coarse sweep) and a candidate design: it treats the pilot's point
562
+ estimates as the truth, simulates whole experiments at that design, refits the same model on each,
563
+ and calls the real `threshold` with your interval method. `severity_grid` builds the levels. Each
564
+ replicate gets its own random stream from `seed`, so the result is the same for any `workers`. With
565
+ `n_clusters` and `icc` the trials come in clusters, with the pilot curve as every level's marginal
566
+ success rate; `icc` is the latent correlation of the underlying normal variables, smaller on the
567
+ binary outcomes (a latent 0.3 is about 0.19 at a success rate of 0.5). `plan_ratio_width` does the
568
+ same for a ratio, independent or paired. A bootstrap interval method takes `n_resamples`, and costs
569
+ about 13 s per replicate at the default 999 resamples.
570
+
571
+ ```python
572
+ import numpy as np
573
+ from scipy.stats import norm
574
+
575
+ from robustmargin import Axis, Definition, Observations, fit_dose_response
576
+ from robustmargin.design import plan_threshold_width, severity_grid
577
+
578
+ axis = Axis(name="severity", unit="unit", scale="log")
579
+ pilot_levels = severity_grid(axis, lowest=0.1, highest=12.8, n_levels=8, control=False)
580
+ rate = 1.0 - norm.cdf(np.log(pilot_levels) / 0.5)
581
+ pilot = fit_dose_response(
582
+ Observations.from_counts(
583
+ axis,
584
+ severity=pilot_levels,
585
+ successes=[round(100 * p) for p in rate],
586
+ trials=[100] * len(pilot_levels),
587
+ outcome="success",
588
+ direction="decreasing",
589
+ ),
590
+ model="binomial",
591
+ link="probit",
592
+ upper=1.0,
593
+ lower=0.0,
594
+ )
595
+
596
+ widths = plan_threshold_width(
597
+ pilot,
598
+ severity=severity_grid(axis, lowest=0.25, highest=4.0, n_levels=5, control=False),
599
+ trials_per_level=100,
600
+ definition=Definition.absolute(0.5),
601
+ interval_method="delta",
602
+ level=0.95,
603
+ n_replicates=20,
604
+ seed=1,
605
+ )
606
+ share, se = widths.share_at_most(1.0) # share of experiments with log(hi / lo) <= 1
607
+ print(widths.quantile(0.5), share, se, widths.outcome_counts())
608
+ ```
609
+
610
+ A replicate with no finite width (a censored threshold, a failed fit) is counted as an infinite
611
+ width and tallied by reason in `widths.outcome_counts()`, never dropped. The reasons are `"OK"`, a
612
+ `Status`, `Censoring` or `IntervalShape` value, `"BOTH_SIDES_OPEN"` and `"LOWER_AT_ZERO"`. A `WidthPlan` serialises with
613
+ `robustmargin.report.to_dict` as its own document, separate from the score-card schema.
614
+
392
615
  ## Grid break-point rule
393
616
 
394
617
  `grid_break_point` finds the smallest tested severity at which observed performance falls
@@ -470,10 +693,11 @@ assert from_dict(json.loads(text)) == card
470
693
  assert load_schema()["$schema"] == "https://json-schema.org/draft/2020-12/schema"
471
694
  ```
472
695
 
473
- The JSON follows `schema/scorecard-v2.json`, a JSON Schema shipped inside the package, and
474
- `schema/scorecard-v1.json` is kept so that cards written by `0.1.1` and `0.2.0` still read.
475
- Schema `"2"` adds increasing outcomes and `baseline_multiple`. It covers binary outcomes only,
476
- and reading is strict: a card with fields this version does not know is rejected, not silently
696
+ The JSON follows `schema/scorecard-v3.json`, a JSON Schema shipped inside the package, and
697
+ `schema/scorecard-v1.json` and `schema/scorecard-v2.json` are kept so that cards written by
698
+ earlier releases still read, and are written back as `"3"`. Schema `"2"` added increasing outcomes,
699
+ `baseline_multiple` and the Gaussian model; schema `"3"` adds the isotonic model, the two bootstrap
700
+ interval methods, `Fit.cell_estimates` and `Threshold.bootstrap`. Reading is strict: a card with fields this version does not know is rejected, not silently
477
701
  trimmed. To read a stored card, use a robustmargin at least as new as the
478
702
  one that wrote it. The card's `library_version` field records which version that was.
479
703
 
@@ -495,6 +719,18 @@ on Python 3.11 against the oldest supported versions, pinned in
495
719
  [`ci/constraints-min.txt`](https://github.com/robustmargin/robustmargin/blob/main/ci/constraints-min.txt).
496
720
  Changes are recorded in [`CHANGELOG.md`](https://github.com/robustmargin/robustmargin/blob/main/CHANGELOG.md).
497
721
 
722
+ ## Citing
723
+
724
+ robustmargin is written and maintained by Babak Badkoubeh
725
+ ([ORCID profile](https://orcid.org/0009-0005-3586-7901)). If you use it in published work, please
726
+ cite it. Each release is archived on Zenodo
727
+ ([doi:10.5281/zenodo.23585829](https://doi.org/10.5281/zenodo.23585829)), and the citation
728
+ metadata is in
729
+ [`CITATION.cff`](https://github.com/robustmargin/robustmargin/blob/main/CITATION.cff); GitHub's
730
+ "Cite this repository" button renders it as BibTeX or APA.
731
+
498
732
  ## License
499
733
 
500
- Apache-2.0. See [`LICENSE`](https://github.com/robustmargin/robustmargin/blob/main/LICENSE).
734
+ Apache-2.0, copyright 2026 Babak Badkoubeh. See
735
+ [`LICENSE`](https://github.com/robustmargin/robustmargin/blob/main/LICENSE) and
736
+ [`NOTICE`](https://github.com/robustmargin/robustmargin/blob/main/NOTICE).