reliability-lab 0.1.5__tar.gz → 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (86) hide show
  1. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/.gitignore +2 -0
  2. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/CHANGELOG.md +14 -1
  3. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/PKG-INFO +33 -3
  4. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/README.md +32 -2
  5. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/docs/design/roadmap.md +38 -2
  6. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/docs/guidance/bootstrap-coverage.md +63 -3
  7. reliability_lab-0.2.0/docs/guidance/figures/power_by_n-dark.png +0 -0
  8. reliability_lab-0.2.0/docs/guidance/figures/power_by_n-light.png +0 -0
  9. reliability_lab-0.2.0/simulations/paired.py +272 -0
  10. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/simulations/report.py +161 -9
  11. reliability_lab-0.2.0/simulations/results/paired/coverage.csv +73 -0
  12. reliability_lab-0.2.0/simulations/results/paired/run_info.json +22 -0
  13. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/simulations/run_coverage.py +11 -3
  14. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/simulations/templates/bootstrap-coverage.md +40 -3
  15. reliability_lab-0.2.0/src/reliability_lab/_version.py +1 -0
  16. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/src/reliability_lab/inference.py +296 -45
  17. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/src/reliability_lab/report.py +48 -8
  18. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/src/reliability_lab/study.py +72 -1
  19. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_bca.py +111 -0
  20. reliability_lab-0.2.0/tests/test_noninferiority.py +227 -0
  21. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_report_io.py +2 -1
  22. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_simulations.py +69 -1
  23. reliability_lab-0.2.0/tests/test_triage.py +94 -0
  24. reliability_lab-0.1.5/src/reliability_lab/_version.py +0 -1
  25. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/.gitattributes +0 -0
  26. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/.github/workflows/ci.yml +0 -0
  27. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/.github/workflows/release.yml +0 -0
  28. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/.python-version +0 -0
  29. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/LICENSE +0 -0
  30. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/docs/design/framing-v2.md +0 -0
  31. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/docs/guidance/figures/coverage_by_factor-dark.png +0 -0
  32. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/docs/guidance/figures/coverage_by_factor-light.png +0 -0
  33. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/docs/guidance/figures/coverage_by_n-dark.png +0 -0
  34. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/docs/guidance/figures/coverage_by_n-light.png +0 -0
  35. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/docs/guidance/figures/coverage_by_spread-dark.png +0 -0
  36. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/docs/guidance/figures/coverage_by_spread-light.png +0 -0
  37. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/docs/releasing.md +0 -0
  38. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/examples/llm_judge.py +0 -0
  39. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/examples/quickstart.py +0 -0
  40. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/examples/simulated.py +0 -0
  41. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/examples/splits.py +0 -0
  42. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/pyproject.toml +0 -0
  43. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/simulations/__init__.py +0 -0
  44. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/simulations/coders.py +0 -0
  45. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/simulations/coverage.py +0 -0
  46. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/simulations/grid.py +0 -0
  47. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/simulations/results/coders/coverage.csv +0 -0
  48. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/simulations/results/coders/run_info.json +0 -0
  49. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/simulations/results/coverage.csv +0 -0
  50. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/simulations/results/run_info.json +0 -0
  51. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/src/reliability_lab/__init__.py +0 -0
  52. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/src/reliability_lab/alpha.py +0 -0
  53. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/src/reliability_lab/data.py +0 -0
  54. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/src/reliability_lab/design.py +0 -0
  55. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/src/reliability_lab/exceptions.py +0 -0
  56. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/src/reliability_lab/llm.py +0 -0
  57. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/src/reliability_lab/py.typed +0 -0
  58. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/src/reliability_lab/simulate.py +0 -0
  59. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/src/reliability_lab/split.py +0 -0
  60. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/fixtures/__init__.py +0 -0
  61. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/fixtures/krippendorff2011.py +0 -0
  62. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/fixtures/llm_logs.py +0 -0
  63. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/helpers.py +0 -0
  64. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/naive_bootstrap.py +0 -0
  65. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/reference.py +0 -0
  66. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_alpha.py +0 -0
  67. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_circular_bipolar.py +0 -0
  68. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_coder_jackknife.py +0 -0
  69. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_coder_resampling.py +0 -0
  70. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_compare.py +0 -0
  71. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_data.py +0 -0
  72. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_design.py +0 -0
  73. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_inference_units.py +0 -0
  74. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_kalpha.py +0 -0
  75. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_krippendorff2011.py +0 -0
  76. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_llm_end_to_end.py +0 -0
  77. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_parity.py +0 -0
  78. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_properties.py +0 -0
  79. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_report.py +0 -0
  80. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_retries.py +0 -0
  81. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_simulate.py +0 -0
  82. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_smoke.py +0 -0
  83. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_split.py +0 -0
  84. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_status_policies.py +0 -0
  85. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/tests/test_trace.py +0 -0
  86. {reliability_lab-0.1.5 → reliability_lab-0.2.0}/uv.lock +0 -0
@@ -26,6 +26,8 @@ simulations/results/replications.parquet
26
26
  simulations/results/quick/
27
27
  simulations/results/coders/replications.parquet
28
28
  simulations/results/coders/quick/
29
+ simulations/results/paired/replications.parquet
30
+ simulations/results/paired/quick/
29
31
 
30
32
  # Editors / OS
31
33
  .vscode/
@@ -1,6 +1,19 @@
1
1
  # Changelog
2
2
 
3
- ## Unreleased
3
+ ## 0.2.0
4
+
5
+ Decisions on a configuration change.
6
+
7
+ - A `category_skew` diagnostic on every result: the normalised entropy H / ln K of the pairable values over the K declared values, the effective number of values e^H, and the largest value and its share. It means the same under every metric. `CategorySkewWarning` keeps its 90% rule.
8
+ - `estimate.per_unit` gains each unit's `abstained`, `inapplicable` and `failed` (parse and provider errors) counts, from the ratings that count after retries. Sorted by `d_o_contribution`, it is the adjudication list.
9
+ - Non-inferiority and equivalence tests in `ReliabilityStudy.compare()`: pass `margin` with `test="non_inferiority"` or `test="equivalence"`, with `a` as the reference.
10
+ - `ComparisonResult.verdicts` gives a decision for every missing-data policy: `non_inferior`, `inferior`, `equivalent`, `not_equivalent` or `inconclusive`. It reads the interval the comparison already reports, so at the default 95% non-inferiority is a one-sided test at 2.5%, and a 5% equivalence test uses `confidence=0.90`. An undefined interval is `inconclusive`.
11
+ - `ComparisonResult.p_values` gives the p-value that inverts the same interval, percentile or BCa, so the test rejects exactly when p < (1 − confidence) / 2.
12
+ - The test and margin appear in the summary, the `differences` table (`verdict`, `p_value`), the JSON (`test`, `margin`, `verdicts`, `p_values`) and the provenance declarations.
13
+ - A paired simulation grid (`python -m simulations.run_coverage --grid paired`) measures the test, in a new [Comparing two configurations](docs/guidance/bootstrap-coverage.md#comparing-two-configurations) section of the guidance. With the true difference on the margin, the 2.5% test at the default 95% rejected 2.9% of the time on average, and 3.75% in the worst scenario. At `confidence=0.96` it rejected 2.4%, and at most 3.15%, so the guidance recommends 0.96 when the test must hold 2.5%. Power curves at margins 0.02, 0.05 and 0.1 show how many units a margin needs.
14
+ - An undefined BCa interval now names its cause in `undefined_reason`: undefined replicates, every replicate on one side of the estimate (infinite bias correction), alpha undefined with a cluster left out, or a constant jackknife (undefined acceleration).
15
+ - `on_undefined="percentile"` in `AlphaResult.bootstrap()` and `ReliabilityStudy.compare()` reports the percentile interval over the same resamples in place of an undefined BCa interval. The interval records `method_used="percentile"` and the BCa failure in `fallback_reason`, and a `BootstrapWarning` says so. The default, `"nan"`, still leaves no interval.
16
+ - Every interval records `method_used`: `"percentile"`, `"bca"`, or `"t"` for the coder jackknife. The new fields appear in tables and JSON reports (`diff_method_used` and `diff_fallback_reason` in comparisons).
4
17
 
5
18
  ## 0.1.5
6
19
 
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: reliability-lab
3
- Version: 0.1.5
3
+ Version: 0.2.0
4
4
  Summary: Study-design library for the reliability of LLM and human judgments, built on Krippendorff's alpha.
5
5
  Project-URL: Repository, https://github.com/Sathapana/reliability-lab
6
6
  Project-URL: Issues, https://github.com/Sathapana/reliability-lab/issues
@@ -31,7 +31,7 @@ Description-Content-Type: text/markdown
31
31
 
32
32
  # reliability-lab
33
33
 
34
- **Status: alpha (0.1.5).** The API may still change between releases.
34
+ **Status: alpha (0.2.0).** The API may still change between releases.
35
35
 
36
36
  reliability-lab is a study-design library for measuring the reliability of LLM and human judgments. It is built on Krippendorff's alpha. Alpha implementations already exist in Python. This package adds the workflow around the coefficient:
37
37
 
@@ -84,6 +84,15 @@ result = study.alpha()
84
84
  print(result.summary())
85
85
  ```
86
86
 
87
+ Each estimate lists its units with their share of the observed disagreement and their abstained, inapplicable and failed ratings. Sorted, that is the list of units to adjudicate first:
88
+
89
+ ```python
90
+ per_unit = result["as_missing"].per_unit
91
+ per_unit.sort_values("d_o_contribution", ascending=False).head(10)
92
+ ```
93
+
94
+ The `category_skew` diagnostic shows how evenly the ratings use the declared scale: the normalised entropy (1 when every value is used equally, 0 when one value holds them all) and the effective number of values. Alpha can be low while observed agreement is high when one value dominates.
95
+
87
96
  ## Simulated ratings
88
97
 
89
98
  `simulate_ratings` makes synthetic coders for any measurement level, including `"bipolar"`, `circular(U)` and custom distances, at the alpha you choose. Use it to try the API, to see what a given alpha looks like in raw ratings, or to plan how many units a study needs.
@@ -115,7 +124,7 @@ boot = result.bootstrap(scheme="units", n_resamples=2000, random_state=7)
115
124
  print(boot.summary()) # 95% percentile intervals for every missing-data policy
116
125
  ```
117
126
 
118
- - `scheme="units"` resamples whole units, and whole groups when a `group` column is declared. It can also resample within declared strata (`strata=True`), and it recomputes the expected disagreement in every replicate. `method="bca"` gave the most accurate coverage in the simulations; on small samples dominated by one category it can fail, and that is warned about.
127
+ - `scheme="units"` resamples whole units, and whole groups when a `group` column is declared. It can also resample within declared strata (`strata=True`), and it recomputes the expected disagreement in every replicate. `method="bca"` gave the most accurate coverage in the simulations; on small samples dominated by one category it can fail. The warning and the interval's `undefined_reason` say why, and `on_undefined="percentile"` reports the percentile interval over the same resamples instead, marked `method_used="percentile"`.
119
128
  - `scheme="kalpha"` reproduces Krippendorff's KALPHA algorithm. It resamples pairs of values and holds the expected disagreement fixed.
120
129
 
121
130
  The units scheme holds the coders fixed: its interval is about these coders on other questions. When the claim is about coders like these, such as 10 runs of an LLM judge standing for the judge or a few annotators standing for trained annotators, the coders are a sample too:
@@ -148,6 +157,27 @@ diff = study.compare(
148
157
  print(diff.summary()) # alpha_b - alpha_a with a paired interval
149
158
  ```
150
159
 
160
+ To decide whether an update changed the judge, declare a margin: the largest drop in alpha that does not matter. `a` is the reference.
161
+
162
+ ```python
163
+ diff = study.compare(
164
+ a={"condition_id": "frontier"}, # reference
165
+ b={"condition_id": "distilled"}, # candidate
166
+ margin=0.03,
167
+ test="non_inferiority", # or "equivalence"
168
+ confidence=0.96, # a test that holds 2.5% at the margin
169
+ n_resamples=2000,
170
+ random_state=7,
171
+ )
172
+ diff.verdicts["as_missing"] # "non_inferior", "inferior" or "inconclusive"
173
+ diff.p_values["as_missing"]
174
+ ```
175
+
176
+ - **Non-inferiority** reads the lower end of the interval: `non_inferior` when it is above −margin. At the default 95% this states a one-sided test at 2.5%. In simulation it rejected 2.9% of the time when the true difference sat on the margin, and up to 3.75% in one scenario. `confidence=0.96` held 2.5%. See [Comparing two configurations](https://github.com/Sathapana/reliability-lab/blob/main/docs/guidance/bootstrap-coverage.md#comparing-two-configurations).
177
+ - **Equivalence** reads both ends: `equivalent` when the interval lies inside (−margin, margin). These are two one-sided tests, so a 5% equivalence test uses `confidence=0.90`.
178
+ - Every missing-data policy gets its own verdict, and an undefined interval is `inconclusive`. Each p-value inverts the interval its verdict reads, so the two never disagree.
179
+ - Fix the margin before looking at the results. Small margins need many units: at a margin of 0.05, 500 units gave 31–67% power when the two configurations were equally reliable.
180
+
151
181
  ## Splits and reports
152
182
 
153
183
  Teams tune the judge (prompt, few-shot examples, rubric) and should report reliability on units the tuning never saw. So split before tuning, freeze the judge, then estimate on the test partition.
@@ -1,6 +1,6 @@
1
1
  # reliability-lab
2
2
 
3
- **Status: alpha (0.1.5).** The API may still change between releases.
3
+ **Status: alpha (0.2.0).** The API may still change between releases.
4
4
 
5
5
  reliability-lab is a study-design library for measuring the reliability of LLM and human judgments. It is built on Krippendorff's alpha. Alpha implementations already exist in Python. This package adds the workflow around the coefficient:
6
6
 
@@ -53,6 +53,15 @@ result = study.alpha()
53
53
  print(result.summary())
54
54
  ```
55
55
 
56
+ Each estimate lists its units with their share of the observed disagreement and their abstained, inapplicable and failed ratings. Sorted, that is the list of units to adjudicate first:
57
+
58
+ ```python
59
+ per_unit = result["as_missing"].per_unit
60
+ per_unit.sort_values("d_o_contribution", ascending=False).head(10)
61
+ ```
62
+
63
+ The `category_skew` diagnostic shows how evenly the ratings use the declared scale: the normalised entropy (1 when every value is used equally, 0 when one value holds them all) and the effective number of values. Alpha can be low while observed agreement is high when one value dominates.
64
+
56
65
  ## Simulated ratings
57
66
 
58
67
  `simulate_ratings` makes synthetic coders for any measurement level, including `"bipolar"`, `circular(U)` and custom distances, at the alpha you choose. Use it to try the API, to see what a given alpha looks like in raw ratings, or to plan how many units a study needs.
@@ -84,7 +93,7 @@ boot = result.bootstrap(scheme="units", n_resamples=2000, random_state=7)
84
93
  print(boot.summary()) # 95% percentile intervals for every missing-data policy
85
94
  ```
86
95
 
87
- - `scheme="units"` resamples whole units, and whole groups when a `group` column is declared. It can also resample within declared strata (`strata=True`), and it recomputes the expected disagreement in every replicate. `method="bca"` gave the most accurate coverage in the simulations; on small samples dominated by one category it can fail, and that is warned about.
96
+ - `scheme="units"` resamples whole units, and whole groups when a `group` column is declared. It can also resample within declared strata (`strata=True`), and it recomputes the expected disagreement in every replicate. `method="bca"` gave the most accurate coverage in the simulations; on small samples dominated by one category it can fail. The warning and the interval's `undefined_reason` say why, and `on_undefined="percentile"` reports the percentile interval over the same resamples instead, marked `method_used="percentile"`.
88
97
  - `scheme="kalpha"` reproduces Krippendorff's KALPHA algorithm. It resamples pairs of values and holds the expected disagreement fixed.
89
98
 
90
99
  The units scheme holds the coders fixed: its interval is about these coders on other questions. When the claim is about coders like these, such as 10 runs of an LLM judge standing for the judge or a few annotators standing for trained annotators, the coders are a sample too:
@@ -117,6 +126,27 @@ diff = study.compare(
117
126
  print(diff.summary()) # alpha_b - alpha_a with a paired interval
118
127
  ```
119
128
 
129
+ To decide whether an update changed the judge, declare a margin: the largest drop in alpha that does not matter. `a` is the reference.
130
+
131
+ ```python
132
+ diff = study.compare(
133
+ a={"condition_id": "frontier"}, # reference
134
+ b={"condition_id": "distilled"}, # candidate
135
+ margin=0.03,
136
+ test="non_inferiority", # or "equivalence"
137
+ confidence=0.96, # a test that holds 2.5% at the margin
138
+ n_resamples=2000,
139
+ random_state=7,
140
+ )
141
+ diff.verdicts["as_missing"] # "non_inferior", "inferior" or "inconclusive"
142
+ diff.p_values["as_missing"]
143
+ ```
144
+
145
+ - **Non-inferiority** reads the lower end of the interval: `non_inferior` when it is above −margin. At the default 95% this states a one-sided test at 2.5%. In simulation it rejected 2.9% of the time when the true difference sat on the margin, and up to 3.75% in one scenario. `confidence=0.96` held 2.5%. See [Comparing two configurations](https://github.com/Sathapana/reliability-lab/blob/main/docs/guidance/bootstrap-coverage.md#comparing-two-configurations).
146
+ - **Equivalence** reads both ends: `equivalent` when the interval lies inside (−margin, margin). These are two one-sided tests, so a 5% equivalence test uses `confidence=0.90`.
147
+ - Every missing-data policy gets its own verdict, and an undefined interval is `inconclusive`. Each p-value inverts the interval its verdict reads, so the two never disagree.
148
+ - Fix the margin before looking at the results. Small margins need many units: at a margin of 0.05, 500 units gave 31–67% power when the two configurations were equally reliable.
149
+
120
150
  ## Splits and reports
121
151
 
122
152
  Teams tune the judge (prompt, few-shot examples, rubric) and should report reliability on units the tuning never saw. So split before tuning, freeze the judge, then estimate on the test partition.
@@ -13,12 +13,13 @@ Two decisions shape the order:
13
13
 
14
14
  | Release | Theme | Contents | Gate |
15
15
  | --- | --- | --- | --- |
16
- | 0.2.0 | Decisions on a configuration change | Non-inferiority and equivalence in `compare()` (2.2); BCa failure reasons and opt-in fallback (1.2); category-skew diagnostic (1.3); unit triage (1.4) | Rejection rate at the margin within Monte Carlo error of the nominal level; a test for each BCa failure cause |
16
+ | 0.2.0 (shipped) | Decisions on a configuration change | Non-inferiority and equivalence in `compare()` (2.2); BCa failure reasons and opt-in fallback (1.2); category-skew diagnostic (1.3); unit triage (1.4) | At `confidence=0.96`, rejection at the margin holds 2.5%; at 0.95 it is measured and reported (revised 30 Oct 2026, see the 0.2.0 gate); a test for each BCa failure cause |
17
17
  | 0.3.0 | Planning, CI gates and Arrow input | Sample-size planning (2.3); CLI quality gates (4.2); Arrow input (4.1) | Planned sizes reach their stated power on fresh seeds; CLI runs the example fixtures end to end; polars, DuckDB and pyarrow inputs give the same ratings fingerprint as pandas |
18
18
  | 0.4.0 | Judge bias evidence | Within- and between-family alpha (3.2); unit covariates for length bias and self-enhancement (3.2); call telemetry, reduced (3.3) | Simulated same-family errors detected; fixtures for each diagnostic |
19
19
  | 0.5.0 | Companion coefficients and plots | `metrics/` with Gwet's AC1 and AC2 (1.3); plotting extra (4.3) | Parity with irrCAC |
20
20
  | 1.0.0 | Human-LLM alignment and a stable release | Mixed panels; alignment averaged over individual annotators and against a consensus label (3.1); stable API; frozen report schema; coverage simulations for interval, ratio and mixed-panel data | Mixed-panel end-to-end fixture with hand-checked values; paired intervals checked against a naive loop; reports from 1.0 read back by later releases; guidance regenerated from full-size runs |
21
21
  | Research | Crossed intervals | A less conservative alternative to `two_way` (2.1) | Beats `two_way` on the coders grid |
22
+ | Research | A calibrated paired interval | A bootstrap-t or double bootstrap for `compare()` | Holds 2.5% at the margin at 95% on the paired grid |
22
23
 
23
24
  Each gate is a test, as in the framing: a release ships when its gate passes.
24
25
 
@@ -51,6 +52,8 @@ Each gate is a test, as in the framing: a release ships when its gate passes.
51
52
 
52
53
  ## 0.2.0: Decisions on a configuration change
53
54
 
55
+ Shipped in 0.2.0: every item below, with its gate passed on 30 Oct 2026.
56
+
54
57
  **Non-inferiority and equivalence (2.2).** `compare(a=reference, b=candidate, margin=δ, test="non_inferiority" | "equivalence")` (decided 29 Oct 2026). The difference stays $\Delta = \alpha_b - \alpha_a$, so `a` is the reference.
55
58
 
56
59
  ```python
@@ -80,7 +83,34 @@ diff.verdicts["as_missing"] # "non_inferior" | "inferior" | "inconclusive"
80
83
  - **Category-skew diagnostic (1.3).** A `category_skew` diagnostic gives the normalised entropy of the pairable values over the declared domain, $H / \ln K$, and the effective number of categories, $e^H$. The existing `CategorySkewWarning` (one value holding 90% or more) keeps its share-based rule. The draft's rule, $D_e < 0.15$ and $1 - D_o > 0.85$, holds for nominal data only. Under the other metrics, $D_o$ and $D_e$ are in the scale's squared units, and $1 - D_o$ is not observed agreement.
81
84
  - **Unit triage (1.4).** `estimate.per_unit` gains each unit's status counts (abstained, inapplicable, failed). Sorted by `d_o_contribution`, it is the adjudication list. No new study methods: coder influence is `coder_jackknife().influence`, and unit disagreement is `estimate.per_unit`.
82
85
 
83
- **Gate.** A simulation grid of paired arms whose true difference sits on the margin. The non-inferiority rejection rate must stay within Monte Carlo error of (1 − `confidence`) / 2, with a power curve at Δ = 0. `tests/test_bca.py` gains a case for each BCa failure cause, and the new interval fields survive a JSON round trip.
86
+ **Gate.** A simulation grid of paired arms whose true difference sits on the margin (`python -m simulations.run_coverage --grid paired`), with a power curve at Δ = 0. `tests/test_bca.py` gains a case for each BCa failure cause, and the new interval fields survive a JSON round trip.
87
+
88
+ The gate as first written required the rejection rate at the margin to stay within Monte Carlo error of (1 − `confidence`) / 2. The first full run (30 Oct 2026) missed it. At 95%, BCa rejected 2.89% on average over the margin cells, against 2.5%, and 3.70% in the worst cell. The investigation found no defect in `compare()`, the verdicts or the p-values:
89
+ - The bootstrap standard error of the difference matched its true spread to within 1% on average.
90
+ - The difference is slightly heavy-tailed: an interval with the true spread still missed 2.62% per tail.
91
+ - Normal intervals, with the bootstrap or the jackknife standard error, rejected 2.74–2.78%, with the same worst cell.
92
+
93
+ The excess is the small-sample undercoverage of bootstrap intervals that the single-alpha guidance also shows. It is about 0.25 points at every level tried.
94
+
95
+ The gate was therefore revised on 30 Oct 2026, before the rerun that is judged against it. The rerun uses a true difference from 2,000,000 units, and BCa intervals at both 95% and 96%:
96
+
97
+ - **At `confidence=0.96`** (one-sided 2%), the BCa test rejects at most 2.72% on average over the margin cells, which is 2.5% plus two pooled Monte Carlo standard errors at 2,000 replications per cell. No margin cell exceeds 3.2%, 2.5% plus two of that cell's standard errors. The guidance recommends this setting when the test must hold 2.5%.
98
+ - **At `confidence=0.95`**, no margin cell exceeds 4.0%. The guidance reports the measured average and worst cell beside the verdicts' stated level.
99
+ - **At both levels**, every verdict agrees with its interval.
100
+
101
+ **Result (30 Oct 2026): the revised gate passes.**
102
+ - At 96%, BCa rejected 2.37% on average over the margin cells, and 3.15% in the worst cell (nominal, n = 500).
103
+ - At 95%, it rejected 2.89%, and 3.75% in the same cell.
104
+ - Every verdict and p-value agreed with its interval.
105
+
106
+ The rerun reused the first run's seed, so its 95% figures repeat the first run's. The 96% arm was therefore judged on the replications that suggested it.
107
+
108
+ A confirmation run of the ten margin cells on a fresh seed also passes. Its results are not committed; it is reproduced by `python -m simulations.run_coverage --grid paired --cells margin --seed 20261030 --out <dir>`.
109
+ - At 96%, BCa rejected 2.29% on average, and 2.70% at worst.
110
+ - At 95%, it rejected 2.88%, and 3.15% at worst.
111
+ - The average excess at 95% repeats, at about 0.4 points. The worst cell does not: nominal at n = 500 rejected 2.85% at 95%, so its 3.75% in the committed run was chance.
112
+
113
+ A calibrated interval (a bootstrap-t, or a double bootstrap that sets the level from the data) would remove the excess at about a hundred times the computation. It is listed under research.
84
114
 
85
115
  ## 0.3.0: Planning, CI gates and Arrow input
86
116
 
@@ -193,6 +223,10 @@ Decided 29 Oct 2026. A hosted API reference was considered and is not required f
193
223
 
194
224
  `scheme="two_way"` is the pigeonhole bootstrap. When coders are alike, it covered 99.3–99.8% and was about 1.5 times as wide as the units interval, because it counts the rating noise twice (Owen 2007). Candidates are Owen's variance correction for the pigeonhole bootstrap and a two-way jackknife. A replacement ships only if, on the coders grid, it covers 93% or more in every scenario and is narrower than `two_way` when τ = 0. There is no release target.
195
225
 
226
+ ## Research: a calibrated paired interval
227
+
228
+ At 95%, the paired intervals of `compare()` miss about 0.25 points too often per tail, so a 2.5% non-inferiority test rejects about 2.9% of the time at the margin (the 0.2.0 gate). Candidates are a bootstrap-t, with a jackknife standard error inside each resample, and a double bootstrap that calibrates the level. Either one ships only if, on the paired grid at 95%, every margin cell rejects within two Monte Carlo standard errors of 2.5%, at a run time usable on studies of a few thousand units. There is no release target.
229
+
196
230
  ## Already shipped
197
231
 
198
232
  | Proposal | Where it is |
@@ -204,6 +238,7 @@ Decided 29 Oct 2026. A hosted API reference was considered and is not required f
204
238
  | `study.unit_disagreement()` (1.4) | `estimate.per_unit`: `pairable_values`, `disagreement`, `d_o_contribution` |
205
239
  | Skew warning (1.3) | `CategorySkewWarning`, `estimate.category_distribution`, `estimate.observed_agreement` |
206
240
  | BCa degeneracy handling (1.2, part) | An undefined BCa interval gives a reason and a `BootstrapWarning` pointing to `method="percentile"`; parity with `scipy.stats.bootstrap` on degenerate data in `tests/test_bca.py` |
241
+ | A ratings generator in the package (2.3, part) | `simulate_ratings` (0.1.5): know-or-guess ratings for any metric at a chosen alpha, with random or near-miss errors. Unlike the generator of `simulations/`, it has no hard units or clusters |
207
242
 
208
243
  ## Decisions
209
244
 
@@ -213,6 +248,7 @@ Decided 29 Oct 2026. A hosted API reference was considered and is not required f
213
248
  - [x] Study file format for the CLI: TOML, read with the standard library (29 Oct 2026).
214
249
  - [x] Arrow input moves from 0.5.0 to 0.3.0, beside the CLI (29 Oct 2026).
215
250
  - [x] 1.0 also requires a stable API, a frozen report schema and coverage simulations for interval, ratio and mixed-panel data; a hosted API reference is not required (29 Oct 2026).
251
+ - [x] The 0.2.0 gate is revised after its first run missed it by about 0.4 points. The test's size at 95% is measured and reported, and `confidence=0.96` must hold 2.5% (30 Oct 2026).
216
252
 
217
253
  ## References
218
254
 
@@ -2,7 +2,7 @@
2
2
 
3
3
  # Bootstrap coverage: which interval to trust
4
4
 
5
- reliability-lab requires you to name a bootstrap scheme; there is no default. This page measures how often each scheme's 95% interval actually contains the true alpha, so you can choose one knowing what it does. The tables, charts and every number quoted in the text come from the committed runs in `simulations/results/` and `simulations/results/coders/`.
5
+ reliability-lab requires you to name a bootstrap scheme; there is no default. This page measures how often each scheme's 95% interval actually contains the true alpha, so you can choose one knowing what it does. It also measures how often the non-inferiority verdict of `compare()` is wrong. The tables, charts and every number quoted in the text come from the committed runs in `simulations/results/`, `simulations/results/coders/` and `simulations/results/paired/`.
6
6
 
7
7
  ## Summary
8
8
 
@@ -17,6 +17,7 @@ reliability-lab requires you to name a bootstrap scheme; there is no default. Th
17
17
  - **KALPHA is too narrow whenever a unit holds more than one pair of values.** It covers 85–86% with three coders on nominal data, 77–80% on a 5-point ordinal scale, and 70–71% with five coders. With two coders, where each unit holds one pair, it covers 92–94%. Use it to reproduce published KALPHA intervals, not for new claims.
18
18
  - **Declare groups when units share a source.** Resampling units of clustered data one by one covers 89–91%. Resampling groups covers 91.8% with 20 clusters and 94.5% with 40.
19
19
  - **No interval repairs informative failures.** When failed attempts concentrate on hard units, dropping them raises alpha by about 0.03. The intervals cover that inflated value correctly, but not the complete-data alpha. Report the missing-data policies side by side, and read their spread as sensitivity to the policy, not as bounds.
20
+ - **For a non-inferiority test that holds 2.5%, use `confidence=0.96`.** At the default 95%, `compare()` wrongly declares non-inferiority in 2.9% of runs at the margin, and up to 3.75% in one scenario. At 96% it does so in 2.4%, and at most 3.15%. Fix the margin first: at a margin of 0.05, even 500 units give 31–67% power when the configurations are equally reliable. See [Comparing two configurations](#comparing-two-configurations).
20
21
 
21
22
  ## Setup
22
23
 
@@ -204,6 +205,62 @@ Each cell gives the coverage of the interval's own claim, with the mean width in
204
205
 
205
206
  **Which coders drive alpha.** `result.coder_jackknife().influence` lists alpha without each coder, and each coder's influence. It shows whether one annotator, model or prompt drives the result. It describes the data; it is not a test of whether coders differ. Choose the claim, and so the interval, from the design, before looking at the results.
206
207
 
208
+ ## Comparing two configurations
209
+
210
+ `study.compare()` resamples the units once and computes both configurations' alphas on every resample. The interval of the difference Δ = α_b − α_a therefore accounts for the units the two share. With a `margin` δ and `test="non_inferiority"`, it also gives a verdict: `non_inferior` when the lower end of the interval is above −δ. Reading one end of a two-sided interval at `confidence` is a one-sided test at level (1 − `confidence`) / 2, so the default 95% states a 2.5% test. This section measures how often the verdict is wrongly `non_inferior`, and how often it is rightly so.
211
+
212
+ **Setup.**
213
+ - **Paired arms.** Two configurations rate the same units, with the same true categories, hard units and clusters, as two judge configurations scoring one item set do. Arm a is the scenario above, with accuracy 0.8.
214
+ - **Margin cells.** Arm b's accuracy is 0.765. The margin is set to −Δ, so the truth sits exactly on the boundary of the null hypothesis, and every `non_inferior` verdict is a false claim. That is 0.049–0.051 on nominal data and 0.024–0.025 on the ordinal scale, where guesses are near misses and alpha falls less.
215
+ - **Equal cells.** Both arms have accuracy 0.8, so Δ = 0. The share of `non_inferior` verdicts at a margin is the test's power at that margin.
216
+ - **Grid.** Nominal and 5-point ordinal data at n = 50, 100, 200 and 500, in both kinds of cell. Two more margin cells at n = 200: clusters of 5 units (groups declared), and failures concentrated on hard units.
217
+ - **Truth.** Δ is the difference of population alphas, from 2,000,000 units rated by both arms, with failures treated as missing.
218
+
219
+ Every replication is compared with percentile and BCa intervals at 95% and 96%, on the same 2,000 resamples.
220
+
221
+ **Run.** 18 scenarios × 2,000 replications × 2,000 bootstrap resamples, seed 20261027, reliability-lab 0.1.5 (numpy 2.5.4, scipy 1.18.1, pandas 3.1.0); 71 minutes on 8 workers. The largest Monte Carlo error of a true difference is 0.0007, and a rejection rate near 2.5% has a Monte Carlo standard error of about 0.35%.
222
+
223
+ | Scenario | Margin (= -Δ) | Percentile 95%: rejects | Percentile 96%: rejects | BCa 95%: rejects | BCa 96%: rejects | BCa 95%: coverage |
224
+ | --- | --- | --- | --- | --- | --- | --- |
225
+ | Nominal, 3 categories, n = 50 | 0.049 | 2.65% | 2.10% | 2.35% | 2.15% | 95.5% |
226
+ | Nominal, 3 categories, n = 100 | 0.050 | 2.80% | 2.30% | 2.85% | 2.25% | 94.8% |
227
+ | Nominal, 3 categories, n = 200 | 0.050 | 2.40% | 1.95% | 2.35% | 2.00% | 94.7% |
228
+ | Nominal, 3 categories, n = 500 | 0.050 | 3.80% | 3.10% | 3.75% | 3.15% | 93.9% |
229
+ | Ordinal 5-point, n = 50 | 0.025 | 2.90% | 2.45% | 3.50% | 2.90% | 94.0% |
230
+ | Ordinal 5-point, n = 100 | 0.025 | 3.00% | 2.60% | 3.05% | 2.80% | 94.0% |
231
+ | Ordinal 5-point, n = 200 | 0.024 | 2.75% | 1.80% | 2.70% | 1.90% | 93.5% |
232
+ | Ordinal 5-point, n = 500 | 0.024 | 2.50% | 2.35% | 2.75% | 2.25% | 95.0% |
233
+ | Clustered units, n = 200 | 0.049 | 2.85% | 2.15% | 2.50% | 2.00% | 94.5% |
234
+ | Failures on hard units, n = 200 | 0.051 | 3.10% | 2.35% | 3.10% | 2.25% | 94.2% |
235
+
236
+ - **At the default 95%, the test is slightly liberal.** Over the ten margin cells, BCa rejects 2.89% of the time against the stated 2.5%, and 3.75% in the worst cell (nominal, n = 500). Percentile intervals do the same: 2.88%, and 3.80% at worst.
237
+ - **The excess is not a defect in the verdict.** The bootstrap standard error of the difference matches its true spread to within 1% on average, and the estimate is unbiased to within 0.08 standard errors in every cell. The difference is slightly heavy-tailed, and the bootstrap interval ends add a little more. The intervals miss about 0.25 points too often per tail, the slight undercoverage the single-alpha intervals also show above. It does not shrink consistently with more units in this grid.
238
+ - **For a test that holds 2.5%, pass `confidence=0.96`.** That states a 2% test. BCa then rejects 2.4% of the time on average, and at most 3.15% in any cell, within Monte Carlo error of 2.5%. The reported interval is then a 96% interval, and the report records the confidence beside the verdict.
239
+ - **Clusters and failures behave alike.** With groups declared, the clustered cell rejects 2.50% at 95% and 2.00% at 96%. With failures on hard units it rejects 3.10% and 2.25%. As for one alpha, the comparison is between the dropped-failure alphas of both arms, not their complete-data alphas.
240
+ - **Verdicts and p-values agree with their intervals** in all 80,000 comparisons of the margin cells.
241
+ - **Equivalence was not simulated.** Two one-sided tests at level γ read both ends of the interval at `confidence` = 1 − 2γ, so a 5% equivalence test uses `confidence=0.90`. Expect the same small per-tail excess, unmeasured at that level.
242
+
243
+ <picture>
244
+ <source media="(prefers-color-scheme: dark)" srcset="figures/power_by_n-dark.png">
245
+ <img alt="Share of non-inferior verdicts from 96% BCa intervals when the two configurations are equally reliable, against the number of units, for margins 0.02, 0.05 and 0.1" src="figures/power_by_n-light.png" width="840">
246
+ </picture>
247
+
248
+ | Scenario | Alpha of each | BCa 96%, margin 0.02 | BCa 96%, margin 0.05 | BCa 96%, margin 0.1 | BCa 95%, margin 0.05 | Width (BCa, 96%) |
249
+ | --- | --- | --- | --- | --- | --- | --- |
250
+ | Nominal, 3 categories, n = 50 | 0.539 | 3.8% | 6.5% | 14.4% | 7.8% | 0.41 |
251
+ | Nominal, 3 categories, n = 100 | 0.539 | 4.2% | 9.2% | 27.3% | 11.1% | 0.29 |
252
+ | Nominal, 3 categories, n = 200 | 0.538 | 5.0% | 13.6% | 47.6% | 15.2% | 0.21 |
253
+ | Nominal, 3 categories, n = 500 | 0.539 | 8.2% | 30.9% | 85.9% | 34.0% | 0.13 |
254
+ | Ordinal 5-point, n = 50 | 0.781 | 5.3% | 12.8% | 33.0% | 14.2% | 0.28 |
255
+ | Ordinal 5-point, n = 100 | 0.781 | 6.5% | 17.8% | 53.8% | 20.6% | 0.19 |
256
+ | Ordinal 5-point, n = 200 | 0.781 | 8.5% | 33.4% | 86.2% | 37.2% | 0.13 |
257
+ | Ordinal 5-point, n = 500 | 0.781 | 15.2% | 67.3% | 99.7% | 70.5% | 0.08 |
258
+
259
+ - **Power depends on the margin far more than on the method.** When the configurations are equally reliable, BCa at 96% declares non-inferiority at a margin of 0.05 in 31% of runs at n = 500 on nominal data, and 67% on the ordinal scale. At a margin of 0.1 that rises to 86% and nearly 100%. A margin of 0.02 is out of reach in this grid: at most 15%.
260
+ - **Ordinal alphas are easier to compare.** They are higher here (0.78 against 0.54) and less variable, so the 96% interval at n = 500 is 0.08 wide against 0.13.
261
+ - **A rough guide.** Power at Δ = 0 reaches about 80% when the margin is about 0.7 times the width of the 96% interval. Run a pilot, read the width, and choose n accordingly.
262
+ - **Fix the margin before looking at the results.** A margin chosen after seeing the interval cannot be wrong, and so says nothing. `compare()` records the margin and the test in the declarations, the provenance and the JSON report.
263
+
207
264
  ## Which scheme to use
208
265
 
209
266
  | Situation | Use | Why |
@@ -214,15 +271,17 @@ Each cell gives the coverage of the interval's own claim, with the mean width in
214
271
  | Fewer than about 30 clusters | group resampling, read as approximate | 92% coverage with 20 clusters |
215
272
  | Reproducing a published KALPHA interval | `scheme="kalpha"` | matches Krippendorff's algorithm; too narrow for new claims with 3 or more coders |
216
273
  | Comparing two configurations | `study.compare()` | units scheme on identical resamples (KALPHA has no paired form) |
274
+ | A non-inferiority verdict on a configuration change | `study.compare(margin=..., test="non_inferiority", confidence=0.96)`, margin fixed in advance | at 95% the 2.5% test rejects 2.9% at the margin; at 96%, 2.4% |
217
275
  | Repeated runs of one judge configuration | `scheme="units"` | runs differ only by chance, so the interval already covers new runs |
218
276
  | Coders differ (annotators, models, prompts), and the claim is about coders like them | `scheme="two_way"`, with 10 coders or more | units covers 48–83% of that claim; `two_way` covers 92–98% |
219
277
  | These questions with other coders, or which coders drive alpha | `result.coder_jackknife()` | 94–96% coverage, and an influence table per coder; the `coders` bootstrap is too narrow |
220
278
 
221
279
  ## Limitations
222
280
 
223
- - **One family of processes.** These scenarios use one family of data-generating processes (know or guess, with difficulty), up to 20 coders, up to 200 units, nominal and 5-point ordinal data, and 95% intervals. Interval and ratio metrics were not simulated.
281
+ - **One family of processes.** These scenarios use one family of data-generating processes (know or guess, with difficulty), up to 20 coders, up to 200 units (500 when comparing), nominal and 5-point ordinal data, and 95% intervals (also 96% when comparing). Interval and ratio metrics were not simulated.
282
+ - **Comparisons differ in one way.** In the comparison grid, the two configurations differ only in accuracy, and the true difference at the margin is about −0.05 (nominal) or −0.025 (ordinal). Equivalence tests were not simulated.
224
283
  - **Coders differ in one way.** In the coder grid, coders differ in accuracy and in a favourite label. Two other kinds of difference were not simulated beyond the one ordinal scenario: coders who use a scale differently, and coders who are good on some kinds of question and poor on others. Every coder rates every question, and no attempts fail.
225
- - **Truth uncertainty.** The true alphas carry a Monte Carlo error of at most 0.0025, small next to interval widths of 0.07 and more.
284
+ - **Truth uncertainty.** The true alphas carry a Monte Carlo error of at most 0.0025, small next to interval widths of 0.07 and more. The true differences carry at most 0.0007.
226
285
  - **BCa's undefined cases are real.** At n = 30 with an 80% category, one run in six produces no BCa interval.
227
286
  - **KALPHA's second pick.** KALPHA's result depends on reading its algorithm note as evalica does: a unit's second pick is redrawn when it repeats the first. Its coverage problem does not come from that detail. It comes from treating dependent pairs as independent.
228
287
 
@@ -231,6 +290,7 @@ Each cell gives the coverage of the interval's own claim, with the mean width in
231
290
  ```bash
232
291
  uv run python -m simulations.run_coverage --workers 8 # about 20 minutes on 8 cores
233
292
  uv run python -m simulations.run_coverage --grid coders --workers 8 # about an hour on 8 cores
293
+ uv run python -m simulations.run_coverage --grid paired --workers 8 # about 70 minutes on 8 cores
234
294
  uv run --group sim python -m simulations.report # tables, figures and this page
235
295
  ```
236
296