reliability-lab 0.1.4__tar.gz → 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (86) hide show
  1. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/.gitignore +2 -0
  2. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/CHANGELOG.md +21 -1
  3. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/PKG-INFO +56 -4
  4. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/README.md +55 -3
  5. reliability_lab-0.2.0/docs/design/roadmap.md +261 -0
  6. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/docs/guidance/bootstrap-coverage.md +63 -3
  7. reliability_lab-0.2.0/docs/guidance/figures/power_by_n-dark.png +0 -0
  8. reliability_lab-0.2.0/docs/guidance/figures/power_by_n-light.png +0 -0
  9. reliability_lab-0.2.0/examples/simulated.py +42 -0
  10. reliability_lab-0.2.0/simulations/paired.py +272 -0
  11. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/simulations/report.py +161 -9
  12. reliability_lab-0.2.0/simulations/results/paired/coverage.csv +73 -0
  13. reliability_lab-0.2.0/simulations/results/paired/run_info.json +22 -0
  14. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/simulations/run_coverage.py +11 -3
  15. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/simulations/templates/bootstrap-coverage.md +40 -3
  16. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/src/reliability_lab/__init__.py +2 -0
  17. reliability_lab-0.2.0/src/reliability_lab/_version.py +1 -0
  18. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/src/reliability_lab/design.py +2 -2
  19. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/src/reliability_lab/inference.py +296 -45
  20. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/src/reliability_lab/report.py +48 -8
  21. reliability_lab-0.2.0/src/reliability_lab/simulate.py +192 -0
  22. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/src/reliability_lab/study.py +72 -1
  23. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_bca.py +111 -0
  24. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_design.py +1 -1
  25. reliability_lab-0.2.0/tests/test_noninferiority.py +227 -0
  26. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_report_io.py +2 -1
  27. reliability_lab-0.2.0/tests/test_simulate.py +172 -0
  28. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_simulations.py +69 -1
  29. reliability_lab-0.2.0/tests/test_triage.py +94 -0
  30. reliability_lab-0.1.4/src/reliability_lab/_version.py +0 -1
  31. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/.gitattributes +0 -0
  32. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/.github/workflows/ci.yml +0 -0
  33. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/.github/workflows/release.yml +0 -0
  34. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/.python-version +0 -0
  35. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/LICENSE +0 -0
  36. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/docs/design/framing-v2.md +0 -0
  37. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/docs/guidance/figures/coverage_by_factor-dark.png +0 -0
  38. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/docs/guidance/figures/coverage_by_factor-light.png +0 -0
  39. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/docs/guidance/figures/coverage_by_n-dark.png +0 -0
  40. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/docs/guidance/figures/coverage_by_n-light.png +0 -0
  41. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/docs/guidance/figures/coverage_by_spread-dark.png +0 -0
  42. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/docs/guidance/figures/coverage_by_spread-light.png +0 -0
  43. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/docs/releasing.md +0 -0
  44. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/examples/llm_judge.py +0 -0
  45. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/examples/quickstart.py +0 -0
  46. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/examples/splits.py +0 -0
  47. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/pyproject.toml +0 -0
  48. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/simulations/__init__.py +0 -0
  49. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/simulations/coders.py +0 -0
  50. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/simulations/coverage.py +0 -0
  51. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/simulations/grid.py +0 -0
  52. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/simulations/results/coders/coverage.csv +0 -0
  53. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/simulations/results/coders/run_info.json +0 -0
  54. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/simulations/results/coverage.csv +0 -0
  55. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/simulations/results/run_info.json +0 -0
  56. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/src/reliability_lab/alpha.py +0 -0
  57. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/src/reliability_lab/data.py +0 -0
  58. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/src/reliability_lab/exceptions.py +0 -0
  59. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/src/reliability_lab/llm.py +0 -0
  60. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/src/reliability_lab/py.typed +0 -0
  61. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/src/reliability_lab/split.py +0 -0
  62. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/fixtures/__init__.py +0 -0
  63. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/fixtures/krippendorff2011.py +0 -0
  64. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/fixtures/llm_logs.py +0 -0
  65. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/helpers.py +0 -0
  66. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/naive_bootstrap.py +0 -0
  67. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/reference.py +0 -0
  68. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_alpha.py +0 -0
  69. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_circular_bipolar.py +0 -0
  70. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_coder_jackknife.py +0 -0
  71. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_coder_resampling.py +0 -0
  72. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_compare.py +0 -0
  73. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_data.py +0 -0
  74. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_inference_units.py +0 -0
  75. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_kalpha.py +0 -0
  76. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_krippendorff2011.py +0 -0
  77. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_llm_end_to_end.py +0 -0
  78. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_parity.py +0 -0
  79. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_properties.py +0 -0
  80. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_report.py +0 -0
  81. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_retries.py +0 -0
  82. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_smoke.py +0 -0
  83. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_split.py +0 -0
  84. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_status_policies.py +0 -0
  85. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/tests/test_trace.py +0 -0
  86. {reliability_lab-0.1.4 → reliability_lab-0.2.0}/uv.lock +0 -0
@@ -26,6 +26,8 @@ simulations/results/replications.parquet
26
26
  simulations/results/quick/
27
27
  simulations/results/coders/replications.parquet
28
28
  simulations/results/coders/quick/
29
+ simulations/results/paired/replications.parquet
30
+ simulations/results/paired/quick/
29
31
 
30
32
  # Editors / OS
31
33
  .vscode/
@@ -1,6 +1,26 @@
1
1
  # Changelog
2
2
 
3
- ## Unreleased
3
+ ## 0.2.0
4
+
5
+ Decisions on a configuration change.
6
+
7
+ - A `category_skew` diagnostic on every result: the normalised entropy H / ln K of the pairable values over the K declared values, the effective number of values e^H, and the largest value and its share. It means the same under every metric. `CategorySkewWarning` keeps its 90% rule.
8
+ - `estimate.per_unit` gains each unit's `abstained`, `inapplicable` and `failed` (parse and provider errors) counts, from the ratings that count after retries. Sorted by `d_o_contribution`, it is the adjudication list.
9
+ - Non-inferiority and equivalence tests in `ReliabilityStudy.compare()`: pass `margin` with `test="non_inferiority"` or `test="equivalence"`, with `a` as the reference.
10
+ - `ComparisonResult.verdicts` gives a decision for every missing-data policy: `non_inferior`, `inferior`, `equivalent`, `not_equivalent` or `inconclusive`. It reads the interval the comparison already reports, so at the default 95% non-inferiority is a one-sided test at 2.5%, and a 5% equivalence test uses `confidence=0.90`. An undefined interval is `inconclusive`.
11
+ - `ComparisonResult.p_values` gives the p-value that inverts the same interval, percentile or BCa, so the test rejects exactly when p < (1 − confidence) / 2.
12
+ - The test and margin appear in the summary, the `differences` table (`verdict`, `p_value`), the JSON (`test`, `margin`, `verdicts`, `p_values`) and the provenance declarations.
13
+ - A paired simulation grid (`python -m simulations.run_coverage --grid paired`) measures the test, in a new [Comparing two configurations](docs/guidance/bootstrap-coverage.md#comparing-two-configurations) section of the guidance. With the true difference on the margin, the 2.5% test at the default 95% rejected 2.9% of the time on average, and 3.75% in the worst scenario. At `confidence=0.96` it rejected 2.4%, and at most 3.15%, so the guidance recommends 0.96 when the test must hold 2.5%. Power curves at margins 0.02, 0.05 and 0.1 show how many units a margin needs.
14
+ - An undefined BCa interval now names its cause in `undefined_reason`: undefined replicates, every replicate on one side of the estimate (infinite bias correction), alpha undefined with a cluster left out, or a constant jackknife (undefined acceleration).
15
+ - `on_undefined="percentile"` in `AlphaResult.bootstrap()` and `ReliabilityStudy.compare()` reports the percentile interval over the same resamples in place of an undefined BCa interval. The interval records `method_used="percentile"` and the BCa failure in `fallback_reason`, and a `BootstrapWarning` says so. The default, `"nan"`, still leaves no interval.
16
+ - Every interval records `method_used`: `"percentile"`, `"bca"`, or `"t"` for the coder jackknife. The new fields appear in tables and JSON reports (`diff_method_used` and `diff_fallback_reason` in comparisons).
17
+
18
+ ## 0.1.5
19
+
20
+ Simulated ratings.
21
+
22
+ - `simulate_ratings` makes synthetic coder ratings for any measurement level (nominal, ordinal, interval, ratio, bipolar, `circular(U)` or a custom `Distance`) at a chosen alpha. Errors are random draws from the population or near misses by the metric's own distance, with optional missing cells and `parse_error` failures. The ratings come as a `RatingFrame`, ready for `ReliabilityStudy`.
23
+ - A [roadmap](docs/design/roadmap.md) for 0.2 to 1.0. Mixed human-LLM panels move from 0.2 to 1.0, and the messages refusing them say so.
4
24
 
5
25
  ## 0.1.4
6
26
 
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: reliability-lab
3
- Version: 0.1.4
3
+ Version: 0.2.0
4
4
  Summary: Study-design library for the reliability of LLM and human judgments, built on Krippendorff's alpha.
5
5
  Project-URL: Repository, https://github.com/Sathapana/reliability-lab
6
6
  Project-URL: Issues, https://github.com/Sathapana/reliability-lab/issues
@@ -31,7 +31,7 @@ Description-Content-Type: text/markdown
31
31
 
32
32
  # reliability-lab
33
33
 
34
- **Status: alpha (0.1.4).** The API may still change between releases.
34
+ **Status: alpha (0.2.0).** The API may still change between releases.
35
35
 
36
36
  reliability-lab is a study-design library for measuring the reliability of LLM and human judgments. It is built on Krippendorff's alpha. Alpha implementations already exist in Python. This package adds the workflow around the coefficient:
37
37
 
@@ -46,7 +46,7 @@ reliability-lab is a study-design library for measuring the reliability of LLM a
46
46
  - versioned JSON and Parquet reports with provenance;
47
47
  - LLM judge workflows: a trace sidecar of judge configurations, declared retry handling, and evidence specific to each design (run agreement, order swaps and position bias).
48
48
 
49
- Mixed human-LLM panels are next on the roadmap ([design document](https://github.com/Sathapana/reliability-lab/blob/main/docs/design/framing-v2.md)).
49
+ Mixed human-LLM panels are planned for 1.0 ([roadmap](https://github.com/Sathapana/reliability-lab/blob/main/docs/design/roadmap.md), [design document](https://github.com/Sathapana/reliability-lab/blob/main/docs/design/framing-v2.md)).
50
50
 
51
51
  > **Reliability is not validity.** Coders can agree consistently and still be wrong in the same way, for example because of a flawed rubric or a bias they share. When results come from different designs (repeatability, inter-model agreement, human panels), they are reported as different quantities and are never pooled.
52
52
 
@@ -84,6 +84,37 @@ result = study.alpha()
84
84
  print(result.summary())
85
85
  ```
86
86
 
87
+ Each estimate lists its units with their share of the observed disagreement and their abstained, inapplicable and failed ratings. Sorted, that is the list of units to adjudicate first:
88
+
89
+ ```python
90
+ per_unit = result["as_missing"].per_unit
91
+ per_unit.sort_values("d_o_contribution", ascending=False).head(10)
92
+ ```
93
+
94
+ The `category_skew` diagnostic shows how evenly the ratings use the declared scale: the normalised entropy (1 when every value is used equally, 0 when one value holds them all) and the effective number of values. Alpha can be low while observed agreement is high when one value dominates.
95
+
96
+ ## Simulated ratings
97
+
98
+ `simulate_ratings` makes synthetic coders for any measurement level, including `"bipolar"`, `circular(U)` and custom distances, at the alpha you choose. Use it to try the API, to see what a given alpha looks like in raw ratings, or to plan how many units a study needs.
99
+
100
+ ```python
101
+ from reliability_lab import circular, simulate_ratings
102
+
103
+ ratings = simulate_ratings(
104
+ circular(24),
105
+ list(range(24)),
106
+ n_units=200,
107
+ n_coders=4,
108
+ alpha=0.7,
109
+ errors="near",
110
+ missing=0.1,
111
+ failures=0.05,
112
+ random_state=1,
113
+ )
114
+ ```
115
+
116
+ Each coder reports the true value or makes an error. With `errors="random"` an error is a draw from the population; with `errors="near"` it lands close to the truth, as the metric defines closeness. Alpha computed from the ratings is the target on average, and varies around it from draw to draw. `missing` leaves cells unrated and `failures` records cells as `parse_error`. [examples/simulated.py](https://github.com/Sathapana/reliability-lab/blob/main/examples/simulated.py) runs every built-in metric.
117
+
87
118
  ## Intervals and comparisons
88
119
 
89
120
  Every interval names its scheme, because the schemes answer different questions. `scheme` is a required argument.
@@ -93,7 +124,7 @@ boot = result.bootstrap(scheme="units", n_resamples=2000, random_state=7)
93
124
  print(boot.summary()) # 95% percentile intervals for every missing-data policy
94
125
  ```
95
126
 
96
- - `scheme="units"` resamples whole units, and whole groups when a `group` column is declared. It can also resample within declared strata (`strata=True`), and it recomputes the expected disagreement in every replicate. `method="bca"` gave the most accurate coverage in the simulations; on small samples dominated by one category it can fail, and that is warned about.
127
+ - `scheme="units"` resamples whole units, and whole groups when a `group` column is declared. It can also resample within declared strata (`strata=True`), and it recomputes the expected disagreement in every replicate. `method="bca"` gave the most accurate coverage in the simulations; on small samples dominated by one category it can fail. The warning and the interval's `undefined_reason` say why, and `on_undefined="percentile"` reports the percentile interval over the same resamples instead, marked `method_used="percentile"`.
97
128
  - `scheme="kalpha"` reproduces Krippendorff's KALPHA algorithm. It resamples pairs of values and holds the expected disagreement fixed.
98
129
 
99
130
  The units scheme holds the coders fixed: its interval is about these coders on other questions. When the claim is about coders like these, such as 10 runs of an LLM judge standing for the judge or a few annotators standing for trained annotators, the coders are a sample too:
@@ -126,6 +157,27 @@ diff = study.compare(
126
157
  print(diff.summary()) # alpha_b - alpha_a with a paired interval
127
158
  ```
128
159
 
160
+ To decide whether an update changed the judge, declare a margin: the largest drop in alpha that does not matter. `a` is the reference.
161
+
162
+ ```python
163
+ diff = study.compare(
164
+ a={"condition_id": "frontier"}, # reference
165
+ b={"condition_id": "distilled"}, # candidate
166
+ margin=0.03,
167
+ test="non_inferiority", # or "equivalence"
168
+ confidence=0.96, # a test that holds 2.5% at the margin
169
+ n_resamples=2000,
170
+ random_state=7,
171
+ )
172
+ diff.verdicts["as_missing"] # "non_inferior", "inferior" or "inconclusive"
173
+ diff.p_values["as_missing"]
174
+ ```
175
+
176
+ - **Non-inferiority** reads the lower end of the interval: `non_inferior` when it is above −margin. At the default 95% this states a one-sided test at 2.5%. In simulation it rejected 2.9% of the time when the true difference sat on the margin, and up to 3.75% in one scenario. `confidence=0.96` held 2.5%. See [Comparing two configurations](https://github.com/Sathapana/reliability-lab/blob/main/docs/guidance/bootstrap-coverage.md#comparing-two-configurations).
177
+ - **Equivalence** reads both ends: `equivalent` when the interval lies inside (−margin, margin). These are two one-sided tests, so a 5% equivalence test uses `confidence=0.90`.
178
+ - Every missing-data policy gets its own verdict, and an undefined interval is `inconclusive`. Each p-value inverts the interval its verdict reads, so the two never disagree.
179
+ - Fix the margin before looking at the results. Small margins need many units: at a margin of 0.05, 500 units gave 31–67% power when the two configurations were equally reliable.
180
+
129
181
  ## Splits and reports
130
182
 
131
183
  Teams tune the judge (prompt, few-shot examples, rubric) and should report reliability on units the tuning never saw. So split before tuning, freeze the judge, then estimate on the test partition.
@@ -1,6 +1,6 @@
1
1
  # reliability-lab
2
2
 
3
- **Status: alpha (0.1.4).** The API may still change between releases.
3
+ **Status: alpha (0.2.0).** The API may still change between releases.
4
4
 
5
5
  reliability-lab is a study-design library for measuring the reliability of LLM and human judgments. It is built on Krippendorff's alpha. Alpha implementations already exist in Python. This package adds the workflow around the coefficient:
6
6
 
@@ -15,7 +15,7 @@ reliability-lab is a study-design library for measuring the reliability of LLM a
15
15
  - versioned JSON and Parquet reports with provenance;
16
16
  - LLM judge workflows: a trace sidecar of judge configurations, declared retry handling, and evidence specific to each design (run agreement, order swaps and position bias).
17
17
 
18
- Mixed human-LLM panels are next on the roadmap ([design document](https://github.com/Sathapana/reliability-lab/blob/main/docs/design/framing-v2.md)).
18
+ Mixed human-LLM panels are planned for 1.0 ([roadmap](https://github.com/Sathapana/reliability-lab/blob/main/docs/design/roadmap.md), [design document](https://github.com/Sathapana/reliability-lab/blob/main/docs/design/framing-v2.md)).
19
19
 
20
20
  > **Reliability is not validity.** Coders can agree consistently and still be wrong in the same way, for example because of a flawed rubric or a bias they share. When results come from different designs (repeatability, inter-model agreement, human panels), they are reported as different quantities and are never pooled.
21
21
 
@@ -53,6 +53,37 @@ result = study.alpha()
53
53
  print(result.summary())
54
54
  ```
55
55
 
56
+ Each estimate lists its units with their share of the observed disagreement and their abstained, inapplicable and failed ratings. Sorted, that is the list of units to adjudicate first:
57
+
58
+ ```python
59
+ per_unit = result["as_missing"].per_unit
60
+ per_unit.sort_values("d_o_contribution", ascending=False).head(10)
61
+ ```
62
+
63
+ The `category_skew` diagnostic shows how evenly the ratings use the declared scale: the normalised entropy (1 when every value is used equally, 0 when one value holds them all) and the effective number of values. Alpha can be low while observed agreement is high when one value dominates.
64
+
65
+ ## Simulated ratings
66
+
67
+ `simulate_ratings` makes synthetic coders for any measurement level, including `"bipolar"`, `circular(U)` and custom distances, at the alpha you choose. Use it to try the API, to see what a given alpha looks like in raw ratings, or to plan how many units a study needs.
68
+
69
+ ```python
70
+ from reliability_lab import circular, simulate_ratings
71
+
72
+ ratings = simulate_ratings(
73
+ circular(24),
74
+ list(range(24)),
75
+ n_units=200,
76
+ n_coders=4,
77
+ alpha=0.7,
78
+ errors="near",
79
+ missing=0.1,
80
+ failures=0.05,
81
+ random_state=1,
82
+ )
83
+ ```
84
+
85
+ Each coder reports the true value or makes an error. With `errors="random"` an error is a draw from the population; with `errors="near"` it lands close to the truth, as the metric defines closeness. Alpha computed from the ratings is the target on average, and varies around it from draw to draw. `missing` leaves cells unrated and `failures` records cells as `parse_error`. [examples/simulated.py](https://github.com/Sathapana/reliability-lab/blob/main/examples/simulated.py) runs every built-in metric.
86
+
56
87
  ## Intervals and comparisons
57
88
 
58
89
  Every interval names its scheme, because the schemes answer different questions. `scheme` is a required argument.
@@ -62,7 +93,7 @@ boot = result.bootstrap(scheme="units", n_resamples=2000, random_state=7)
62
93
  print(boot.summary()) # 95% percentile intervals for every missing-data policy
63
94
  ```
64
95
 
65
- - `scheme="units"` resamples whole units, and whole groups when a `group` column is declared. It can also resample within declared strata (`strata=True`), and it recomputes the expected disagreement in every replicate. `method="bca"` gave the most accurate coverage in the simulations; on small samples dominated by one category it can fail, and that is warned about.
96
+ - `scheme="units"` resamples whole units, and whole groups when a `group` column is declared. It can also resample within declared strata (`strata=True`), and it recomputes the expected disagreement in every replicate. `method="bca"` gave the most accurate coverage in the simulations; on small samples dominated by one category it can fail. The warning and the interval's `undefined_reason` say why, and `on_undefined="percentile"` reports the percentile interval over the same resamples instead, marked `method_used="percentile"`.
66
97
  - `scheme="kalpha"` reproduces Krippendorff's KALPHA algorithm. It resamples pairs of values and holds the expected disagreement fixed.
67
98
 
68
99
  The units scheme holds the coders fixed: its interval is about these coders on other questions. When the claim is about coders like these, such as 10 runs of an LLM judge standing for the judge or a few annotators standing for trained annotators, the coders are a sample too:
@@ -95,6 +126,27 @@ diff = study.compare(
95
126
  print(diff.summary()) # alpha_b - alpha_a with a paired interval
96
127
  ```
97
128
 
129
+ To decide whether an update changed the judge, declare a margin: the largest drop in alpha that does not matter. `a` is the reference.
130
+
131
+ ```python
132
+ diff = study.compare(
133
+ a={"condition_id": "frontier"}, # reference
134
+ b={"condition_id": "distilled"}, # candidate
135
+ margin=0.03,
136
+ test="non_inferiority", # or "equivalence"
137
+ confidence=0.96, # a test that holds 2.5% at the margin
138
+ n_resamples=2000,
139
+ random_state=7,
140
+ )
141
+ diff.verdicts["as_missing"] # "non_inferior", "inferior" or "inconclusive"
142
+ diff.p_values["as_missing"]
143
+ ```
144
+
145
+ - **Non-inferiority** reads the lower end of the interval: `non_inferior` when it is above −margin. At the default 95% this states a one-sided test at 2.5%. In simulation it rejected 2.9% of the time when the true difference sat on the margin, and up to 3.75% in one scenario. `confidence=0.96` held 2.5%. See [Comparing two configurations](https://github.com/Sathapana/reliability-lab/blob/main/docs/guidance/bootstrap-coverage.md#comparing-two-configurations).
146
+ - **Equivalence** reads both ends: `equivalent` when the interval lies inside (−margin, margin). These are two one-sided tests, so a 5% equivalence test uses `confidence=0.90`.
147
+ - Every missing-data policy gets its own verdict, and an undefined interval is `inconclusive`. Each p-value inverts the interval its verdict reads, so the two never disagree.
148
+ - Fix the margin before looking at the results. Small margins need many units: at a margin of 0.05, 500 units gave 31–67% power when the two configurations were equally reliable.
149
+
98
150
  ## Splits and reports
99
151
 
100
152
  Teams tune the judge (prompt, few-shot examples, rubric) and should report reliability on units the tuning never saw. So split before tuning, freeze the judge, then estimate on the test partition.
@@ -0,0 +1,261 @@
1
+ # reliability-lab: Roadmap, 0.2 to 1.0
2
+
3
+ Oct 29, 2026 · @Sathapana
4
+
5
+ The phases of the [framing](framing-v2.md) (P0 to P5) shipped in 0.1.0a1 to 0.1.4. This plan sets the releases from 0.2 to 1.0. It assigns the proposals of the "Development Roadmap (v0.2.0 → v0.5.0)" draft, after checking each one against the code at 0.1.4. Proposal numbers (1.1 to 4.3) refer to that draft.
6
+
7
+ Two decisions shape the order:
8
+
9
+ - **Human-LLM alignment moves to 1.0.** The framing committed mixed panels to v0.2, and the code refused mixed rows with "planned for v0.2". On 29 Oct 2026 they were moved to 1.0, and the refusals now say so. The draft's `study.calibrate()` answers part of the same question, so it is folded into the 1.0 design.
10
+ - **About a third of the draft has shipped.** The vectorised bootstrap, the pigeonhole (two-way) bootstrap, crossed coverage simulations, coder influence and per-unit disagreement already exist. They are listed in [Already shipped](#already-shipped) with where they live, and are not rebuilt.
11
+
12
+ ## Summary
13
+
14
+ | Release | Theme | Contents | Gate |
15
+ | --- | --- | --- | --- |
16
+ | 0.2.0 (shipped) | Decisions on a configuration change | Non-inferiority and equivalence in `compare()` (2.2); BCa failure reasons and opt-in fallback (1.2); category-skew diagnostic (1.3); unit triage (1.4) | At `confidence=0.96`, rejection at the margin holds 2.5%; at 0.95 it is measured and reported (revised 30 Oct 2026, see the 0.2.0 gate); a test for each BCa failure cause |
17
+ | 0.3.0 | Planning, CI gates and Arrow input | Sample-size planning (2.3); CLI quality gates (4.2); Arrow input (4.1) | Planned sizes reach their stated power on fresh seeds; CLI runs the example fixtures end to end; polars, DuckDB and pyarrow inputs give the same ratings fingerprint as pandas |
18
+ | 0.4.0 | Judge bias evidence | Within- and between-family alpha (3.2); unit covariates for length bias and self-enhancement (3.2); call telemetry, reduced (3.3) | Simulated same-family errors detected; fixtures for each diagnostic |
19
+ | 0.5.0 | Companion coefficients and plots | `metrics/` with Gwet's AC1 and AC2 (1.3); plotting extra (4.3) | Parity with irrCAC |
20
+ | 1.0.0 | Human-LLM alignment and a stable release | Mixed panels; alignment averaged over individual annotators and against a consensus label (3.1); stable API; frozen report schema; coverage simulations for interval, ratio and mixed-panel data | Mixed-panel end-to-end fixture with hand-checked values; paired intervals checked against a naive loop; reports from 1.0 read back by later releases; guidance regenerated from full-size runs |
21
+ | Research | Crossed intervals | A less conservative alternative to `two_way` (2.1) | Beats `two_way` on the coders grid |
22
+ | Research | A calibrated paired interval | A bootstrap-t or double bootstrap for `compare()` | Holds 2.5% at the margin at 95% on the paired grid |
23
+
24
+ Each gate is a test, as in the framing: a release ships when its gate passes.
25
+
26
+ ## Review of the proposals
27
+
28
+ | Proposal | Verdict | Reason | Assigned to |
29
+ | --- | --- | --- | --- |
30
+ | 1.1 Vectorised multinomial bootstrap | Shipped | 0.1.3 draws one multinomial weight matrix and computes every replicate as `o = W @ P` in batches, shared by all missing-data policies. The draft's formulas also contain two errors (below). | — |
31
+ | 1.2 BCa degeneracy fallback | Reduced | Undefined BCa intervals already give no interval, a reason and a warning pointing to percentile. A fallback by default would decide silently. | 0.2.0 |
32
+ | 1.3 Prevalence paradox and skew | Split | The skew warning exists. The proposed flag rule holds for nominal data only. AC1/AC2 are coefficients, not diagnostics, and belong in the planned `metrics/` module. | 0.2.0 (entropy), 0.5.0 (AC1/AC2) |
33
+ | 1.4 Coder influence and unit triage | Mostly shipped | `coder_jackknife().influence` and `estimate.per_unit` hold the proposed columns. Only a per-unit status breakdown is missing. | 0.2.0 (status breakdown) |
34
+ | 2.1 Crossed resampling, G-theory | Shipped / research / deferred | `scheme="two_way"` is the pigeonhole bootstrap, and the coders grid simulates it. A less conservative crossed interval is open research. G-theory variance components are an ICC-family estimand. | Research; `metrics/` backlog |
35
+ | 2.2 Non-inferiority and TOST | Accepted | Answers the primary user's question, "did an update change it?", with a declared margin. | 0.2.0 |
36
+ | 2.3 Sample-size planning | Accepted, re-scoped | Target alpha, marginals and coder count do not fix a data-generating process; the plan must name one. It belongs in a new module, not `design.py`. | 0.3.0 |
37
+ | 3.1 `study.calibrate()` | Merged | Same question as the human-LLM alignment design. One log with the reference as rows, named "alignment", not "validity" or "gold". | 1.0.0 |
38
+ | 3.2 Verbosity and family bias | Accepted, moved | Output length is a property of the unit, and the trace has one row per judge configuration, so lengths cannot live there. `model_family` is a ratings field, not a trace field. | 0.4.0 |
39
+ | 3.3 Cost, latency, energy | Reduced | Tokens, latency and cost are per call, so they belong in the long-form log. Energy and the "Δα per 1,000 tokens" ratio are dropped. | 0.4.0 |
40
+ | 4.1 Zero-copy DataFrame input | Accepted, other protocol | pandas is deprecating the `__dataframe__` interchange protocol in favour of the Arrow PyCapsule interface. Validation ends in pandas, so the input is convenient, not zero-copy. | 0.3.0 |
41
+ | 4.2 CLI quality gates | Accepted, with a spec file | A study's declarations do not fit in command-line flags. | 0.3.0 |
42
+ | 4.3 Terminal and figure plots | Accepted | Text interval bars in the core; figures in an optional extra. | 0.5.0 |
43
+
44
+ **Errors in proposal 1.1.** With $d_{o,u} = \frac{1}{m_u - 1}\sum_{j,k} C_{u,j} C_{u,k} D_{j,k}$, a replicate's observed disagreement is $\sum_u w_u d_{o,u} / \sum_u w_u m_u$. The draft multiplies the numerator by $m_u$ a second time. The draft also precomputes $\mathbf{D}$ once, which is wrong for ordinal data: ordinal distances depend on the marginals, so `_batch_delta2` rebuilds them from each replicate's marginals.
45
+
46
+ **Factual corrections.**
47
+
48
+ - The baseline is 0.1.4 (bipolar and circular distances), and the package supports Python 3.11 and later (`requires-python`, CI on 3.11 to 3.13), not 3.12 and later.
49
+ - There is no `BootstrapConfig`: bootstrap options are keyword arguments of `AlphaResult.bootstrap()`. There is no `StudyReport`: summaries are methods of `AlphaResult`, `BootstrapResult`, `ComparisonResult` and `JackknifeResult`. The trace is built with `TraceSidecar.from_frame()`, not `from_records()`.
50
+ - CI runs `uv run ruff check`, `uv run ruff format --check`, `uv run mypy` (strict, on `src` only) and `uv run pytest` on Linux, Windows and macOS, plus an isolated irrCAC parity job. `tests/test_parity.py` holds parity with third-party packages; parity of the vectorised path with the naive loop is in `tests/test_inference_units.py`.
51
+ - `python -m simulations.run_coverage --quick` runs 50 replications. At 95% that is a Monte Carlo standard error of about 3 points, so it is a smoke run and cannot show that coverage did not regress. A change to inference reruns the affected cells at full size (`--cells`), commits the results and regenerates the guidance.
52
+
53
+ ## 0.2.0: Decisions on a configuration change
54
+
55
+ Shipped in 0.2.0: every item below, with its gate passed on 30 Oct 2026.
56
+
57
+ **Non-inferiority and equivalence (2.2).** `compare(a=reference, b=candidate, margin=δ, test="non_inferiority" | "equivalence")` (decided 29 Oct 2026). The difference stays $\Delta = \alpha_b - \alpha_a$, so `a` is the reference.
58
+
59
+ ```python
60
+ diff = study.compare(
61
+ a={"condition_id": "frontier"}, # reference
62
+ b={"condition_id": "distilled"}, # candidate
63
+ margin=0.03,
64
+ test="non_inferiority", # or "equivalence"
65
+ confidence=0.95, # one-sided 2.5% test
66
+ n_resamples=2000,
67
+ random_state=7,
68
+ )
69
+ diff.verdicts["as_missing"] # "non_inferior" | "inferior" | "inconclusive"
70
+ ```
71
+
72
+ - The verdict reads the paired interval that `compare()` already reports at `confidence`. There is no separate significance parameter, so a report holds one interval and the verdict can always be checked against it by eye.
73
+ - Reading the lower end of a two-sided interval at `confidence` is a one-sided non-inferiority test at level (1 − `confidence`) / 2. The default 95% gives the conventional 2.5% test. Equivalence (two one-sided tests) at level γ needs `confidence` = 1 − 2γ, so a 5% equivalence test uses `confidence=0.90`; the guidance says so.
74
+ - Each test has its own verdicts. The draft's single list mixes them.
75
+ - **Non-inferiority:** `non_inferior` when the lower end is above −δ; `inferior` when the upper end is below −δ; otherwise `inconclusive`.
76
+ - **Equivalence:** `equivalent` when the interval lies inside (−δ, δ); `not_equivalent` when it lies outside [−δ, δ]; otherwise `inconclusive`.
77
+ - There is a verdict for every missing-data policy. An undefined interval is `inconclusive`. The margin and the test go into the declarations, the provenance and the JSON. The guidance tells users to fix the margin before looking at results.
78
+ - Bootstrap p-values are reported only if they invert the same interval (percentile or BCa), so a p-value can never contradict its verdict.
79
+
80
+ **Riders from the draft's phase 1.**
81
+
82
+ - **BCa failure reasons (1.2).** `Interval.undefined_reason` names the cause: undefined replicates (resamples holding one category, the common case in the guidance's skewed scenario), every replicate on one side of the estimate (infinite bias correction), or zero jackknife variance (undefined acceleration). An opt-in `on_undefined="percentile"` (the default `"nan"` keeps today's behaviour) returns the percentile interval over the defined replicates. That interval records `method_used="percentile"` in tables and JSON, and the `BootstrapWarning` says so. There is no fallback by default: in the guidance's skewed scenario at n = 30, the percentile interval covered 90.5%, and a reader of a BCa report should not get it without being told.
83
+ - **Category-skew diagnostic (1.3).** A `category_skew` diagnostic gives the normalised entropy of the pairable values over the declared domain, $H / \ln K$, and the effective number of categories, $e^H$. The existing `CategorySkewWarning` (one value holding 90% or more) keeps its share-based rule. The draft's rule, $D_e < 0.15$ and $1 - D_o > 0.85$, holds for nominal data only. Under the other metrics, $D_o$ and $D_e$ are in the scale's squared units, and $1 - D_o$ is not observed agreement.
84
+ - **Unit triage (1.4).** `estimate.per_unit` gains each unit's status counts (abstained, inapplicable, failed). Sorted by `d_o_contribution`, it is the adjudication list. No new study methods: coder influence is `coder_jackknife().influence`, and unit disagreement is `estimate.per_unit`.
85
+
86
+ **Gate.** A simulation grid of paired arms whose true difference sits on the margin (`python -m simulations.run_coverage --grid paired`), with a power curve at Δ = 0. `tests/test_bca.py` gains a case for each BCa failure cause, and the new interval fields survive a JSON round trip.
87
+
88
+ The gate as first written required the rejection rate at the margin to stay within Monte Carlo error of (1 − `confidence`) / 2. The first full run (30 Oct 2026) missed it. At 95%, BCa rejected 2.89% on average over the margin cells, against 2.5%, and 3.70% in the worst cell. The investigation found no defect in `compare()`, the verdicts or the p-values:
89
+ - The bootstrap standard error of the difference matched its true spread to within 1% on average.
90
+ - The difference is slightly heavy-tailed: an interval with the true spread still missed 2.62% per tail.
91
+ - Normal intervals, with the bootstrap or the jackknife standard error, rejected 2.74–2.78%, with the same worst cell.
92
+
93
+ The excess is the small-sample undercoverage of bootstrap intervals that the single-alpha guidance also shows. It is about 0.25 points at every level tried.
94
+
95
+ The gate was therefore revised on 30 Oct 2026, before the rerun that is judged against it. The rerun uses a true difference from 2,000,000 units, and BCa intervals at both 95% and 96%:
96
+
97
+ - **At `confidence=0.96`** (one-sided 2%), the BCa test rejects at most 2.72% on average over the margin cells, which is 2.5% plus two pooled Monte Carlo standard errors at 2,000 replications per cell. No margin cell exceeds 3.2%, 2.5% plus two of that cell's standard errors. The guidance recommends this setting when the test must hold 2.5%.
98
+ - **At `confidence=0.95`**, no margin cell exceeds 4.0%. The guidance reports the measured average and worst cell beside the verdicts' stated level.
99
+ - **At both levels**, every verdict agrees with its interval.
100
+
101
+ **Result (30 Oct 2026): the revised gate passes.**
102
+ - At 96%, BCa rejected 2.37% on average over the margin cells, and 3.15% in the worst cell (nominal, n = 500).
103
+ - At 95%, it rejected 2.89%, and 3.75% in the same cell.
104
+ - Every verdict and p-value agreed with its interval.
105
+
106
+ The rerun reused the first run's seed, so its 95% figures repeat the first run's. The 96% arm was therefore judged on the replications that suggested it.
107
+
108
+ A confirmation run of the ten margin cells on a fresh seed also passes. Its results are not committed; it is reproduced by `python -m simulations.run_coverage --grid paired --cells margin --seed 20261030 --out <dir>`.
109
+ - At 96%, BCa rejected 2.29% on average, and 2.70% at worst.
110
+ - At 95%, it rejected 2.88%, and 3.15% at worst.
111
+ - The average excess at 95% repeats, at about 0.4 points. The worst cell does not: nominal at n = 500 rejected 2.85% at 95%, so its 3.75% in the committed run was chance.
112
+
113
+ A calibrated interval (a bootstrap-t, or a double bootstrap that sets the level from the data) would remove the excess at about a hundred times the computation. It is listed under research.
114
+
115
+ ## 0.3.0: Planning, CI gates and Arrow input
116
+
117
+ **Sample-size planning (2.3).** A new `planning.py`; `design.py` holds coder definitions and design labels. The draft's inputs (target alpha, marginals, number of coders) do not determine how ratings are generated, so each plan names its generator:
118
+
119
+ 1. For nominal data, the minimum-sample-size rule in Krippendorff's *Content Analysis*: the number of values needed for a given minimum alpha, smallest category probability and significance level. The exact table and edition are to be verified before building.
120
+ 2. For every metric, simulation with the latent-class "know or guess" generator of `simulations/`. That generator moves into the package, because the wheel ships `src/reliability_lab` only. The search over N uses common random numbers and a monotone fit, because a plain binary search on noisy power estimates can stop on noise.
121
+ 3. The same machinery plans the non-inferiority test of 0.2.0: the number of units needed to show non-inferiority at margin δ when the true difference is zero.
122
+
123
+ The result records the generator and its parameters, and the documentation says the answer holds for that generator only.
124
+
125
+ **CLI quality gates (4.2).** `[project.scripts] reliability-lab = "reliability_lab.cli:main"`, written with `argparse` and `tomllib`, both in the standard library from 3.11, so there are no new core dependencies.
126
+
127
+ - Declarations come from a study file in TOML (decided 29 Oct 2026), not from flags. A study needs a measurement, values in scale order, coders, a design, missing-data policies, `abstain_as`, `inapplicable_as` and retries. These should be versioned with the judge, and comments in the file can record why a threshold or margin was chosen. The standard library reads TOML but cannot write it, so the CLI only reads study files.
128
+
129
+ ```toml
130
+ [study]
131
+ measurement = "ordinal"
132
+ values = [1, 2, 3, 4, 5]
133
+ coders = ["model_id", "condition_id"]
134
+ design = "prompt_robustness"
135
+ missing_policies = ["as_missing", "sensitivity"]
136
+ retries = "last"
137
+
138
+ [columns]
139
+ unit = "response_id"
140
+ rater = "rater_id"
141
+ value = "score"
142
+ status = "rating_status"
143
+
144
+ [gate]
145
+ min_alpha = 0.70
146
+ min_low = 0.60 # lower end of the 95% interval
147
+ ```
148
+
149
+ - `reliability-lab evaluate study.toml --min-alpha 0.70 --min-low 0.60 --output report.json`
150
+ - `reliability-lab compare study.toml --a condition_id=prompt_v1 --b condition_id=prompt_v2 --margin 0.03 --test non_inferiority`. It reads one log, or several concatenated, and selects the arms as the API does.
151
+ - Ratings are read from CSV, or from Parquet with the `parquet` extra.
152
+ - Exit codes: 0 pass; 1 usage or input error; 2 a threshold or verdict fails; 3 a design or trace finding (`TraceWarning`, `CoderDependenceWarning`), with `--strict` only.
153
+ - Thresholds apply to every reported missing-data policy unless `--policy` names one. A judge that passes only when its failures are dropped does not pass silently.
154
+
155
+ **Arrow input (4.1).** Moved from 0.5.0 on 29 Oct 2026, because the CLI reads Parquet through pyarrow anyway: one path from Arrow to pandas serves both. `RatingFrame.from_long` and `TraceSidecar.from_frame` accept any object with the Arrow PyCapsule stream interface (`__arrow_c_stream__`), such as polars data frames, DuckDB relations and pyarrow tables. The import goes through pyarrow (the `parquet` extra) and ends in pandas, because validation builds pandas columns. The promise is "no manual `to_pandas()`", not zero-copy.
156
+
157
+ **Gate.** Planned sample sizes reach their stated power when re-simulated with fresh seeds. CLI tests run the example fixtures end to end and check exit codes and the JSON report. The same ratings passed as pandas, polars, DuckDB and pyarrow give the same ratings fingerprint.
158
+
159
+ ## 0.4.0: Judge bias evidence
160
+
161
+ **Within- and between-family alpha (3.2).** For `inter_model`, pairwise alpha within model families and between them, from `model_family` in the ratings. That field is constant per `model_id`; the trace records `provider`, not family. This turns today's `CoderDependenceWarning` into a number. Intervals use the units scheme, paired with the overall alpha.
162
+
163
+ **Unit covariates (3.2).** A unit sidecar keyed by `unit_id` (for example `UnitSidecar.from_frame(df, unit=..., length_a=..., length_b=..., output_family=...)`), checked against the ratings as the trace is. It serves two diagnostics:
164
+
165
+ - **Length bias:** rank correlation between the length difference (or the response length, for pointwise scores) and the judgment. For `order_robustness`, it also checks whether order-consistent wins favour the longer candidate. A correlation with length is not bias on its own, because longer answers can be better. The diagnostic describes the data; a causal claim needs pairs matched on length.
166
+ - **Self-enhancement:** the judged output's model family against the judge's family. The framing deferred this because it needs exactly this field.
167
+
168
+ **Call telemetry, reduced (3.3).** Optional numeric columns in `RatingFrame.from_long`: `prompt_tokens`, `completion_tokens`, `latency_ms`, `cost`. They are one value per attempt, retries included. Alpha and comparison reports show per-condition totals and means per counted rating beside alpha. Two parts of the draft are left out:
169
+
170
+ - `energy_wh`: providers do not report it, so it would be an estimate with no provenance.
171
+ - The "Δα per 1,000 tokens" ratio: a noisy difference over a cost has no interval here, and invites over-reading. The two are reported side by side instead.
172
+
173
+ The library still only summarises logs and never calls providers (framing, non-goals).
174
+
175
+ **Gate.** Simulated same-family correlated errors (the existing correlated-errors process) give higher within-family than between-family alpha. Fixtures cover each diagnostic and the unit-sidecar checks.
176
+
177
+ ## 0.5.0: Companion coefficients and plots
178
+
179
+ **`metrics/` (1.3).** Gwet's AC1 (nominal) and AC2 (weighted) open the module the framing planned for later coefficients. They are reported as separate, labelled coefficients, never as diagnostics of alpha. They take the same declarations (the distance gives the weights) and the same missing-data policies. Kappas and ICC follow in the same module. G-theory variance components (2.1) belong with ICC, since they are the variance decomposition ICC is built from.
180
+
181
+ **Plots (4.3).**
182
+
183
+ - Core: text interval bars in the bootstrap and comparison summaries, one row per missing-data policy, with no new dependency.
184
+ - Optional `plot` extra (matplotlib, already in the `sim` group): `plot_policy_forest`, `plot_coincidence` (observed against expected coincidences) and `plot_coder_influence`. They reuse the light and dark styles of `simulations/report.py`.
185
+
186
+ **Gate.** AC1 and AC2 match irrCAC in the isolated parity job CI already runs. Plot tests check figure structure (axes, labels, one mark per estimate), not pixels.
187
+
188
+ ## 1.0.0: Human-LLM alignment and a stable release
189
+
190
+ Alignment moved from 0.2.0 on 29 Oct 2026. Until then, `design="human_llm_alignment"` and ratings that mix human and LLM rows are refused with a message naming 1.0. Besides alignment, 1.0 requires a stable API, a frozen report schema and wider coverage simulations ([below](#release-requirements)).
191
+
192
+ **Mixed panels.** `check_design` accepts human and LLM rows together under `design="human_llm_alignment"`. Coders are defined as now: a human coder by `rater_id`, an LLM coder by model and condition.
193
+
194
+ **The alignment report.** Each quantity is labelled and none is pooled with another:
195
+
196
+ 1. **Human alpha**: the human coders among themselves. It is the reference's own reliability, and it limits how much can be read into alignment. Beside it, the average pairwise alpha over pairs of human coders, the counterpart of the first alignment coefficient.
197
+ 2. **LLM alpha**: the LLM coders among themselves.
198
+ 3. **Alignment**, as two coefficients (decided 29 Oct 2026):
199
+ - **Averaged over annotators.** For each LLM configuration, alpha of the two-coder panel it forms with each human coder, averaged over the human coders. It is compared with the humans' average pairwise alpha, computed the same way, because a two-coder alpha and an alpha of the whole panel are different quantities.
200
+ - **Against a consensus label.** For each LLM configuration, alpha of the two-coder panel it forms with a declared consensus. The consensus enters as rows of the same `RatingFrame` (for example `rater_id="consensus"`) and is declared as the reference, so one log carries one fingerprint and one split manifest. It is derived from the human coders, so it is left out of human alpha and of the averaged coefficient.
201
+
202
+ For each coefficient, proposal 3.1 adds the mean shift (ordinal, interval and ratio data) and a status-aware confusion table: LLM value or failure status against the reference value.
203
+
204
+ Reports call this "alignment with the human reference". The framing holds that human agreement is not automatic ground truth, so "criterion validity" and "gold" are not used.
205
+
206
+ **Intervals.** The units scheme resamples the same units for every quantity, as `compare()` does for two arms, and recomputes each average in every replicate. The gap between an LLM configuration's average and the humans' average then gets a paired interval. It answers whether the judge agrees with each annotator about as well as annotators agree with each other. Against a consensus, the interval is on the coefficient alone. No human counterpart is reported, because an annotator compared with a consensus built partly from their own labels would be favoured. Whether an LLM can replace a given annotator is a different test (`alttest`, Calderon et al. 2025) and stays out of scope.
207
+
208
+ ### Release requirements
209
+
210
+ Decided 29 Oct 2026. A hosted API reference was considered and is not required for 1.0.
211
+
212
+ - **Stable API.** Semantic versioning from 1.0: a name exported by `reliability_lab` (its `__all__`) is removed or changed only in a major release, after a deprecation warning for at least one minor release. Leftovers are removed before the freeze, such as `BootstrapResult.experimental`, which has always been false since 0.1.1. The README drops its "alpha" status, and the classifier moves to "Development Status :: 5 - Production/Stable".
213
+ - **Frozen report schema.** JSON and Parquet reports already carry `schema_version` 1. From 1.0, 1.x releases may add fields but never remove or rename one; a change that would break a reader raises `schema_version`. Fields added before 1.0 (non-inferiority verdicts, BCa failure reasons, telemetry) are part of the frozen schema.
214
+ - **Wider coverage simulations.** The guidance simulates nominal and 5-point ordinal data only. Before 1.0 it covers interval and ratio data, and mixed panels: the intervals on both alignment coefficients and on the gap between the LLM's average and the humans' average. Each recommendation in the guidance's "Which scheme to use" table holds or is revised.
215
+
216
+ **Gate.**
217
+
218
+ - A provider-neutral end-to-end fixture of a mixed-panel study, with and without a consensus (validation plan, item 6). Hand-checked values cover human alpha, the averaged coefficient and the consensus coefficient. The paired intervals are checked against a naive loop, as the units scheme is.
219
+ - Golden report files written by 1.0 are committed, and every later 1.x release reads them in its tests.
220
+ - The guidance is regenerated from committed full-size runs of the new grids.
221
+
222
+ ## Research: a less conservative crossed interval
223
+
224
+ `scheme="two_way"` is the pigeonhole bootstrap. When coders are alike, it covered 99.3–99.8% and was about 1.5 times as wide as the units interval, because it counts the rating noise twice (Owen 2007). Candidates are Owen's variance correction for the pigeonhole bootstrap and a two-way jackknife. A replacement ships only if, on the coders grid, it covers 93% or more in every scenario and is narrower than `two_way` when τ = 0. There is no release target.
225
+
226
+ ## Research: a calibrated paired interval
227
+
228
+ At 95%, the paired intervals of `compare()` miss about 0.25 points too often per tail, so a 2.5% non-inferiority test rejects about 2.9% of the time at the margin (the 0.2.0 gate). Candidates are a bootstrap-t, with a jackknife standard error inside each resample, and a double bootstrap that calibrates the level. Either one ships only if, on the paired grid at 95%, every margin cell rejects within two Monte Carlo standard errors of 2.5%, at a run time usable on studies of a few thousand units. There is no release target.
229
+
230
+ ## Already shipped
231
+
232
+ | Proposal | Where it is |
233
+ | --- | --- |
234
+ | Vectorised multinomial bootstrap (1.1) | `draw_weights` and `replicate_alphas` in `inference.py` (0.1.3); parity with the naive loop in `tests/test_inference_units.py` |
235
+ | Pigeonhole bootstrap (2.1) | `scheme="two_way"` (0.1.2) |
236
+ | Crossed coverage simulations (2.1) | `python -m simulations.run_coverage --grid coders`; "Generalising over coders" in the [coverage guidance](../guidance/bootstrap-coverage.md) |
237
+ | `study.coder_influence()` (1.4) | `result.coder_jackknife().influence`: `alpha_without`, `influence` (alpha minus alpha without the coder), `units_rated`; per-coder error rates in `result.coverage.per_coder` |
238
+ | `study.unit_disagreement()` (1.4) | `estimate.per_unit`: `pairable_values`, `disagreement`, `d_o_contribution` |
239
+ | Skew warning (1.3) | `CategorySkewWarning`, `estimate.category_distribution`, `estimate.observed_agreement` |
240
+ | BCa degeneracy handling (1.2, part) | An undefined BCa interval gives a reason and a `BootstrapWarning` pointing to `method="percentile"`; parity with `scipy.stats.bootstrap` on degenerate data in `tests/test_bca.py` |
241
+ | A ratings generator in the package (2.3, part) | `simulate_ratings` (0.1.5): know-or-guess ratings for any metric at a chosen alpha, with random or near-miss errors. Unlike the generator of `simulations/`, it has no hard units or clusters |
242
+
243
+ ## Decisions
244
+
245
+ - [x] Human-LLM alignment moves from 0.2 to 1.0 (29 Oct 2026).
246
+ - [x] Alignment coefficients: both, averaged over individual annotators and against a consensus label (29 Oct 2026).
247
+ - [x] Non-inferiority API: one `margin` with a `test` name, read from the `confidence` interval; no separate significance parameter (29 Oct 2026).
248
+ - [x] Study file format for the CLI: TOML, read with the standard library (29 Oct 2026).
249
+ - [x] Arrow input moves from 0.5.0 to 0.3.0, beside the CLI (29 Oct 2026).
250
+ - [x] 1.0 also requires a stable API, a frozen report schema and coverage simulations for interval, ratio and mixed-panel data; a hosted API reference is not required (29 Oct 2026).
251
+ - [x] The 0.2.0 gate is revised after its first run missed it by about 0.4 points. The test's size at 95% is measured and reported, and `confidence=0.96` must hold 2.5% (30 Oct 2026).
252
+
253
+ ## References
254
+
255
+ 1. Krippendorff, K. *Content Analysis: An Introduction to Its Methodology.* Sage. Minimum-sample-size rule; edition and table to be verified.
256
+ 2. Owen, A. B. (2007). [The pigeonhole bootstrap](https://doi.org/10.1214/07-AOAS122). Annals of Applied Statistics 1(2).
257
+ 3. Calderon, N., Reichart, R., & Dror, R. (2025). [The Alternative Annotator Test for LLM-as-a-Judge](https://arxiv.org/pdf/2501.10970). ACL 2025.
258
+ 4. Gwet, K. L. (2008). [Computing inter-rater reliability and its variance in the presence of high agreement](https://doi.org/10.1348/000711006X126600). British Journal of Mathematical and Statistical Psychology 61(1).
259
+ 5. pandas: [`DataFrame.__dataframe__`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.__dataframe__.html), which recommends the Arrow PyCapsule interface for new development.
260
+
261
+ Verified on 29 Oct 2026: every "shipped" claim against the code at 0.1.4, the pandas interchange deprecation, and the Owen (2007) and Gwet (2008) citations. The Krippendorff sample-size table was not re-opened.