evalaudit 0.3.0__tar.gz → 0.5.0__tar.gz

evalaudit-0.5.0/CHANGELOG.md

@@ -0,0 +1,840 @@
+# Changelog
+
+Every change to evalaudit that a caller can see is recorded here. The format
+follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the
+version numbers follow [semantic versioning](https://semver.org/). Before 1.0
+a minor version can break things, and 0.3.0, 0.4.0 and 0.5.0 all do.
+
+An entry marked **Breaking** changes what a call that worked on the previous
+release returns, raises or prints. The mark goes on the entry wherever it
+sits, under Changed, Fixed or Removed.
+
+Every release is dated by the day PyPI received its upload, in UTC. 0.3.0 is
+the first release with a tag, and 0.4.0 and 0.5.0 have one too. The entries
+below 0.3.0 were reconstructed after the fact, and their contents were
+checked against the published wheels and the commit each wheel matches.
+
+## [Unreleased]
+
+## [0.5.0] - 2026-09-26
+
+`judge_validation` can now measure the judge against a second human. Most
+of the rest is text. Several printed sentences change, most of them for
+hyphens and plurals, and one `audit` finding changes severity. Figures
+change in one case. In the both-orders design, `position_bias` now scores a
+pair judged more than once in an order when its first two reversed
+judgements held a tie and a later two did not. That moves the count of
+pairs scored, the consistency rate and its interval, and the figures on the
+flips. A pair judged once in each order scores as before. Anything that
+matches on the old sentences will stop matching. Every entry was checked by
+running the same calls against the v0.4.0 tag and against this tree.
+
+### Changed
+
+- Nothing.
+
+### Added
+
+- `judge_validation(..., item_ids=None, human_baseline=None, ties=None)`
+  measures the judge against a human baseline. `human_baseline` is a frame
+  with `item_id`, `cluster_id`, `rater_id` and `rating`, one row per human
+  rating, and `item_ids` names the item at each position of `human` and
+  `judge`. The result carries judge-human and human-human alpha on the same
+  items, and a percentile interval on judge-human minus human-human that
+  resamples whole clusters. The verdict reads that interval and nothing
+  else, and is called provisional when it rests on fewer than 30 clusters.
+  The interval is refused when fewer than 90% of resamples leave both
+  alphas defined.
+
+  `ties` is required with a baseline. `"category"` keeps a tie as a label of
+  its own and `"drop"` sets it aside as no label, and the choice applies to
+  every figure in the result, the headline included. Baseline items are set
+  aside by three rules in a fixed order, and the summary counts what each
+  one took. A baseline supports only `level="nominal"`. On the MT-Bench
+  human judgements the figures reproduce `analysis/mt-bench.md` to the third
+  decimal. The new arguments are refused, each with its own message, in the
+  combinations that cannot work, such as `ties` or `item_ids` without a
+  baseline. Missing values in `item_id`, `cluster_id`, `rater_id` or
+  `item_ids` are refused, and the message names the column, the count and
+  where the first one is. Every check on the baseline runs before anything
+  is resampled.
+
+- `JudgeValidation.n_dropped_ties`, `JudgeValidation.ties`, eleven fields
+  that start with `baseline_`, and the properties `has_baseline` and
+  `has_baseline_interval`. The new fields are the last in the dataclass and
+  have defaults, so building a `JudgeValidation` by position with 0.4.0's
+  fields still works. A call that passes none of the three new arguments
+  returns the same numbers as 0.4.0.
+
+### Fixed
+
+- **Breaking.** `JudgeValidation.summary()` said "No interval was computed,
+  so nothing here is placed against sampling error." when the resamples had
+  been drawn and too few of them left alpha defined. On three items, two of
+  them unanimous, with `n_boot=2000` and `seed=0`, it now reads "There is
+  no interval on that alpha. 591 of 2000 resamples left it undefined, above
+  the 10% this reports through. Those are the resamples where every label
+  was the same, so percentiles of the rest would understate the upper
+  bound." When one resample is left undefined it reads "That is the
+  resample where every label was the same", and when only one was drawn
+  the count reads "1 of 1 resample". With `n_boot=0` it still says no
+  interval was computed. The `audit` judge finding carries the new
+  sentence.
+
+- **Breaking.** The `audit` finding on a margin whose interval clears zero
+  opened "This is the margin between new and old, and it survives the
+  fit." No model is fitted there. It now reads "This is the margin between
+  new and old, and its interval excludes zero."
+
+- **Breaking.** A count or figure of one was printed with a plural noun, or
+  with a verb or pronoun that assumed more than one. Each now takes the
+  singular. Anything other than one prints as before.
+
+  - `ScoreCI.summary()`. "With only 1 observations" now reads "With only 1
+    observation", and "The interval spans 1 points." now reads "The
+    interval spans 1 point."
+  - `ComparisonResult.summary()`. "1 of 1 items changed between systems."
+    now reads "1 of 1 item changed between systems."
+  - `KappaResult.summary()`. "(1 items, 2 raters, 2 categories)" now reads
+    "(1 item, 2 raters, 2 categories)".
+  - `AgreementResult.summary()`. "1 of 1 items graded more than once" now
+    reads "1 of 1 item graded more than once". Where the interval is
+    refused, "1 of 1 resamples came back undefined" now reads "1 of 1
+    resample came back undefined", and after one undefined resample "Those
+    resamples are the unanimous ones" now reads "That resample is the
+    unanimous one".
+  - `BTResult.summary()`. With two models, "0 of 1 pairs is separable" now
+    reads "0 of 1 pair is separable". With one pair separable among
+    several, "1 of 3 pairs are separable" now reads "1 of 3 pairs is
+    separable", since the verb follows the separable count. Where the
+    interval is refused, "1 of 1 resamples came back undefined" now reads
+    "1 of 1 resample came back undefined".
+  - The `ValueError` from `bradley_terry` for a missing item id. "item_id is
+    missing on 1 of 1 comparisons" now reads "item_id is missing on 1 of 1
+    comparison".
+  - `PositionBias.summary()`, randomised design. "Position A won 100.0% of 1
+    judgements" now reads "of 1 judgement". "1 pairs were also run in the
+    reverse order" now reads "1 pair was also run in the reverse order".
+    "1 judgements were ties and are left out of the rates above." now reads
+    "1 judgement was a tie and is left out of the rates above."
+  - `PositionBias.summary()`, both orders. "1 of 1 pairs were run in both
+    orders" now reads "1 of 1 pair was run in both orders", and "1 of 2
+    pairs were run in both orders" now reads "1 of 2 pairs was run in both
+    orders", since the verb follows the pairs run both ways. "on 0.0% of 1
+    pairs" now reads "on 0.0% of 1 pair", and "Of the 1 pairs it flipped
+    on" now reads "Of the 1 pair it flipped on".
+  - `LengthBias.summary()`. "(1 pairs)" now reads "(1 pair)", and "(1
+    disagreements)" now reads "(1 disagreement)", with or without a fitted
+    model. A length step that prints as 1, "1 characters of extra length
+    multiplies" or "1 characters multiplies", now reads "1 character".
+  - `PowerResult.summary()`. "you need roughly 1 paired comparisons" now
+    reads "you need roughly 1 paired comparison", and "about 1 discordant
+    pairs, and McNemar reads only those." now reads "about 1 discordant
+    pair, and McNemar reads only that one." A difference that prints as 1,
+    "is 1 points" or "a difference of 1 points", now reads "1 point".
+
+  The `audit` compare, agreement, position, length and power findings carry
+  these sentences word for word.
+
+- **Breaking.** Compound modifiers are hyphenated in printed text.
+
+  - `PowerResult.summary()`. "To detect a 5 point difference", "leaves at
+    most a 5 point difference" and "To find a 5 point difference" now read
+    "5-point", and the same holds for any figure. "the continuity
+    corrected and exact forms" now reads "continuity-corrected", and
+    "Pairing removes the item to item difficulty" now reads
+    "item-to-item". The `audit` power finding carries these sentences.
+  - The `audit` action under "Rater agreement holds up". "Keep the double
+    graded sample" now reads "Keep the double-graded sample".
+  - The `ValueError` from `detectable_effect` and `min_sample_size` when
+    power does not exceed alpha. "A two sided test" now reads "A two-sided
+    test".
+
+- **Breaking.** `PositionBias.summary()` printed a rate, an interval and a
+  verdict when nothing could be scored.
+
+  - Randomised design. On three pairs judged once, each a tie, it printed
+    "Each pair was judged once, so this reports the position-A win rate.
+    Position A won nan% of 0 judgements (95% CI: nan% to nan%, exact
+    binomial p=nan). The interval includes 50%, so the data cannot show
+    that position moved the judge." and went on to the caveat about
+    randomised order. It now reads "Each pair was judged once, so this
+    would report the position-A win rate. Every judgement was a tie, so
+    there is no rate to report and nothing here tests position."
+  - Both orders. On two pairs run both ways, each with a tie on one side,
+    and one pair judged once, it printed "2 of 3 pairs were run in both
+    orders, so this reports the consistency rate. The judge named the same
+    output under both orderings on nan% of 0 pairs (95% CI: nan% to nan%).
+    It never flipped, so there is no direction to test and nothing here
+    points at position." It now reads "2 of 3 pairs were run in both
+    orders, so this would report the consistency rate. In every pair run
+    both ways, the judge called a tie every time in at least one of the two
+    orders, so no pair could be scored and there is no rate to report.
+    Nothing here tests position."
+
+  The randomised summary ended "3 judgements were ties and are left out of
+  the rates above." and the both-orders one ended the same way with 2.
+  There are no rates for the ties to be left out of, so that sentence is
+  gone from both. The `audit` position finding on this data was an info
+  finding titled "The data cannot show a position effect in the pairwise
+  judgements". It is now a warning titled "Position bias is undefined on
+  this data", and it carries the new summary.
+
+- **Breaking.** In the both-orders design, `position_bias` formed each
+  pair's couple, its first two judgements that are reverses, before looking
+  at ties, and dropped the pair if either was a tie. A pair judged more than
+  once in an order was dropped for a tied first couple even when a later
+  couple had no tie, and the summary did not say so. Ties are now set aside
+  first, so the pair is scored on its first two decisive judgements that are
+  reverses. On two consistent pairs and a third judged x/y and y/x as ties
+  and then x/y for x and y/x for y, it printed "The judge named the same
+  output under both orderings on 100.0% of 2 pairs (95% CI: 34.2% to
+  100.0%). It never flipped, so there is no direction to test and nothing
+  here points at position." It now reads "The judge named the same output
+  under both orderings on 66.7% of 3 pairs (95% CI: 20.8% to 93.9%). Of the
+  1 pair it flipped on, 1 went to whichever output was shown first (100.0%,
+  95% CI: 2.5% to 100.0%, exact binomial p=1.0000)." and goes on to say the
+  data cannot show that the flips have a direction. `n_pairs_scored` goes
+  from 2 to 3 and `n_decisive` from 0 to 1. A pair judged once in each
+  order scores as before, since setting its ties aside leaves a couple
+  exactly when neither judgement was a tie. The `audit` position finding
+  carries the new figures.
+
+- The `rater_agreement` docstring said that without an interval nothing in
+  the result would name an outlier rater. Since 0.4.0 the summary lists
+  alpha with each rater left out and names no rater either way, and the
+  docstring now says so.
+
+- The README said scipy computes McNemar's test. It is statsmodels. It also
+  said the package builds fifteen intervals, while its own lists named
+  seventeen. It now counts eighteen, with the interval on the human baseline
+  among the eight pinned against a reference.
+
+### Removed
+
+- Nothing.
+
+## [0.4.0] - 2026-09-11
+
+Nearly everything in 0.4.0 changes what a
+summary or a report prints. One rule is behind it. A result that does not
+clear its threshold has failed to show something, and it has not shown the
+opposite. Summaries that described such a result as if the opposite were
+shown, and verdicts decided on a point estimate where an interval existed,
+now say what the data did and did not show. Several `audit` findings change
+severity on data you have already run. Each breaking entry says what it
+was, what it is now and what to do about it. Every entry was checked by
+running the same calls against 0.3.0 from PyPI and against the code
+released as 0.4.0.
+
+### Changed
+
+- **Breaking.** `audit` reads rater agreement and judge agreement off the
+  interval on alpha. An interval wholly above the threshold keeps "Rater
+  agreement holds up" or "The judge tracks the humans", and one wholly below
+  it keeps the "below the working threshold" title. An interval running both
+  sides of the threshold, or no interval at all, is now a `warning` titled
+  "The data cannot show that rater agreement clears the working threshold"
+  or "The data cannot show that the judge clears the working threshold".
+  0.3.0 decided on the point estimate. Ratings with alpha 0.730 and an
+  interval of 0.466 to 0.933 came back `info` as "Rater agreement holds up",
+  and so did any ratings audited with `n_boot=0`. Judge labels with alpha
+  0.581 and an interval of 0.467 to 0.691 came back "Judge-human agreement
+  is below the working threshold". Rerun any audit you have sent. A finding
+  that moved from `info` to `warning` also changes the report's verdict.
+
+- **Breaking.** Without an `effect_of_interest`, `audit` no longer sizes
+  power on the margin it measured. 0.3.0 made the observed margin the effect
+  at issue, as in "The effect at issue is 5.0 points, the margin this eval
+  reports." Power computed on an observed effect is a monotone function of
+  the p-value, so on a margin whose interval includes zero the finding came
+  out `critical` almost every time and repeated the comparison. The finding
+  now reports the design's reach, the smallest difference the eval could
+  find at the configured power, and makes no claim about the margin. It is
+  a `warning` titled "No effect of interest was stated, so this reports the
+  eval's reach", because without a stated effect the audit does not know
+  what size of difference matters. Its lead says "This reports what the
+  design could find and makes no claim about the margin it measured." On 400
+  paired items with a 5.0-point margin and an interval of 1.1% to 9.0%,
+  0.3.0 opened the report with a `critical` power finding and the verdict
+  said the conclusion did not hold. Two systems with a margin of exactly
+  zero, which 0.3.0 skipped, get the same warning.
+
+  With one system and no stated effect the check is still skipped, and its
+  reason no longer offers the observed margin as a stand-in. It read
+  "Nothing to size the eval against. Pass effect_of_interest in the config,
+  or scores for two systems so the observed margin can stand in for it."
+  and now reads "No effect of interest was stated and there is no comparison
+  whose reach could be reported. Pass effect_of_interest in the config, or
+  scores for two systems."
+
+  An audit of two pass-rate systems with no stated effect now always
+  carries this warning, so its report never comes back with no warnings.
+  Set `effect_of_interest` to the difference the decision turns on.
+
+- **Breaking.** With an `effect_of_interest` stated, `audit` puts a power
+  finding at `info` in two cases where 0.3.0 made it `critical`, because the
+  margin's interval already answers the question power asks.
+
+  - The margin's interval clears zero. The finding is titled "The margin is
+    established, and a tighter estimate would take more items". 0.3.0
+    titled it "This eval could not have detected the effect at issue".
+  - The margin's interval includes zero and stops short of the stated
+    effect in both directions. The finding is titled "The interval on the
+    margin rules out the effect at issue".
+
+  Rerun any audit whose report carried a power finding.
+
+- **Breaking.** With an `effect_of_interest` stated, `audit` rewords the
+  power finding on a margin whose interval includes zero. Its title is now
+  "This eval had under 80% power for the effect at issue", with the
+  percentage taken from `power`, because a design
+  short of that power can still find the effect, only less often. Its action
+  no longer says "No conclusion about an effect this size can be drawn from
+  this data, in either direction. A null result here says the eval was too
+  small and says nothing about the systems." It reads the margin's interval
+  one side at a time and says one of three things. Either "The interval on
+  the margin runs past an effect this size in both directions, so the data
+  cannot show an effect this size and cannot rule one out.", or "The
+  interval on the margin rules out old ahead by this much, and it cannot
+  rule out new ahead by this much." with your system names, or for a single
+  system "No margin was measured here, so there is no interval to read
+  against an effect this size." Anything matching on the old title or action
+  will stop matching.
+
+- **Breaking.** `AuditReport.summary()` rewords its verdict, which is the
+  first line of every report.
+
+  - With critical findings, "the conclusion this data is being asked to
+    support does not hold as stated." is now "this data cannot support the
+    conclusion as stated. That does not mean the conclusion is wrong."
+  - With warnings and no criticals, "The result stands and the design
+    weakens it, so report it with what the warning say attached." is now
+    "The checks that ran did not find a problem that leaves the conclusion
+    unsupported. That does not mean the eval is sound. Report the result
+    with what the warning says attached."
+  - With neither, "All 4 checks that ran came back clean, so the eval holds
+    up on everything this report could test." is now "The 3 checks that ran
+    did not find a problem in what they could test. That does not mean the
+    eval is sound." The number changed too. 0.3.0 counted findings, and the
+    score check gives one finding per system, so two systems made three
+    checks read as four. A single check reads "The one check that ran", and
+    a report where nothing could run reads "No check could run, so this
+    report has tested nothing."
+
+  Anything matching on the old verdicts will stop matching. The severity
+  counts are unchanged by this entry.
+
+- **Breaking.** `audit` rewords several titles and actions.
+
+  - "No position effect in the pairwise judgements" is now "The data cannot
+    show a position effect in the pairwise judgements". Its action opens
+    "The data has not cleared the judge of position bias." where it opened
+    "Nothing to do here."
+  - "No length effect in the judge's choices" is now "The data cannot show
+    a length effect in the judge's choices". Its action "Nothing to do
+    here." is now "The data has not cleared the judge of rewarding length."
+  - An alpha that does not exist is titled "Rater agreement is undefined on
+    this data" or "Judge-human agreement is undefined on this data", and the
+    lead ends "and this data gives no alpha to hold against it." 0.3.0
+    titled it "Rater agreement is below the working threshold" or
+    "Judge-human agreement is below the working threshold", and said the
+    grades "reproduce less well than the 0.667".
+  - The action on a margin that clears zero said "The lower bound is the
+    size of the win the data actually supports." When the second system
+    leads, the lower bound is the far end of the interval. It now says "The
+    end of the interval nearer zero is the smallest margin this data
+    supports."
+  - The length lead "Two models run here and they point opposite ways." now
+    also needs the preference fit's interval to clear zero. 0.3.0 printed it
+    off the sign of a preference coefficient whose interval ran over zero,
+    and said the judge "picks the longer answer more often" beside an odds
+    ratio of 0.94 to 1.40. Such data now gets the plain lead for the
+    direction the finding reports.
+
+- **Breaking.** Module summaries put every result that falls short of its
+  threshold in one form, "so the data cannot show that" followed by "That
+  does not mean".
+
+  - `ComparisonResult.summary()`. "The interval crosses zero, so the data
+    cannot confirm that either system is better." is now "The interval
+    includes zero, so the data cannot show that either system is better.
+    That does not mean the two are level."
+  - `PositionBias.summary()`, randomised design. "The interval covers 50%,
+    so the data cannot show that position moved the judge." is now "The
+    interval includes 50%, so the data cannot show that position moved the
+    judge. That does not mean the judge ignores position."
+  - `PositionBias.summary()`, both orders. "The flips split evenly across
+    the two positions, so this is an unsteady judge rather than a
+    position-biased one." is now "That share is not clear of 50% at this
+    many flips, so the data cannot show that the flips have a direction.
+    That does not mean the judge is free of position bias." 0.3.0 printed
+    the old sentence at 7 flips of 9 toward the answer shown first.
+  - `LengthBias.summary()`. "The interval covers no effect, so the data
+    cannot show that length moved the judge." is now "The interval includes
+    an odds ratio of 1, so the data cannot show that length moved the
+    judge. That does not mean length plays no part in its choices." For the
+    disagreement fit, "That interval covers no effect, so the judge does
+    not depart from the humans in the direction of length." is now "That
+    interval includes an odds ratio of 1, so the data cannot show that the
+    judge breaks with the humans toward longer or shorter answers. That does
+    not mean it follows them on length."
+  - `AgreementResult.summary()`. "so the data cannot distinguish the raters"
+    is now "so the data cannot show that any one rater is pulling alpha
+    down" where every leave-one-out alpha is undefined. Where there are
+    leave-one-out figures the summary no longer judges the raters at all,
+    as the entry on rater dropout below says.
+  - `JudgeValidation.summary()`. "the data cannot single out a slice" is now
+    "the data cannot show that the judge does worse on any one slice".
+    Where the intervals overlap it adds "That does not mean it does equally
+    well on all of them."
+
+  Anything matching on the old sentences will stop matching.
+
+- **Breaking.** `AgreementResult.summary()` and `JudgeValidation.summary()`
+  name one of Krippendorff's bands only when the whole interval sits in it.
+  0.3.0 named the band from the point estimate, with "That is at or above
+  0.800, the conventional bar for treating coded data as reliable.", "That
+  sits between 0.667 and 0.800, which supports tentative conclusions and no
+  firm ones." and "That is below 0.667, the conventional floor for drawing
+  any conclusion from coded data." These now read "The whole interval sits
+  above 0.800", "The whole interval sits between 0.667 and 0.800" and "The
+  whole interval sits below 0.667". An interval running over 0.667 reads
+  "The interval runs both sides of 0.667, the conventional floor for drawing
+  any conclusion from coded data, so the data cannot show that rater
+  agreement clears it. That does not mean it falls short of it." In
+  `JudgeValidation` the subject is "the judge's agreement with the humans".
+  An interval that clears 0.667 and runs over 0.800 gets the same form at
+  0.800, and a summary with no interval names no band. Judge labels with
+  alpha 0.801 and an interval of 0.601 to 0.951 printed "That is at or
+  above 0.800" in 0.3.0. Anything matching on the old sentences will stop
+  matching.
+
+- **Breaking.** `AgreementResult.summary()` no longer names a rater. 0.3.0
+  named one when dropping them raised alpha by more than half the width of
+  the interval on alpha. That compares the shift with the sampling error on
+  alpha, and the shift has a sampling error of its own that can be much
+  larger. Across simulated evals with three raters who differed only by
+  chance, the rule named a rater in 4.2% of 500 when everyone graded every
+  item. When the third rater graded a fifth of the items it named one in
+  32.0% of 500, always one of the two who graded everything. In one of those
+  evals 0.3.0 printed "Dropping r2 raises alpha to 0.878, a shift of 0.326
+  against a sampling error of 0.200." about a rater who graded all 60 items.
+  The same rule caught a rater wrong three times as often as the others in
+  72.4% of 500.
+
+  The summary now lists alpha with each rater left out, as in "Leaving one
+  rater out at a time gives alpha of 0.878 without r2 (+0.326), 0.524
+  without r3 (-0.028), and 0.483 without r1 (-0.069).", and says "The
+  package does not judge whether any of these shifts is larger than noise,
+  so it names no rater. A rater near the top of this list is a lead to check
+  rather than a finding." It lists five raters at most and points to
+  `rater_dropout` for the rest. `top_dropout_rater` always returns None.
+  `dropout_is_distinguishable` still computes the old rule, its docstring
+  says why the rule was withdrawn, and nothing in the package reads it.
+
+  The `audit` agreement finding carries the new sentence. Its severity never
+  came from this rule, so no finding changes severity. If you acted on a
+  rater a summary named, check that rater by hand.
+
+- **Breaking.** `PositionBias` puts the Clopper-Pearson interval, the exact
+  binomial interval, on the rate it tests against a half, in both designs,
+  and decides on it. The exact binomial p-value printed beside it comes from
+  the same test, so the two cannot disagree about 50%. 0.3.0 used a Wilson
+  interval in the randomised design. Wilson and the exact test disagree at
+  50% in 188 of 20,300 (n, k) cells up to 200 judgements, and in every one
+  the Wilson interval clears 50% while the p-value does not. At 11 of 14 in
+  the randomised design 0.3.0 printed "(95% CI: 52.4% to 92.4%, exact
+  binomial p=0.0574)" and then "The interval clears 50%, so the judge
+  favours whichever output it sees first." It now prints "(95% CI: 49.2% to
+  95.3%, exact binomial p=0.0574)" and "The interval includes 50%, so the
+  data cannot show that position moved the judge." Every randomised interval
+  moves, and a randomised verdict flips wherever the two used to disagree.
+
+  In the both-orders design 0.3.0 printed the share of flips with a bare
+  p-value, "(77.8%, exact binomial p=0.1797).", and decided on that p-value.
+  It now prints "(77.8%, 95% CI: 40.0% to 97.2%, exact binomial p=0.1797)."
+  and decides on the interval, which gives the same answer as the p-value in
+  every case. The consistency rate has no test beside it and keeps its
+  Wilson interval. Rerun any position check near the line.
+
+### Added
+
+- `BTResult.n_missing_item_ids`, the number of comparisons with no item id.
+  It is the last field of the dataclass and defaults to 0, so building a
+  `BTResult` by position with 0.3.0's fields still works. `to_elo` carries
+  it across.
+- `PositionBias.position_a_ci_low` and `PositionBias.position_a_ci_high`,
+  the Clopper-Pearson interval on `position_a_rate`. In the randomised
+  design they
+  equal `ci_low` and `ci_high`. They are the last two fields and default to
+  NaN, so building a `PositionBias` by position with 0.3.0's fields still
+  works. A both-orders result built that way has no interval, so its
+  `has_position_effect` is False.
+
+### Fixed
+
+- **Breaking.** `KappaResult.summary()` said "Krippendorff's alpha does not
+  have that problem and handles missing data, so prefer rater_agreement for
+  anything you report." Alpha has the problem too. Two raters who agree on
+  90 of 100 items score kappa 0.444 and alpha 0.447 when one category takes
+  90% of the ratings, and 0.800 and 0.801 when both are used equally often.
+  It now says "Krippendorff's alpha moves the same way, because it also
+  corrects for chance using how often each category is used." and gives
+  missing ratings, more than two raters and ordered scales as the reasons to
+  prefer `rater_agreement`. The `cohens_kappa` docstring made the same claim
+  and now quotes the four figures.
+
+- **Breaking.** `ScoreCI.summary()` told the reader to "treat differences
+  smaller than that as unresolved." A paired comparison resolves
+  differences far smaller than either score's own interval. The sentence
+  now reads "To compare this score with another, read the interval on the
+  difference, which compare_paired and compare_independent report. Two
+  score intervals that overlap do not show the systems are level." The
+  `scores` module docstring made the same claim and no longer does.
+
+- **Breaking.** `PowerResult.summary()` from `detectable_effect` said
+  "anything smaller was out of reach before the first item was graded". A
+  difference below the reach can still reach significance, only less often.
+  It now says "A smaller difference could still reach significance here,
+  with a chance below 80%." Where no difference reaches the requested power,
+  "This eval could not have found anything, whatever the two systems really
+  do." and, for independent samples, "and this eval could not have found
+  anything." are now "A null result from this eval says little about the
+  systems." The `detectable_effect` docstring said "An eval could not have
+  found anything smaller" and now says "No smaller difference reaches the
+  requested power".
+
+- **Breaking.** `JudgeValidation.summary()` on a single item said "Every
+  label that could be compared was identical" whatever the labels were,
+  beside a plain accuracy of 0.0% when they differed, and counted "1 items".
+  It now reads "(nominal, 1 item). One item cannot carry a reliability
+  estimate, so there is no number to report and no interval around it." The
+  count of items set aside reads "1 item was set aside" where it read "1
+  items were set aside".
+
+- **Breaking.** `BTResult.summary()` under `resample="comparisons"` said
+  "these 240 comparisons share 239 items" when one comparison had no item
+  id and no two shared anything. Sharing is now counted among the
+  comparisons that carry an id. When none share, the sentence is left out.
+  When some share and ids are missing, it reads "the 239 comparisons that
+  carry an item id share 26 items".
+
+- The README example and the prose under it show the new wording.
+
+### Removed
+
+- Nothing.
+
+### Known limitations
+
+This rule is unchanged since 0.3.0 and is not fixed in this release. The
+figures come from simulations run against 0.3.0 and against the code released
+as 0.4.0, which give the same results.
+
+- `JudgeValidation` names a slice, and `audit` raises a critical finding on
+  it, only when the slice's own interval sits wholly below the interval on
+  the overall figure. That reads a difference off two separate intervals
+  instead of an interval on the difference, and it is conservative. With
+  five slices of 60 items that differ only by chance, it named a slice in
+  0.5% of 200 simulated evals. It misses real differences. A slice where the
+  judge flipped 25% of the labels, against 10% in the other four, was named
+  in 32.5% of 200. When the summary says the data cannot show that the
+  judge does worse on any one slice, this rule is what did not show it.
+
+## [0.3.0] - 2026-09-10
+
+0.3.0 changes what `bradley_terry` reports on data you have already run, and
+it turns several calls that used to return numbers into errors. Each
+breaking entry says what it was, what it is now and what to do about it.
+
+### Changed
+
+- **Breaking.** `bradley_terry` reads separability off the interval on the
+  gap between two ratings. Before, a pair was separable when its two rating
+  intervals did not overlap. Now a pair is separable when the percentile
+  interval on the difference between the two ratings excludes zero. Two
+  rating intervals can overlap while the gap between them is well measured,
+  because both ratings carry the error of the field average they are
+  measured against and that error cancels in the gap. So the old test
+  undercounted the pairs the data orders, and the pairs it left out were
+  never shown to be unordered. There is no option to bring the old rule
+  back. `n_separable` and `separable_pairs` can change on data you have
+  already run. On the README example they go from 5 of 15 pairs to 7. On
+  twelve simulated leaderboards where every item carried one comparison,
+  the count went up on five and stayed the same on seven. Nothing changes
+  in your code. Rerun, and replace any separability count you have
+  published.
+
+- **Breaking.** `separable_pairs` has two new columns, `ci_low` and
+  `ci_high`, the interval on `difference`. They sit between `difference` and
+  `p_a_beats_b`, so code that reads columns by position will read the wrong
+  ones. Code that selects columns by name is unaffected.
+
+- **Breaking.** `bradley_terry` resamples items by default. Before, the
+  bootstrap drew single comparisons, which treats judgements of the same
+  prompt as independent. Now it draws item ids with replacement and takes
+  every comparison made on each drawn item.
+
+  Where every item carries one comparison the default draws the same
+  resamples from the same seed as 0.2.1, and the ratings, their intervals
+  and the win matrix come out identical. Separability can still change,
+  for the reason in the entry above.
+
+  Where items carry several comparisons the intervals widen and fewer pairs
+  may separate. On a simulated board of 60 items with 20 comparisons each,
+  the rating intervals came out 1.1 to 1.5 times as wide as 0.2.1's. On
+  simulated data shaped like MT-Bench's human judgements, the rating
+  intervals 0.2.1 reported covered 88% of the time at a nominal 95% once
+  prompts shifted model strength by a spread of 0.5 on the log-odds scale,
+  and 80% at a spread of 1.0. `examples/pairwise_cluster_study.py` has the
+  table for 0.3.0.
+
+  To get the ratings and rating intervals an earlier version gave on the
+  same data and seed, pass `resample="comparisons"`. They come out
+  identical to 0.2.1's. `separable_pairs`, `n_separable` and the summary
+  follow the 0.3.0 rules, so those can differ.
+
+- **Breaking.** `bradley_terry` raises `ValueError` in two cases that 0.2.1
+  accepted. Both apply only when `n_boot` is above zero and `resample` is
+  left at its default of `"items"`.
+
+  - Some row has no `item_id`. The item bootstrap has to know which item
+    each comparison belongs to.
+  - Every row has the same `item_id`, an empty string included. With one
+    item, every resample is the full data and every interval would have
+    width zero.
+
+  This is the change most likely to break a working script, so here is what
+  to do. Earlier versions required `item_id` and read it for nothing but a
+  count, so a constant, a blank or the row number got past the check and
+  changed nothing. Look at what yours holds.
+
+  - It names the prompt each comparison was made on. Fill in any gaps and
+    change nothing else. Your intervals now account for comparisons sharing
+    a prompt, which is the reason for the change.
+  - It is the row number, or anything else unique per row. Nothing raises,
+    and the ratings and their intervals come out as they did in 0.2.1.
+  - It is a constant or a blank. Record the prompt if you can. If you
+    cannot, pass `resample="comparisons"` to keep the intervals you had.
+    They come out identical to 0.2.1's on the same seed. They treat every
+    judgement as independent and run narrow wherever prompts carry several.
+    The summary says so when the column holds a constant or an empty
+    string. It says nothing when every `item_id` is missing, since then
+    nothing records which comparisons share a prompt.
+
+  With `n_boot=0` no resampling happens and any `item_id` is accepted.
+
+- **Breaking.** `score_ci(method="t")` raises `ValueError` on three kinds of
+  input it used to return numbers for. What to do depends on the input.
+
+  - 0/1 data, including all passes, all fails and a single pass or fail.
+    The interval could run past 100%, as 49 passes in 50 did at 94.0% to
+    102.0%, come back with width zero on all passes or all fails, or come
+    back as NaN on a single pass. Use `method="wilson"`, which
+    `method="auto"` already picks for 0/1 data and which works on a single
+    item.
+  - A single score that is not 0 or 1. It came back as NaN bounds behind two
+    RuntimeWarnings. One score has no spread, and the bootstrap returns
+    width zero on it too. Score more items.
+  - Continuous scores that are all the same. The interval came back with
+    width zero, which claims the observations fixed the mean exactly. Check
+    that the scorer is not returning a constant.
+
+- **Breaking.** `score_ci(method="wilson")` raises `ValueError` on scores
+  that are not 0 or 1. Wilson counts passes, and on other scores it took the
+  count by truncating the sum of the scores. Ten scores of 0.99 printed a
+  mean of 0.990 beside an interval of 0.596 to 0.982. For scores that vary,
+  use `method="bootstrap"`, the interval on a mean, which `method="auto"`
+  already picks. Scores that are all the same, like that example, get width
+  zero from the bootstrap too, and 0.3.0 does not refuse that.
+
+- **Breaking.** `score_ci`, `compare_paired` and `compare_independent`
+  raise `ValueError` for a confidence level that is not strictly between 0
+  and 1, with the message `confidence must be in (0, 1), got ...` that
+  `bradley_terry`, `rater_agreement`, `judge_validation` and `audit` already
+  used. Before, what came back at `confidence=1.5` depended on the method.
+  The bootstrap in all three raised from inside numpy. Wilson and both t
+  intervals returned NaN bounds. The two score intervals returned bounds
+  clipped to -100% and 100%. The signed-rank interval in `compare_paired`
+  returned an ordinary-looking interval under a "150% CI" label, which is
+  the case nothing would have flagged. Pass a level strictly between 0 and
+  1.
+
+- **Breaking.** Several summary strings are reworded, so any test or
+  pipeline that matches on them will fail.
+
+  - `BTResult.summary()` defines separable as "the interval on the gap
+    between the two ratings excludes zero". For pairs that do not separate
+    it says the data does not establish an order for them, that this does
+    not mean the models are level, and that more comparisons could separate
+    them. The phrases "meaning their intervals do not overlap", "cannot be
+    ordered", "an order the data does not carry" and "a ranking of noise"
+    are gone.
+  - `BTResult.summary()` adds a sentence when `resample="comparisons"` was
+    used on data where items carry several comparisons.
+  - `AgreementResult.summary()` for a scale nobody varied now reads "This
+    comes from a scale nobody varied. It is not perfect agreement." where it
+    read "That is not perfect agreement, it is a scale nobody varied."
+  - `JudgeValidation.summary()` for a rubric with one label now reads "This
+    comes from a rubric with one label in it. It is not perfect agreement."
+    where it read "That is not perfect agreement, it is a rubric with one
+    label in it."
+
+- The Development Status classifier moved from Alpha to Beta.
+
+### Added
+
+- `bradley_terry(..., resample="items")`, with `"comparisons"` as the other
+  choice.
+- `BTResult.pairs`, one row per pair of models with the gap, the interval on
+  the gap and a `separable` column. `separable_pairs` is its separable rows.
+- `BTResult.resample`, recording which unit the bootstrap drew. It and
+  `pairs` are the last two fields of the dataclass, so building a
+  `BTResult` by position with 0.2.1's fields still works.
+- `to_elo` stretches the gap intervals with the gaps and carries `pairs` and
+  `resample` across.
+- `examples/pairwise_cluster_study.py`, in the source distribution, which
+  measures how narrow the comparison bootstrap runs on MT-Bench-shaped data.
+
+### Fixed
+
+- The `scores` module docstring said the interval at n=50 is roughly plus or
+  minus 10 points. It is about 10 at an 85% pass rate and about 13 at 50%.
+- The README said every interval in the package is tested by simulation.
+  Several were not. It now says which intervals are tested by simulation,
+  which are checked against a reference implementation instead, and which
+  one is checked by neither.
+- The README said `audit` runs five checks. It runs seven, drawn from five
+  modules.
+
+### Removed
+
+- `AUDIT.md`, an internal review of the package, is no longer tracked, so
+  it is not in the source distribution.
+
+## [0.2.1] - 2026-09-08
+
+Reconstructed after the fact. The published files match commit `809a650`.
+
+### Changed
+
+- `__version__` is read from the installed package metadata. 0.0.1 and
+  0.2.0 both reported `__version__` as 0.1.0, because a literal in
+  `__init__.py` had drifted from `pyproject.toml`.
+
+### Added
+
+- `AUDIT.md`, an internal review of the package, in the source
+  distribution.
+
+### Fixed
+
+- **Breaking.** The length-bias check read every coefficient as a
+  preference for long answers. In 0.2.0 a judge that preferred shorter
+  answers got the `audit` title "The judge is pulled by how long the answer
+  is", and `LengthBias.summary()` printed "A positive coefficient here is
+  not bias on its own" beside a negative coefficient. The `audit` finding
+  now takes its title, lead, action and verdict from the fit that decides,
+  with its own lead for when the two fits point opposite ways, and
+  `LengthBias.summary()` has its own sentence for a negative coefficient.
+  Anything matching on the old title or sentence will stop matching.
+
+### Removed
+
+- Nothing.
+
+## [0.2.0] - 2026-09-07
+
+Reconstructed after the fact. The wheel matches commit `242e354`. It was
+uploaded twelve seconds before that commit was made, so it was built from
+the working tree the commit then recorded.
+
+### Changed
+
+- The project URLs point at github.com/mokayahezron/evalaudit. 0.1.0 pointed
+  at github.com/ondibahezron-glitch/evalaudit.
+- `choix` is a development dependency on Python 3.10 and later, used to
+  check the Bradley-Terry fit.
+
+### Added
+
+- The `pairwise` module, with `bradley_terry`, `to_elo` and `BTResult`.
+  Ratings come with bootstrap intervals and a count of separable pairs.
+  Models that fall into groups that never met, and models that never lost
+  or never won, get no ratings and a summary naming them. Ties are split or
+  dropped, and `ties="davidson"` is refused by name.
+- The README code blocks are run by the test suite and checked against the
+  output they show.
+
+### Fixed
+
+- Nothing.
+
+### Removed
+
+- Nothing.
+
+## [0.1.0] - 2026-09-07
+
+Reconstructed after the fact. The wheel matches commit `c1230f1`.
+
+### Changed
+
+- Nothing.
+
+### Added
+
+- `compare_paired` and `compare_independent`. McNemar with Tango's score
+  interval for paired binary data, the paired bootstrap, and the signed-rank
+  test with the Hodges-Lehmann interval. The score interval, the bootstrap
+  and Welch's t for independent samples.
+- `rater_agreement`, Krippendorff's alpha with a bootstrap interval, an item
+  disagreement table and a rater dropout table. `cohens_kappa` and
+  `fleiss_kappa`.
+- `judge_validation`, `position_bias` and `length_bias`.
+- `detectable_effect` and `min_sample_size`.
+- `audit`, which runs every check the supplied data supports and ranks the
+  findings by severity.
+- The result types are exported from the package, `ScoreCI` among them.
+
+### Fixed
+
+- Nothing.
+
+### Removed
+
+- Nothing.
+
+## [0.0.1] - 2026-09-04
+
+Reconstructed after the fact. The wheel's package files match commits
+`aa8ad97`, `8a4cab0` and `b090c66`, which do not differ in them. Its version
+number first appears in `pyproject.toml` at `b090c66`, committed about an
+hour after the upload. It reported `__version__` as 0.1.0.
+
+### Changed
+
+- Nothing.
+
+### Added
+
+- `score_ci`, with Wilson, bootstrap and Student-t intervals, and a default
+  that picks Wilson for 0/1 data and the bootstrap otherwise.
+
+### Fixed
+
+- Nothing.
+
+### Removed
+
+- Nothing.
+
+[Unreleased]: https://github.com/mokayahezron/evalaudit/compare/v0.5.0...main
+[0.5.0]: https://github.com/mokayahezron/evalaudit/compare/v0.4.0...v0.5.0
+[0.4.0]: https://github.com/mokayahezron/evalaudit/compare/v0.3.0...v0.4.0
+[0.3.0]: https://github.com/mokayahezron/evalaudit/compare/809a650...v0.3.0
+[0.2.1]: https://github.com/mokayahezron/evalaudit/compare/242e354...809a650
+[0.2.0]: https://github.com/mokayahezron/evalaudit/compare/c1230f1...242e354
+[0.1.0]: https://github.com/mokayahezron/evalaudit/compare/b090c66...c1230f1
+[0.0.1]: https://github.com/mokayahezron/evalaudit/tree/b090c66