accuracy-claim-kit 0.0.0-stage → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,3 +1,473 @@
1
- # Temporary Holding Version
1
+ # Accuracy Claim Kit
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ Does this accuracy or error-rate number have the evidence to carry it? You give the kit the counts behind a published figure, who labeled the items and how they were chosen. It says whether those counts can carry the figure, and writes a cautious statement the evidence supports, as plain text you can publish instead.
4
+
5
+ It is for people who publish a figure from their own evaluation: an AI-eval author, a team with a model or detector in production, a maintainer writing a benchmark README or a methodology page. Most real claims fail on a missing declaration (nobody wrote down who labeled, how items were picked, or what was measured), so expect `refused` often and read it as "write this down", not "false".
6
+
7
+ The kit checks evidence. It does not certify that a figure is true or that a system is accurate, and it makes no legal or regulatory determination. A `supported` verdict means the claim is no more favorable than the counts you supplied, or is an honest report of the rate you observed. See "What it does not do" and [METHOD.md](https://github.com/lkopietz3-byte/accuracy-claim-kit/blob/main/METHOD.md), which gives the reason for every rule and default.
8
+
9
+ Zero runtime dependencies, TypeScript types included, ESM, Node 20 or newer. It reads no clock and uses no randomness: the same input gives the same output (dates are passed in, never read).
10
+
11
+ **Reproducible benchmark.** From a clone, `npm run bench` (it needs `python3`) rebuilds a test set from 216 percentages printed with counts in 75 public FDA 510(k) summaries, runs the kit over it and checks 103 hand-labeled claims from other sources, and it fails if a pinned number moves. The oracle that judges the first set shares an author with the kit, so read it as "the kit does what it documents". Numbers and limits: [Agreement with an independent oracle and a hand-labeled holdout](#agreement-with-an-independent-oracle-and-a-hand-labeled-holdout).
12
+
13
+ ## Install
14
+
15
+ ```sh
16
+ npm i accuracy-claim-kit
17
+ ```
18
+
19
+ ESM only: `import` it. A CommonJS file can `require()` it on Node 20.19 or newer and 22.12 or newer (the releases that can require an ES module); on older Node 20 use `await import('accuracy-claim-kit')`.
20
+
21
+ ## Quick start
22
+
23
+ The same evidence (990 of 1000 right, labeled by people, drawn at random) against three claims. Save this as `check.mjs` and run `node check.mjs`:
24
+
25
+ ```js
26
+ import { assessAccuracyClaim } from 'accuracy-claim-kit'
27
+
28
+ const evidence = { sample: { n: 1000, correct: 990 }, provenance: { source: 'human' }, sampling: 'random' }
29
+
30
+ for (const [display, value, more] of [['98%', 0.98, {}], ['99.5%', 0.995, {}], ['98%', 0.98, { provenance: { source: 'model' } }]]) {
31
+ const r = assessAccuracyClaim({ claim: { metric: 'accuracy', value, display }, ...evidence, ...more })
32
+ console.log(display, r.verdict, r.reasons.join(', '))
33
+ }
34
+ ```
35
+
36
+ It prints:
37
+
38
+ ```
39
+ 98% supported claim-within-bound
40
+ 99.5% contradicted claim-outside-interval, precision-too-fine
41
+ 98% refused no-human-labels
42
+ ```
43
+
44
+ - "98%" is `supported`: the lower end of the 95% interval is 98.16% (the kit rounds a lower limit down, so it never prints more than the interval holds), so 98% is no more than the evidence shows.
45
+ - "99.5%" is `contradicted`: the interval tops out at 99.46%, so 99.5% is more than the counts allow.
46
+ - The third has the same counts with labels from a model. It is `refused`: a model's labels are not ground truth, so no figure can be judged.
47
+ - A figure between the lower end and the observed rate is not `supported`, and the verdicts are not a straight line in the claimed figure: 98% passes (at or below the lower end), 98.5% does not, 99.0% passes (it is the observed rate) and 99.5% is `contradicted`. 98.5% is `overstated`: it is below the observed rate (99.0%) but above the lower end (98.16%), so the counts neither rule it out nor carry it. The explanation says so, and the way out is to print the observed rate or go down to the lower end. An honest round-down, such as 99.6% printed as "99%", lands in the same place.
48
+
49
+ `refused` means "cannot tell", not "false": it lists what to write down (who labeled, how items were picked, how many) so a figure can be judged. Every result has `reasons`, one plain sentence each in `explanation`, and a `suggestedStatement` you can publish instead of the claim.
50
+
51
+ ## TypeScript
52
+
53
+ The types ship with the package. Annotate the input as `AccuracyClaimInput`, and hoist shared evidence into a const with `satisfies`: a plain `const evidence = { provenance: { source: 'human' }, ... }` widens `'human'` to `string` (literal widening) and fails to compile, because `source` must be one of the label sources.
54
+
55
+ ```ts
56
+ import { assessAccuracyClaim } from 'accuracy-claim-kit'
57
+ import type { AccuracyClaimInput } from 'accuracy-claim-kit'
58
+
59
+ const evidence = {
60
+ sample: { n: 1000, correct: 990 },
61
+ provenance: { source: 'human' },
62
+ sampling: 'random',
63
+ } satisfies Pick<AccuracyClaimInput, 'sample' | 'provenance' | 'sampling'>
64
+
65
+ const input: AccuracyClaimInput = { claim: { metric: 'accuracy', value: 0.98, display: '98%' }, ...evidence }
66
+ console.log(assessAccuracyClaim(input).verdict) // supported
67
+ ```
68
+
69
+ Every public type is exported from the package root (`AccuracyAssessment`, `ReasonCode`, `UnsupportedMetric` and the rest). The package is ESM only. With `"module": "node16"`, a TypeScript file that compiles to CommonJS (a `.ts` file in a package without `"type": "module"`, or a `.cts` file) gets TS1479 on `import ... from 'accuracy-claim-kit'`, and TS1541 on a type-only import. Make the file ESM (`"type": "module"` in your `package.json`, or the `.mts` extension) or load the package with `await import('accuracy-claim-kit')`. `"module": "nodenext"` accepts the import in TypeScript 5.9, the version checked.
70
+
71
+ ## Agreement with an independent oracle and a hand-labeled holdout
72
+
73
+ How often does the kit agree with a check that is not itself? Two re-runnable checks answer this, and neither is a sample of all accuracy claims. From a clone of the repository, `npm run bench` reproduces every number below (it needs `python3`; `bench/` is not in the npm package). They were measured on the 0.1.0 source at revision `373258b` of the repository; later commits also changed `src/` (an absolute 100% or 0% in `claim.text` is flagged against a rounded value, two explanations were reworded, the Wilson upper limit stays below 1 at about 4.5e15 items, the text scan reads the Arabic percent sign, and a type is exported; `git diff 373258b HEAD -- src` lists them). No benchmark verdict changed: `bench/expected.json` pins every number below, and `npm run bench` reproduces them on the release head.
74
+
75
+ **Field benchmark: 216 printed rates with counts in 75 public FDA 510(k) summaries.** Each printed figure was altered in nine ways (rounded up, plus three points, claimed as 100%, five points under, a floor taken from the printed interval, the complement error rate, and error ceilings), and every claim was judged by `bench/oracle.py`, a second implementation of the rules in this README.
76
+
77
+ - Of the 957 altered claims the oracle says the counts do not carry, the kit passed 0 as `supported`.
78
+ - Of the 726 claims the oracle says the counts carry, the kit said `supported` for 726.
79
+ - The kit's verdict equals the oracle's on 2,001 of 2,001 judged claims (also under `strictPrecision` and `conservative`), 216 of 216 with each document's declared sampling and label source, 3,498 of 3,498 as claims about a whole fixed set, and 95 of 95 pooled site tables.
80
+
81
+ The oracle restates this README's rules and shares an author with the kit, so agreement shows the kit does what it documents, not that the rules are right. The FDA corpus is one document family, and the altered claims are built by fixed rules, so 0 of 957 is not the rate on claims people actually write.
82
+
83
+ **Holdout: 103 public claims from other sources.** 103 claims with printed counts, from 30 open-access papers and one vendor benchmark page, none from the FDA corpus. Each was labeled by hand from the rules before the kit ran on it. The kit's verdict equals the label on 103 of 103 (73 supported, 24 refused, 4 contradicted, 2 overstated). Five of the six contradicted or overstated rows are printed percentages that are not the rate their own counts give (four truncated instead of rounded, one mistyped). 71 of the 103 are scores over a fixed set, so the interval path is exercised by only 6 claims.
84
+
85
+ **Cross-check of the arithmetic.** Wilson intervals (8,899 cases), z, percent agreement, Cohen's kappa and Gwet's AC1 (1,500 two-rater tables) match statsmodels, scipy, scikit-learn and irrCAC to a largest difference of 4.9e-15 for Wilson and 6.7e-14 for kappa. See "Cross-checked against reference libraries".
86
+
87
+ **What these numbers do not show.** They do not show that the policy defaults are right (the minimum sample of 30, the precision rule and the strata check are choices; METHOD.md lists each with its measured rate). They do not show how often declared labels and sampling are true, because the kit takes declarations at your word. A `supported` verdict on an honest report of the observed rate is not a claim about the true rate: in a simulation, the true rate was below an honest whole-percent report in 41% of samples (3,645 of 9,000), which is why the statement always quotes the interval.
88
+
89
+ ## What it checks
90
+
91
+ A claim like "99.8% accurate" or "0.2% error" is only as good as the evidence under it. The kit refuses to judge when any of these hold, and lists every one that applies:
92
+
93
+ | Reason code | Refused because |
94
+ | --- | --- |
95
+ | `metric-not-supported` | The metric is one this kit cannot judge (F1, AUC, WER, BLEU, composite scores). See "Metrics". |
96
+ | `n-undisclosed` | The sample counts were not supplied at all. |
97
+ | `no-human-labels` | The labels came from a model, so there is no human ground truth. (Use `automated` for a deterministic procedure with known answers, such as a test suite or an exact search, and `by-construction` for ground truth fixed by how the data was made, such as text from a known generator or labels taken from a corpus's language. Both are declarations the kit cannot verify and are accepted like `human`; never use them for a model's judgment.) |
98
+ | `unknown-label-source` | Nobody can say where the labels came from. |
99
+ | `mixed-label-source` | Some labels are from a model and the counts are not split. Assess only the human-labeled items. |
100
+ | `ranked-sample` | The sample was picked by rank (for example the top-scored items). |
101
+ | `convenience-sample` | The sample is whatever was easy to collect. |
102
+ | `unknown-sampling` | Nobody can say how the sample was picked. |
103
+ | `sample-too-small` | `n` is below `options.minN` (default 30). |
104
+ | `merged-strata-differ` | Strata were given, and two of them have intervals that do not overlap, so one combined number hides a real difference. |
105
+ | `stratified-without-strata` | `sampling` is `stratified` but no `strata` were supplied, so the pooled counts are not the population rate. |
106
+ | `claim-not-scoped-to-benchmark`, `benchmark-unnamed` | A `complete` (whole fixed set) claim whose wording does not name the benchmark, or that names none. See "Claims about a fixed benchmark". |
107
+ | `evaluation-stale`, `evaluation-date-missing` | `options.maxAgeDays` is set and the evaluation is older than that, or has no date. |
108
+ | `scope-mismatch`, `scope-undeclared` | The claimed population differs from the evaluated one, or (with `options.requireScope`) neither was declared. |
109
+ | `retractions-not-included` | Retracted items are known (`count > 0`) but are not in the counts. |
110
+ | `agreement-missing`, `agreement-undefined`, `agreement-below-minimum` | `options.minAgreement` is set and the supplied agreement statistic is absent, undefined, or lower than the minimum. |
111
+ | `agreement-sample-too-small` | The supplied agreement statistic carries an `n` below `options.minN`: a kappa of 1.0 from 6 items is noise. A hand-written `{ statistic, value }` with no `n` cannot be checked and is trusted. |
112
+
113
+ If nothing blocks it, the claim is compared with the Wilson interval (or, for a whole fixed set, with the exact count):
114
+
115
+ | Verdict | Higher-is-better claim (accuracy, precision, recall) | Lower-is-better claim (error rate, FPR, FNR) |
116
+ | --- | --- | --- |
117
+ | `contradicted` | above the top of the interval | below the bottom of the interval |
118
+ | `overstated` | inside the interval, above the lower end, and not the observed rate to its printed precision: a figure more favorable than the observed rate, or a more cautious one that is still above the lower end (98.5% for 990 of 1000, whose lower end is 98.16%; say the observed 99.0% or go down to the lower end); or (only with `strictPrecision`) printed with more decimals than the interval supports | the mirror image |
119
+ | `supported` | at or below the lower end (`claim-within-bound`), **or** equal to the observed rate to its printed precision (`matches-observed-rate`) | at or above the upper end, or equal to the observed rate |
120
+
121
+ An honest report of what was measured ("95%" after 950 of 1000) is `supported` with `matches-observed-rate`, and the suggested statement still gives the interval. `matches-observed-rate` says the figure is what was observed; it does not say the observation is precise. 29 of 30 right claimed as "97%" is `supported` with an interval of 83.3% to 99.4%, so read `interval` (or the suggested statement, which quotes it) for how much the counts can carry; the reason `claim-within-bound` is the one that says the claim sits at or below the conservative end. This check comes before `contradicted`, so a rounded figure that lands just outside the interval it was rounded from (98.9% printed as "99%" on 98,900 of 100,000) is still an honest report. It does not ride this allowance if it is a bound ("99%+", "<1%"), or is `0` or `1` (`0%` / `100%`: a sample cannot show certainty. The Wilson interval itself ends at 100% when every item was right, but its conservative lower limit never establishes 100%, so such a claim is `overstated` and the explanation says so, then gives the bound that can be said: 290 of 290 right supports "at least 98.69%"). An exact tie accepts both neighbors: when the observed rate sits exactly half a unit from the last printed digit (197 of 200 is 98.5%, 42 errors in 48 is 87.5%, 120 of 128 is 93.75%), both rounded figures ("98%" and "99%", "87%" and "88%", "93.7%" and "93.8%") equal it to their printed precision and are `supported`, because the kit does not pick a rounding rule (half up, half even and truncation all appear in published figures). For a lower-is-better metric that includes the neighbor half a unit better than observed. Printing finer than the interval supports is noted (`precision-too-fine`) but does not change a `supported` verdict unless `strictPrecision` or `conservative` is set. Pass `options.conservative: true` for the strict rule, where only a claim at or below the conservative bound is supported.
122
+
123
+ Three informational codes never change a verdict. The first, `claim-worse-than-counts`, is added when a figure is far *less* favorable than the counts support (outside the 99.9% interval on the pessimistic side). That pattern is what a stale or mistyped number looks like (a published 44% error rate against 56 errors in 189 items, which is 29.6%), so the explanation says it may be stale or wrong. The verdict stays `supported`, because an understated claim is not an overstatement.
124
+
125
+ The second, `strata-not-reweighted`, is added to every stratified result: the interval pools the strata as one simple random sample and is not reweighted by population share (the kit has no shares to reweight with), and the statement says "stratified sample, unweighted pooled". If the strata were sampled at different rates, report per stratum. The third, `text-figure-differs`: `claim.text` is judged for the benchmark name (complete sets) and otherwise only scanned: any percent, range, bound or `1 in N` in it that describes neither `value` nor the printed `display` (nor their complement, so "87.7% right" can sit beside a 12.3% error rate) adds the flag, and so does "perfect", "flawless", "error-free", "no errors" or "zero errors" when the figure is not exactly 100% (0% for an error rate). A figure that is a rounding of `value` ("80%" for 79.9%) is the same figure; one with finer digits is not. A printed 100% or 0% is the exception: it is the same figure only for a value of exactly 1 or 0, so "100% accurate" beside 99.6% is flagged (on a complete set as well). Written as approximate, "about 100%" and "~100%" keep their rounding band and are not flagged, because they do not claim certainty. "99.9 percent" reads as "99.9%". The scan cannot tell a figure about something else ("up from 70%") from the claim, so it flags those too. It reads only a percent (the Arabic percent sign included), a range, a bound or `1 in N`. A range is one figure judged at its favorable end, so "improved from 70% to 95%" reads as the range 70-95% and is not flagged beside a 95% claim; a proportion such as `0.999` and a spelled-out number such as "ninety-nine percent" are not read at all. The verdict never changes, and the sentence you publish is still yours to check.
126
+
127
+ The "conservative bound" is the end of the interval a claim should not beat: the lower end for higher-is-better metrics, the upper end for lower-is-better ones. A claim at or below it (say "90%" when the lower end is 98%) is supported; one between it and the observed rate is not (see `overstated` above).
128
+
129
+ ## Worked example
130
+
131
+ 499 of 500 items were right, labeled by two people, drawn at random. The claim is "99.8% accurate".
132
+
133
+ ```js
134
+ import { assessAccuracyClaim } from 'accuracy-claim-kit'
135
+
136
+ const result = assessAccuracyClaim({
137
+ claim: { metric: 'accuracy', value: 0.998, display: '99.8%' },
138
+ sample: { n: 500, correct: 499 },
139
+ provenance: { source: 'human', raters: 2 },
140
+ sampling: 'random',
141
+ })
142
+
143
+ result.verdict // 'supported'
144
+ result.reasons // ['matches-observed-rate', 'precision-too-fine']
145
+ result.explanation[1]
146
+ // 'The figure is printed to 1 decimal place, but the interval is 1.09 percentage points wide and supports whole percents only.'
147
+ result.interval // { low: 0.98875..., high: 0.99964..., point: 0.998, confidence: 0.95, z: 1.95996..., successes: 499, n: 500 }
148
+ result.suggestedStatement
149
+ // 'On this sample, at least 98.87% accurate (95% Wilson interval 98.87% to 99.97%, n = 500, human labels, random sample); the interval covers sampling error only; print it as a whole percent'
150
+ ```
151
+
152
+ The figure equals the observed rate (499 of 500), so it is an honest report and is `supported`. Printed to a tenth of a percent it implies more precision than an interval 1.09 points wide supports (the data cannot rule out a true rate as low as 98.87%), so the result says to print it to whole percents, and with `strictPrecision` that alone makes it `overstated`. Printed as "99%" the claim would be more cautious than the observed 99.8% but still above the lower end of the interval (98.87%), which is `overstated` with `claim-beyond-conservative-bound`: say the observed 99.8%, or go down to the lower end. With `strictPrecision` neither 99.8% (printed too fine) nor 100% (contradicted) passes, so the explanation advises only the lower end at whole percents (98%), and the advice never names a figure the same assessment would reject. The suggested statement starts with its scope ("On this sample"), says the interval covers sampling error only, describes the evidence and rounds the lower end down (98.8759 prints as 98.87, never 98.88). The headline bound and the interval it quotes always use the same precision.
153
+
154
+ Other results from the same style of input:
155
+
156
+ - 990 of 1000 right, claimed "98%": `supported`. The lower end is 98.16%.
157
+ - 900 of 1000 right, claimed 95%: `contradicted`. The top of the interval is 91.71%.
158
+ - A 0.2% error claim where a model did the labeling: `refused`, reason `no-human-labels`, no interval returned.
159
+ - 980 of 1000 right in one stratum and 10 of 30 in another, claimed 95% overall: `refused`, reason `merged-strata-differ`.
160
+
161
+ ## Metrics
162
+
163
+ Each metric is a proportion with a stated meaning of `sample.n`:
164
+
165
+ | `claim.metric` | `n` counts | give | better when |
166
+ | --- | --- | --- | --- |
167
+ | `accuracy` | items checked | `correct` | higher |
168
+ | `error-rate` | items checked | `errors` | lower |
169
+ | `precision` | predicted positives | `correct` (true positives) | higher |
170
+ | `recall` | actual positives | `correct` (found) | higher |
171
+ | `false-positive-rate` | actual negatives | `errors` (false positives) | lower |
172
+ | `false-negative-rate` | actual positives | `errors` (misses) | lower |
173
+ | `specificity` | actual negatives | `correct` (true negatives) | higher |
174
+ | `npv` | predicted negatives | `correct` (true negatives) | higher |
175
+
176
+ Diagnostic and agreement tables use other names for the same proportions. Each is accepted, judged exactly as the metric it stands for, and the statement uses the claimant's word instead of "accurate":
177
+
178
+ | Name | Same as | The statement says |
179
+ | --- | --- | --- |
180
+ | `sensitivity`, `ppa` | `recall` (n = actual positives, `correct` = found) | sensitivity, positive percent agreement |
181
+ | `npa` | `specificity` (n = actual negatives, `correct` = true negatives) | negative percent agreement |
182
+ | `ppv` | `precision` (n = predicted positives) | positive predictive value |
183
+ | `opa` | `accuracy` (n = items compared with the comparator, `correct` = items that agree) | overall percent agreement |
184
+
185
+ A published "Negative Agreement 82.7% (755/913)" is `metric: 'npa'` (or `'specificity'`) with `n: 913, correct: 755`; encoding it as `accuracy` would make the statement call a negatives-only figure "accurate". The kit cannot tell which population `n` counts: the metric name is your declaration, as with `accuracy`. Names are exact and lower case; `Specificity` or `sens` throws `RangeError` and lists the valid names.
186
+
187
+ `correct` or `errors` works for every metric (give either; the other is `n` minus it), so an error-rate claim can be counted by `correct` and an accuracy claim by `errors`.
188
+
189
+ `f1`, `auc`, `roc-auc`, `wer`, `cer`, `bleu`, `rouge`, `map`, `composite` and `other` are not proportions of independently checked items; they come back `refused` with `metric-not-supported` (their `value` is only required to be a number, since a word error rate can pass 1). The refused names are recognized in any case at run time (`F1`, `AUC`, `WER` come back refused); the TypeScript types accept the lower-case, upper-case and capitalized forms (`f1`, `F1`, `Auc`) and not a mixed case such as `rOc-AuC`. A judged name must be written exactly (lower case); a metric name the kit does not know at all (a typo, `Accuracy`, `FPR`) throws `RangeError` whose message lists both groups of names and, when only the case is wrong, says which name was meant. A display that disagrees with `value` throws `RangeError` naming both figures and the allowance.
190
+
191
+ If the source does not give counts, leave `sample` out (or give an object with no `n`, `correct` or `errors`): the result is `refused` with `n-undisclosed` plus any other failed precondition, never a throw. Strata with no pooled `sample` are refused the same way, and the explanation gives their sums (`sample: { n: 1000, correct: 950 }`) to pass with them. Counts that are half given, fractional or inconsistent still throw, and so does a misspelled key (`{ total: 1000, right: 900 }` throws "unknown key"; it is never read as "no counts"). Any object with none of those three, including a `Map` or a `Number` object, is an object with no counts: the result is refused, which is the safe direction. `options` must be a plain object, for `assessAccuracyClaim`, `wilsonInterval` and `gwetAC1` alike: a `Map`, `Set`, boxed primitive or `Date` there would drop every setting silently, so it throws `TypeError`; class instances, null-prototype objects and objects from another realm are fine.
192
+
193
+ ## Printed forms of a figure
194
+
195
+ `display` is read into the figure a reader takes away, using the favorable reading, and the result records the parsed form in `parsedDisplay` (`form`, `figure`, `decimals`):
196
+
197
+ | Printed | Form | Judged as |
198
+ | --- | --- | --- |
199
+ | `99.8%` | `percent` | the point figure, with the allowance and precision rule |
200
+ | `~95%`, `about 95%`, `approximately`, `around`, `roughly`, `≈95%` | `approx` | the same as the plain figure (the stated precision is kept) |
201
+ | `99%+`, `99+%`, `at least 99%`, `≥99%`, `>=99%`, `greater than or equal to 99%`, `not less than`, `no less than`, `minimum of`, `99% or higher` (also `or more`, `or above`, `or greater`) | `floor` (inclusive) | "at least 99%" as a bound: checked against the conservative end, never excused by matching the observed rate. Higher-is-better metrics only |
202
+ | `over 99%`, `above`, `more than`, `greater than`, `>99%` | `floor` (strict) | "more than 99%": the bound itself does not satisfy it (on a complete set, exactly 99% fails; on an interval, the conservative end must clear it) |
203
+ | `no more than 1%`, `at most`, `up to`, `≤1%`, `<=1%`, `less than or equal to 1%`, `not more than`, `maximum of`, `1% or lower` (also `or less`, `or below`) | `ceiling` (inclusive) | "at most 1%" as a bound. Lower-is-better metrics only |
204
+ | `<1%`, `under 1%`, `below`, `less than`, `fewer than` | `ceiling` (strict) | "less than 1%": exactly 1% fails |
205
+ | `95-98%`, `95 to 98%`, `95–98%` | `range` | the favorable end (98% for accuracy, 95% for an error rate) as a point figure, on every path (a complete set is judged at that end too, not as a span); `value` must lie inside the range exactly, with one ulp of representation error allowed at each end |
206
+ | `1 in 10,000` | `one-in-n` | 1/N with N exact, to plus or minus one half (`1 in 100` is 1 in 99.5 to 1 in 100.5; a true 1.5% is not `1 in 100`) |
207
+ | `fewer than 1 in 1,000`, `less than`, `under`, `below`, `<1 in 1,000`, `at most 1 in 1,000` | `ceiling` | a bound at 1/N, judged as the percent bounds are (the value must sit at 1/N, N to plus or minus one half). `at least` and `more than 1 in N` fit higher-is-better metrics. A bound written after `1 in N` (`1 in 100 or lower`) is ambiguous and throws |
208
+ | `about 1 in 10,000`, `~1 in 100`, `around`, `roughly`, `approximately`, `≈` | `one-in-n` | 1/N with the trailing zeros of N as placeholders: `about 1 in 100` is 1 in 50 to 1 in 150, `about 1 in 10,000` is 5,000 to 15,000 |
209
+
210
+ A bound is judged exactly. One copied from the end of a published interval is usually rounded to the nearest digit, which can land just past the real limit (a lower bound of `99.1%+` on 589 of 590 against a limit of 99.046%); the result is `overstated`, and when the shortfall is less than one unit of the last printed digit the explanation gives the figure to print instead, rounded down for a floor and up for a ceiling (`99.0%`).
211
+
212
+ A bound pointing the wrong way for the metric ("<99%" accuracy), an impossible one ("over 100%", "under 0%"), an inverted range, a figure above 100% and any other text throw `RangeError`. Words before or after the figure that are not in the table (`99.8% accurate`, `a score of 99%`), a decimal comma (`99,8%`) and a written-out `percent` are not read: clean them first, as the kit never guesses at a figure. On a complete set, every figure the reader can see is judged by its form, and a `value` more favorable than the printed figure is judged as its own point figure (a value of 0.994 printed as `99%+` on 990 of 1000 is contradicted). The decimals rule skips bounds and "1 in N". Case, spacing, full-width forms and one trailing period are tolerated.
213
+
214
+ ## Sampling and label sources
215
+
216
+ - `sampling: 'held-out-split'` is a fixed, named test split. It follows the complete-set path below (`benchmark.name` is the split's name, and the claim text must mention it); otherwise it is refused as `benchmark-unnamed` or `claim-not-scoped-to-benchmark`. The statement says "held-out split".
217
+ - `sampling: 'cross-validation'` is judged like a random sample and the statement says "cross-validation estimate". Caveat: the folds share data, so the items are not independent and the interval is optimistic.
218
+ - `provenance.source: 'by-construction'`: see the label table above.
219
+
220
+ ## Claims about a fixed benchmark
221
+
222
+ Many real claims are not estimates from a sample: they are a score over the whole of a fixed, named set (a published validation set, a full test split). There is nothing to infer, so an interval is the wrong tool. Declare it with `sampling: 'complete'`, the set's name, and claim wording that names it:
223
+
224
+ ```js
225
+ assessAccuracyClaim({
226
+ claim: { metric: 'accuracy', value: 0.799, display: '79.9%', text: 'Reaches 79.9% top-1 accuracy on the Ridgeview validation set' },
227
+ sample: { n: 50000, correct: 39950 },
228
+ provenance: { source: 'human' },
229
+ sampling: 'complete',
230
+ benchmark: { name: 'Ridgeview validation set' },
231
+ })
232
+ // verdict 'supported', reasons ['census-matches-count'], basis 'census', interval null
233
+ // suggestedStatement: '39950 of 50000 correct on Ridgeview validation set (complete set, human labels): 79.90% accuracy. This describes that set only; it is not an estimate for other data'
234
+ ```
235
+
236
+ What is checked for a complete set:
237
+
238
+ - **The figure against the exact count**, in exact arithmetic, to the precision it is printed at (`79.9%` matches 39950 of 50000; `80%` also matches, as a rounding; `81%` does not). A figure that does not match is `contradicted` (`claim-differs-from-count`) in either direction, which catches stale and mistyped numbers. Every figure the reader can see must match: the `value` and, when given, the `display` (a value of 0.9849 printed as `98%` on 976 of 1000 is contradicted). Figures in `claim.text` are not judged; a different one is flagged (`text-figure-differs`). A `100%` or `0%` in the text counts as different from 99.6% or 0.4%, since those figures are absolute. A `value` is read as the shortest decimal that is within about 9e-16 (2^-50) of it (`5e-7` is exact to the digit, not to binary noise; `82.7 / 100`, which is 0.8270000000000001 in floating point, reads as 0.827, and `1 - 800 / 1000`, which is 0.19999999999999996, reads as 0.2), and never coarser than a whole percent: a bare `0.9` means 90% to within half a point, not 90 plus or minus 5 points. That allowance lets a value computed as `correct / n`, `1 - accuracy`, `pct / 100` or a mean of folds match its own count. A value within it of 0 or 1 is read as 0 or 1 (ten `0.1`s added up is 100%, an absolute claim), including one that overshoots by that much (`1.0000000000000002`, `-2.8e-17`); further outside 0 to 1 it throws `RangeError`. `0` and `1`, and `0%` and `100%` displays, are absolute claims and are never rounded: `100%` matches only `n` of `n`.
239
+ - **Precision**: digits beyond what `n` resolves (more than `d` percent decimals when `10^(d+2)` exceeds `10 n`) are `precision-too-fine`.
240
+ - **Scope wording**: `claim.text` must mention `benchmark.name`.
241
+ - Matching: the text and the name are compared after Unicode compatibility folding, lower-casing and collapsing whitespace; curly quotes and apostrophes fold to `'` and `"`, every dash variant and the minus sign fold to `-`, and every invisible character (soft hyphen, zero-width space and joiners, word joiner, direction marks, BOM, variation selectors, tag characters) is dropped, so text copied from the web still matches and an invisible character cannot hide a longer name (`MMLU`, zero-width space, `-Pro`); look-alike letters from other scripts are not folded.
242
+ - The name must contain at least one letter or digit (`.`, `%` and invisible characters are rejected with `RangeError`). Names, population identifiers and stratum names are echoed into explanations and statements, and `claim.text` is scanned for the figures a reader will see (it is never quoted whole; only the figures read from it are).
243
+ - In all of them a control character (newline, NUL, escape; a horizontal tab is fine), a line or paragraph separator, or a bidirectional embedding, override or isolate character (U+202A to U+202E, U+2066 to U+2069) throws `RangeError` naming the field and the character, so the text the kit reads is the text a person sees.
244
+ - A tag character or variation selector in a name, population identifier or stratum name also throws, because those can carry hidden text; the other invisible characters (zero-width space, word joiner, BOM, fillers) are dropped from what is echoed, while joiners, direction marks and the emoji selectors are kept.
245
+ - Error messages write any control or invisible character they quote as `\uXXXX`. A claim copied from wrapped or CRLF text needs its lines joined first (`text.replace(/\s+/g, ' ')`); the error says so.
246
+ - A numeral name is not found where it is part of a percent figure (the name `70` is not in `70%`). A score on a fixed set worded as a general claim ("our model is 79.9% accurate") is refused as `claim-not-scoped-to-benchmark`.
247
+ - To make a general claim, use a random sample and the interval.
248
+ - There is no minimum `n` for a complete set, and strata are not checked for heterogeneity. Label source, retractions, agreement gates, evaluation age and population scope apply as usual.
249
+ - `benchmark` is rejected (`RangeError`) with any `sampling` other than `complete` or `held-out-split`.
250
+
251
+ ## Age and scope
252
+
253
+ - **Date**: `evaluatedAt` is printed in the suggested statement as declared. Without `options.now` the kit cannot know that a date such as `2999-12-31` is in the future (it never reads a clock), so pass `now` whenever the statement will be published.
254
+ - **Age**: set `options.maxAgeDays` and `options.now` (a `YYYY-MM-DD` string; the kit never reads a clock) and pass `evaluatedAt` (`YYYY-MM-DD`). An evaluation older than the limit is refused (`evaluation-stale`); a missing `evaluatedAt` is `evaluation-date-missing`. Both must be real calendar dates, `evaluatedAt` may not be after `now`, and `maxAgeDays` without `now` throws `TypeError`.
255
+ - **Scope**: pass `population: { claimed, evaluated }`, short identifiers for what the claim says it covers and what was actually measured. They are compared after the same folding as the benchmark name (and must contain a letter or digit); different identifiers are refused as `scope-mismatch`. This catches a number tuned on one domain and published for another, but only as far as you declare it: the kit cannot see the data. `options.requireScope: true` refuses a claim with no `population`.
256
+ - **Echoed, not judged**: `population`, `benchmark.name` and stratum names are echoed into the explanation and the statement as typed (control, bidirectional and hidden-text characters are rejected or dropped, and nothing else is checked), and the figure scan runs on `claim.text` only. A name such as `'99.9% accurate'` reaches the statement unchanged and is not judged. Build them from fixed identifiers, never from untrusted metadata.
257
+
258
+ ## Preparing a real claim
259
+
260
+ Real claims come as prose. Pull out these declarations first, and write "unknown" where the source does not say; do not guess:
261
+
262
+ | From the claim | Becomes |
263
+ | --- | --- |
264
+ | "99.8% accurate", "error rate 0.2%" | `claim.metric` and `claim.value` (a proportion), plus `display` exactly as printed |
265
+ | the number of items checked and how many were right or wrong | `sample.n` and `correct` or `errors` |
266
+ | who decided what was right (people, a test suite, a model) | `provenance.source`: `human`, `automated`, `by-construction`, `model`, `mixed`, `unknown` |
267
+ | how items were chosen (all of a named set, a random draw, by score, whatever was handy) | `sampling`: `complete`, `held-out-split`, `random`, `stratified`, `cross-validation`, `ranked`, `convenience`, `unknown` |
268
+
269
+ Pass `display` exactly as printed whenever it is one of the forms in the display table above (plain, approximate, floor, ceiling, range or `1 in N`). Do not drop or rewrite a bound or range: leaving out `<2%` turns a ceiling into a rounded point claim, which can change the verdict from `contradicted` to `supported`. Only prose the kit cannot read (a bare `1.8`, text around the figure) needs cleaning first; the kit throws on a display it cannot read rather than guessing.
270
+
271
+ ## API
272
+
273
+ ```ts
274
+ wilsonInterval(successes, n, { confidence = 0.95 }): WilsonInterval
275
+ zForConfidence(confidence): number
276
+ percentAgreement(pairs): AgreementResult
277
+ cohenKappa(pairs): AgreementResult
278
+ gwetAC1(pairs, { categories? }): AgreementResult
279
+ assessAccuracyClaim(input, options?): AccuracyAssessment
280
+ maxDisplayDecimals(width): number
281
+ DEFAULT_CONFIDENCE (0.95), DEFAULT_MIN_N (30), COMPARISON_TOLERANCE (1e-12)
282
+ ```
283
+
284
+ ### wilsonInterval
285
+
286
+ Returns `{ low, high, point, confidence, z, successes, n }`. `point` is `successes / n`, and `low <= point <= high` within `[0, 1]`.
287
+
288
+ - `successes = 0` gives `low` exactly 0. `successes = n` gives `high` exactly 1.
289
+ - Swapping `successes` and `n - successes` swaps the limits (`low` becomes `1 - high`) to within 1e-15.
290
+ - `n` up to `Number.MAX_SAFE_INTEGER` stays finite; limits near 0 keep their relative accuracy.
291
+ - Throws `TypeError` if an argument is not a number, or if `options` has a key other than `confidence`. Throws `RangeError` if `n` is not a whole number from 1 to `Number.MAX_SAFE_INTEGER`, if `successes` is not a whole number from 0 to `n`, or if `confidence` is not strictly between 0 and 1. `-0` is read as `0`.
292
+
293
+ ### Agreement statistics
294
+
295
+ `pairs` is an array of `[labelA, labelB]`, one pair per item, for two raters. Labels are strings, booleans or finite numbers, compared exactly (`1` and `"1"` differ; nothing is trimmed or normalized). Each function returns one of two shapes:
296
+
297
+ ```ts
298
+ { statistic, status: 'defined', value: number, n, agreements, categories }
299
+ { statistic, status: 'undefined', value: null, reason, n, agreements, categories }
300
+ ```
301
+
302
+ `value` is never `NaN`. Undefined cases:
303
+
304
+ | Statistic | Undefined when | `reason` |
305
+ | --- | --- | --- |
306
+ | all three | there are no pairs | `no-pairs` |
307
+ | `cohenKappa` | chance agreement is exactly 1: both raters used one and the same single label for every item | `expected-agreement-is-1` |
308
+ | `gwetAC1` | fewer than two categories (pass `{ categories: Q }` if the scale allows more than were used) | `single-category` |
309
+
310
+ `categories` is the number of distinct labels seen (for `gwetAC1`, the Q used). Kappa and AC1 are computed from integer counts with `BigInt`, so there is no subtraction error. Malformed pairs throw (`TypeError` for a wrong type, `RangeError` for a wrong length, a non-finite number, a string label over 1000 characters, or more than 1,000,000 distinct labels: agreement is about a small set of categories, and the cap keeps the label table from growing until V8's own Map limit fails after about ten seconds). An array's `length` is validated before it is used (a whole number from 0 to 10,000,000), so a Proxy that reports `Infinity`, `NaN` or a huge length throws instead of looping; the same limit applies to `strata`, and a stratum name is capped at 1000 characters like a label.
311
+
312
+ Pass a result straight into `provenance.agreement`; its `n` is then checked against `options.minN` (see `agreement-sample-too-small`).
313
+
314
+ ### assessAccuracyClaim
315
+
316
+ ```ts
317
+ assessAccuracyClaim({
318
+ claim: { metric, value, display?, text? }, // metric: see Metrics; value in 0..1; display: see Printed forms; text for fixed sets
319
+ sample?: { n, correct? , errors? }, // correct or errors (or both, adding to n); omit when undisclosed
320
+ provenance: { source: 'human' | 'automated' | 'by-construction' | 'model' | 'mixed' | 'unknown', raters?, agreement? },
321
+ sampling: 'random' | 'stratified' | 'ranked' | 'convenience' | 'unknown' | 'complete' | 'held-out-split' | 'cross-validation',
322
+ strata?: [{ name, n, correct? , errors? }], // required with 'stratified'; must add up to the sample
323
+ retractions?: { count, includedInSample },
324
+ benchmark?: { name }, // for 'complete' and 'held-out-split'; rejected with any other sampling
325
+ population?: { claimed, evaluated },
326
+ evaluatedAt?: 'YYYY-MM-DD',
327
+ }, { confidence?, minN?, minAgreement?, maxAgeDays?, now?, requireScope?, conservative?, strictPrecision? })
328
+ ```
329
+
330
+ Returns `{ verdict, reasons, explanation, metric, claimed, assessedValue, n (null if undisclosed), basis, parsedDisplay, checks, interval, conservativeBound, suggestedStatement }`. `reasons` has at least one code and `explanation` has one plain sentence per code. When refused, `interval` and `conservativeBound` are `null` so no number is there to quote, and `suggestedStatement` says why no figure can be given.
331
+
332
+ Rules you can rely on:
333
+
334
+ - **Malformed input throws and never returns a verdict.** Wrong runtime type: `TypeError`. Right type, unusable value (NaN, fractional counts, `correct > n`, a figure outside 0..1, an unknown enum string, a `display` that is not a percent or does not match `value`, strata that do not add up to the sample, duplicate stratum names, more included retractions than items): `RangeError`.
335
+ - **Every property is read once**, in a fixed order, before any judgment. Proxies, getters, class instances, inherited fields and null-prototype objects all work. A getter that throws propagates its own error. `toJSON` and `then` are never called. **An unknown key throws.** The own enumerable keys of the input, `claim`, `sample`, `provenance`, `provenance.agreement`, each stratum, `retractions`, `benchmark`, `population` and the options (also for `wilsonInterval` and `gwetAC1`) are listed once, and a key the kit does not read, such as `retraction`, `maxAgeDay`, `minAgreemnt`, `minn` or `strictPrecison`, is a `TypeError` that names it and the nearest valid key, because a misspelled optional key would otherwise switch its check off and return `supported`. Inherited and symbol keys are ignored. The result's `checks` object says which optional checks ran: `retractions` (a `retractions` block was supplied), `scope` (a `population` was supplied), `age` (`options.maxAgeDays` was set), `agreement` (`options.minAgreement` was set, or the supplied statistic has a value and carries its `n`, so its size was checked; a hand-written statistic with no `n`, or an undefined statistic with no `options.minAgreement`, is not checked) and `strata` (two or more strata were supplied for a sample, so they could be compared), each `applied` or `not-supplied`. `not-supplied` means no check ran, whatever was passed.
336
+ - **`sampling: 'stratified'` needs `strata`.** Without them the pooled counts are not the population rate, so the claim is refused (`stratified-without-strata`). With them, the combined number is refused if strata clearly differ. Otherwise the result carries `strata-not-reweighted` (the pooled interval is not weighted by population share).
337
+ - **A refusal outranks a verdict.** A ranked sample with a plainly false claim is refused, not contradicted.
338
+ - **`display`** is optional (forms in "Printed forms of a figure").
339
+ - Without one, `value` is the only figure judged, and it is read as the shortest decimal within about 9e-16 of it, never coarser than a whole percent: a bare `0.9` is 90% to within half a point, so it does not pass for a true 85.1% on 1,000,000 items, while `0.851` does.
340
+ - A value within about 9e-16 of a printed figure (`82.7 / 100` beside `"82.7%"`) is that figure, not a more favorable one. Readers see the printed figure, so when it is given the verdict judges the **more favorable of `value` and the parsed display** (the higher for higher-is-better metrics, the lower for the others).
341
+ - That figure is returned as `assessedValue`, and the explanation says "printed figure" when it came from the display. A `value` of 0.996 printed as "100%" on 1000 of 1000 correct is judged as 100% and comes out `overstated`, not `supported`.
342
+ - A display that disagrees with `value` by more than the allowance throws `RangeError`, so the two can never quietly diverge. The comparison is exact (the double's exact value against the display's decimal digits, no float tolerance) and the allowance is half a unit of the display's last digit plus one ulp of the value.
343
+ - A figure a person rounded by hand (0.985 printed as `99%`) is therefore never rejected for the double's own representation error, while `98.00000000009%` for 0.98 still is.
344
+ - The display must be a percent from 0 to 100 (its decimal digits are checked before conversion, so `100.000000000000001%` is rejected rather than rounded to 100%; signs are not accepted).
345
+ - `assessedValue` is computed before any refusal, so a refused result reports the same selected figure. The number of decimals is also checked against the interval (see below); whole-percent figures are never flagged.
346
+ - **`provenance.agreement`**: a defined statistic needs a finite `value` from -1 to 1. An undefined one (`status: 'undefined'`, as `cohenKappa` and `gwetAC1` return) must carry `value: null`; `NaN`, numbers, text or objects in its place throw. An undefined statistic is accepted as input but can never pass a gate (`agreement-undefined`), and without `options.minAgreement` it is not checked at all. The other keys a kit result carries (`agreements`, `categories`, `reason`) are accepted so a result can be passed straight in and are validated but not used: `agreements` and `categories` must be whole numbers (`agreements` no more than `n`), and `reason` one of the reasons this kit returns. A statistic that carries `n` (every result from this kit does, and a hand-written one may add it) is refused when `n` is below `options.minN` (`agreement-sample-too-small`), with or without `minAgreement`, because a handful of items says nothing about the raters. `provenance.agreement` with `provenance.raters` of 1 throws `RangeError`: agreement is between raters.
347
+ - **`provenance.raters`** is a declaration the verdict does not use: `raters: 1` and `raters: 2` give the same result, and the statement never mentions it. It must be a whole number of at least 1, and with a `provenance.agreement` it must be at least 2 (agreement is between raters; `raters: 1` throws `RangeError`). Two raters do not make a claim stronger; what carries it is the agreement statistic you pass and gate on.
348
+ - **`options.confidence`** is from 0.5 up to (not including) 1; a lower level throws `RangeError`, because `0.05` typed for `0.95` gives an interval narrower than the data's own noise and an "at least" statement that means nothing. `wilsonInterval` itself accepts any level strictly between 0 and 1.
349
+ - **`options.minAgreement`** (from -1 to 1) turns agreement into a gate. Equal to the minimum passes. Without it, a supplied agreement statistic is only echoed in the statement. The gate compares whichever statistic you pass with one number, and the statistics are not on one scale: percent agreement is not chance-corrected and runs higher than Cohen's kappa or Gwet's AC1 on the same labels (two raters who both call 95% of items "yes" agree on about 90% of them by chance alone), so a threshold written for kappa is easy to clear with percent agreement. Pick the statistic and the threshold together, and pass the statistic computed on the labels behind this sample: the kit cannot see which labels it came from.
350
+ - **Strata**: if two strata have non-overlapping intervals the combined number is refused. The intervals use a Bonferroni split of the error budget across strata (`k` strata use confidence `1 - (1 - confidence) / k`), so many strata do not cause refusals by chance. One stratum never triggers it. A stratum needs at least one item (`n` of 1 or more); leave an empty stratum out, since it has no rate to compare.
351
+
352
+ ### Floating point and boundaries
353
+
354
+ Comparisons between a claim and an interval limit use an absolute tolerance of `COMPARISON_TOLERANCE = 1e-12`, always in the claimant's favor. A claim within 1e-12 of a limit counts as being at it: for accuracy, exactly at the lower end is `supported`, and exactly at the upper end is `overstated` (not `contradicted`). The tolerance is far below the width of any interval met in practice. It can matter only above about 4e12 items with a rate within about 1e-12 of 0 or 1, where the interval itself is narrower than the tolerance (the width of an all-correct sample is about 3.84 divided by `n`). A claim of exactly 0 or 1 (`0%`, `100%`) is absolute and never uses the tolerance, so it is never `supported` as a bound at any size. Agreement gates use the same tolerance. The test table has rows exactly at each boundary and 1e-9 on either side.
355
+
356
+ ### Precision of a displayed figure
357
+
358
+ A figure printed with `D` decimals has a last digit worth `10^-D` percentage points. It is accepted while `10^-D` is at least one tenth of the interval width in percentage points; otherwise the last digits are noise and the result carries `precision-too-fine`. `maxDisplayDecimals(width)` returns the largest accepted `D` for an interval width given as a proportion. For 499 of 500 the width is 1.09 points, so only whole percents pass and "99.8%" is flagged. This is a rule of thumb stated here, not a standard. By default this is informational: `precision-too-fine` is listed in `reasons` and `explanation`, the suggested statement ends with "; print it as a whole percent" or "; print it to at most N decimal places", and the verdict is not changed, because excess printed precision is not evidence that the figure is wrong. Pass `options.strictPrecision: true` to make it turn a `supported` claim into `overstated` (`options.conservative` implies it). A contradicted claim stays contradicted either way.
359
+
360
+ ### Suggested statements
361
+
362
+ For non-refused results the statement leads with its scope and ends with what the interval does not cover: `On this sample, at least X% accurate (95% Wilson interval L% to H%, n = N, human labels, random sample); the interval covers sampling error only`, or `On this sample, at most X% error rate (...)`, with `automated labels` and `evaluated YYYY-MM-DD` shown when they apply. When `population` is declared the lead names it (`On this sample of English tweets, at least ...`). A cross-validation estimate adds that the interval is optimistic because the folds share data. When a printed figure carries more decimals than the interval supports, the statement ends with `; print it as a whole percent` or `; print it to at most N decimal places`. For a complete set it reads like `39950 of 50000 correct on <set> (complete set, human labels): 79.90% accuracy. This describes that set only; it is not an estimate for other data`. Bounds always round in the cautious direction (lower end down, upper end up), exactly: the double is split into mantissa and exponent and rounded with integer arithmetic, so a printed bound is never on the favorable side of the number it describes (an all-correct sample of 1e14 prints `At least 99.9999...%`, never 100%). The confidence level is labeled from its shortest decimal text, rounded down, and never prints as 100% (`0.95` reads `95%`). The statement describes the evidence; it never says the system is accurate.
363
+
364
+ ## Formulas
365
+
366
+ With `z` the two-sided normal quantile for the confidence level (`z = 1.95996...` at 0.95), `s` successes in `n` trials:
367
+
368
+ ```
369
+ centre = (s + z^2/2) / (n + z^2)
370
+ margin = z * sqrt( s(n - s)/n + z^2/4 ) / (n + z^2)
371
+ Wilson interval = [centre - margin, centre + margin]
372
+ ```
373
+
374
+ The code evaluates the lower limit as `s^2 / (n * (A + B))` with `A = s + z^2/2` and `B = z * sqrt(s(n - s)/n + z^2/4)`, which is the same number without cancellation when `s` is small. Above the midpoint the upper limit is `1 - (lower limit for n - s)`.
375
+
376
+ `z` comes from an inverse normal CDF: Acklam's rational approximation followed by one Halley refinement step against an `erf`/`erfc` written out in the source. It matches Python's `NormalDist().inv_cdf` to about 2e-15 relative on the reference points. For confidence levels up to 0.5 the quantile is solved from the displacement `c/2` from the median rather than from `0.5 + c/2`, which would round away every digit of a tiny `c` (`zForConfidence(1e-16)` is about 1.2533e-16, not 0); below 1e-9 it is `c * sqrt(pi/2)`, which also keeps `Number.MIN_VALUE` from underflowing to 0; the Python generator checks that range with a Newton solve of the erf Taylor series in 60-digit decimal because `NormalDist` has the same cancellation there. Below p of about 1e-300 the refinement is skipped and the answer is good to about 1e-9.
377
+
378
+ For two raters over `N` items, `a_k` and `b_k` the counts of label `k` from each rater, `agree` the items with equal labels:
379
+
380
+ ```
381
+ po = agree / N
382
+ Cohen pe = sum_k (a_k/N)(b_k/N)
383
+ kappa = (po - pe) / (1 - pe) undefined if pe = 1
384
+ Gwet pi_k = (a_k + b_k) / (2N)
385
+ pe = sum_k pi_k (1 - pi_k) / (Q - 1)
386
+ AC1 = (po - pe) / (1 - pe) undefined if Q < 2
387
+ ```
388
+
389
+ ## What it does not do
390
+
391
+ - It does not judge whether the labels are right. "Human labels" is a declaration you make; the kit cannot see who labeled anything.
392
+ - It cannot detect sampling fraud or a sample that was filtered before you counted it. `sampling: 'random'` is taken at your word.
393
+ - A `supported` verdict is not proof that the system is accurate. It says only that the claim is no more favorable than the lower end of an interval computed from counts you supplied, or is an honest report of the rate you observed (`matches-observed-rate`). The interval describes sampling noise alone; it does not cover label errors, drift, a different population in production, or choosing the best of several runs.
394
+ - The Wilson interval treats the pooled counts as one simple random sample. A stratified sample with unequal sampling rates is not a simple random sample; the interval is not reweighted, so use per-stratum results for those.
395
+ - It does not compare two systems, test significance between them, or estimate calibration.
396
+ - It does not give advice, legal conclusions, or compliance conclusions. It checks evidence; it does not certify truth, and it does not decide whether evidence meets any legal standard.
397
+ - The 30-item default minimum, the 1/10-of-width precision rule and the strata check are policy choices written down here, not statistical laws. Change `minN` if your situation differs.
398
+ - It does not check an interval you quote. A published "95% CI 0.0 to 1.2" is not compared with the computed interval; a claim that carries its own correct interval can still come back `overstated` when the point figure sits above the interval's lower end. Use the suggested statement, which quotes the computed interval.
399
+ - It does not compare systems ("+27 points over the old one", "best of ten models") and does not reweight a stratified sample by population share; for that, report per-stratum results.
400
+ - It does not model excluded or non-answered items (a rate computed only over items every system answered), or clustered, non-independent items; the interval assumes independent items.
401
+ - The scope and benchmark-wording checks compare strings you provide. They catch a mismatch you declared, not one you did not. Naming the benchmark in `claim.text` is necessary for a complete-set claim, not sufficient: the kit cannot read whether the sentence still generalizes. The name must appear as its own word or phrase (`test` is not found inside `latest`), and a plural or a different spelling of the name is treated as not named. A match followed by `+`, or by `-`, `_`, `/` or `.` and then a letter or digit, is a longer name and does not count (`MMLU` is not found in `MMLU-Pro`, `HumanEval` is not found in `HumanEval+`; the same joiners before the match count too, as in `mini-HumanEval`), while a sentence-ending period (`on CIFAR-10.`) is fine. A variant set off by a space (`SWE-bench Verified`, `SWE-bench Lite`) cannot be told apart from ordinary words, so declare the full name of the set you measured.
402
+ - `1 in N` is read with N exact (plus or minus one half), so it is as tight as the figure looks for a long N. For a short N the band in rate terms is still wide (`1 in 3` covers 1 in 2.5 to 1 in 3.5, a rate from 28.6% to 40%), and `about 1 in N` allows the whole placeholder band; the explanation says which band was applied. `1 in 1` is not a band: it is 100%, an absolute claim that matches only `n` of `n`.
403
+ - Naming the benchmark is checked as a word, not as an identifier: a name of one letter, a bare number or an ordinary word (`a`, `2048`, `math`, `The`) is found wherever it occurs as its own word (a numeral is not found inside a percent figure, so `70` is not found in `70%`). Declare the real name of the set; the kit cannot tell that `set` is not one.
404
+ - Consecutive or prospective enrollment (a clinical study that enrolls every eligible patient in order, often against a comparator method) is not one of the `sampling` declarations. `convenience` is refused and `random` would not be true, so there is no honest declaration for it here. Report the counts with `wilsonInterval` and state the enrollment and the reference method yourself.
405
+ - A key that is present with the value `undefined` is read as absent. `{ minAgreement: config.minAgreemnt }`, with a misspelled property read, turns the gate off exactly as leaving it out does, and the kit cannot tell the two apart (a misspelled key in the object itself does throw). When a gate must run, check `result.checks` after the call (`result.checks.agreement === 'applied'`).
406
+ - JSON has no `undefined`, and the kit does not read `null` as absent: leave a key out instead of writing `null` (`sample: null`, `benchmark: null` and `display: null` throw `TypeError`), pick the fields you mean rather than passing a whole row (a column such as `id` is an unknown key), and pass a plain object as `options` (an ES module namespace or a class with a `Symbol.toStringTag` is not one). A sample with `n: 0` throws `RangeError`: nothing was evaluated, so there is nothing to judge.
407
+ - Length caps: `benchmark.name`, `population.claimed` and `population.evaluated` at most 200 characters, `claim.text` at most 2000, `claim.display` at most 64, agreement labels and stratum names at most 1000; at most 10,000,000 pairs or strata and 1,000,000 distinct labels. Anything over a cap throws `RangeError`.
408
+ - Multiple raters beyond two, ordinal weighting (weighted kappa), and confidence intervals for kappa or AC1 are not included.
409
+
410
+ ## Relationship to advice-ledger-kit
411
+
412
+ `advice-ledger-kit` grades a log of recommendations against human judgments and outcomes, and it deliberately produces no confidence interval and never blends its counts into one accuracy number. This kit is the other half: when someone does want to publish an accuracy figure, it says whether the counts can carry it. They share no code.
413
+
414
+ ## Tests and vectors
415
+
416
+ `npm run verify` runs lint, type check, tests, the build and a packaging check that installs the packed tarball into a clean project. `npm run attw` checks the type resolution of the packed package, and `npm run audit:dependencies` audits the dev tooling in the lockfile (it needs registry access).
417
+
418
+ ### Cross-checked against reference libraries
419
+
420
+ `test/reference.test.ts` requires the kit to match statsmodels, scipy, scikit-learn and irrCAC to a maximum absolute difference of 1e-12. The numbers those libraries produced are in `test/fixtures/reference.json`, written by `scripts/reference/gen.py` (a developer tool that is not shipped; the versions are recorded in the file: statsmodels 0.15.0, scipy 1.18.1, scikit-learn 1.9.1, irrCAC 0.4.4). The kit has no runtime dependency on any of them.
421
+
422
+ | Quantity | Compared with | Cases | Largest difference |
423
+ | --- | --- | --- | --- |
424
+ | Wilson interval, both limits | statsmodels `proportion_confint(method="wilson")` and scipy `binomtest(...).proportion_ci(method="wilson")` | 8,899 (n from 1 to 10^12, confidence 50% to 99.99%) | 4.9e-15 |
425
+ | z for a confidence level | scipy `norm.isf` | 9 levels | 1.1e-15 |
426
+ | Percent agreement | scikit-learn `accuracy_score`, irrCAC | 1,499 of 1,500 two-rater tables | 6.7e-16 |
427
+ | Cohen's kappa | scikit-learn `cohen_kappa_score`, irrCAC (table and raw classes) | 1,314 defined, plus 186 undefined in every library and in the kit | 3.3e-16 against scikit-learn, 6.7e-14 against irrCAC |
428
+ | Gwet's AC1 | irrCAC (table and raw classes) | 1,353 defined in both irrCAC classes (the kit defines 1,354) | 1.1e-14 |
429
+
430
+ The tables run from 1 to 8 categories and 1 to 1,000 items, with skewed, diagonal-heavy and negative-kappa shapes, unused categories, and irrCAC's own documentation tables. The one AC1 the kit defines and irrCAC's raw class does not (one item, both raters on the first of two declared categories) is not compared. A kappa or AC1 that every library leaves undefined is undefined in the kit and never `NaN`. The libraries disagree with one another on one degenerate case: when every item falls in one category, Gwet's formula divides by zero, irrCAC's table class returns nothing and its raw class returns 1; the kit returns `undefined` with reason `single-category` (146 tables in the grid).
431
+
432
+ ### Field benchmark
433
+
434
+ From a clone, `npm run bench` runs the kit over 216 percentages with counts printed in 75 public FDA 510(k) summaries (derived numbers and source URLs only; `bench/README.md` has the method and licenses). Each is judged by `bench/oracle.py`, a second implementation of the rules in this README. The oracle restates this README's policy, so agreement shows the kit does what it documents, not that the policy is right.
435
+
436
+ - Of the 957 altered claims the oracle says the counts do not carry, the kit passed 0 as `supported`.
437
+ - Of the 726 claims the oracle says the counts carry, the kit said `supported` for 726.
438
+ - The kit's verdict equals the oracle's on 2,001 of 2,001 judged claims (also under `strictPrecision` and `conservative`), 216 of 216 with each document's declared sampling and label source, 3,498 of 3,498 as claims about a whole fixed set, and 95 of 95 pooled site tables.
439
+
440
+ A slice of the corpus (547 cases) runs in the test suite (`test/field-corpus.test.ts`).
441
+
442
+ A separate holdout (`bench/holdout/`) has 103 public claims with printed counts from 30 open-access papers and one vendor benchmark page, none from the FDA corpus. Each was labeled by hand from the README's rules and the label committed before the kit ran on it. The kit's verdict equals the label on 103 of 103 (73 supported, 24 refused, 4 contradicted, 2 overstated). The six contradicted or overstated rows are five percentages that are not the rate their own counts give (four truncated instead of rounded, one mistyped) and one vendor sentence. 71 of the 103 are scores over a fixed set, so the interval is exercised by only 6 claims; `bench/holdout/README.md` says what the result does not show.
443
+
444
+ ### Vectors
445
+
446
+ Expected values for the printed-form, complete-set and noisy-value grids come from `test/fixtures/generate_vectors.py`, which uses only the Python standard library (`math`, `decimal`, `fractions`, `random` with a fixed seed, `statistics.NormalDist`). The Wilson reference runs in 60-digit decimal arithmetic; agreement statistics run in exact rational arithmetic. The committed output is `test/fixtures/vectors.json`.
447
+
448
+ Where published values were available they are included and labeled in the fixture:
449
+
450
+ - Cohen's kappa: the worked examples on the Cohen's kappa Wikipedia page (0.40, 0.1304, 0.2593, 0.0089, -0.0667). These were read from the page.
451
+ - Wilson's 1927 formula and Gwet's (2008) AC1 example could not be retrieved. Wilson's formula is checked against statsmodels and scipy, and AC1 against irrCAC, as above. No value in the fixtures is recalled from memory.
452
+
453
+ ## Layout
454
+
455
+ ```
456
+ src/index.ts public exports
457
+ src/types.ts input, option and result types
458
+ src/wilson.ts Wilson interval, z for a confidence level
459
+ src/normal.ts inverse normal CDF
460
+ src/agreement.ts percent agreement, Cohen's kappa, Gwet's AC1
461
+ src/assess.ts assessAccuracyClaim
462
+ src/display.ts printed-form parser (percent, bounds, ranges, 1 in N)
463
+ src/census.ts exact comparison for complete sets and held-out splits
464
+ src/exact.ts exact (BigInt) comparison of a double with decimal digits
465
+ src/format.ts cautious rounding, display precision rule
466
+ src/dates.ts YYYY-MM-DD checks for age limits
467
+ src/text.ts identifier folding for scope and benchmark names
468
+ src/validate.ts input readers
469
+ test/ verdict table, numeric vectors, reference cross-check, field slice, hostile inputs
470
+ scripts/ package verification and consumer probes; reference/gen.py (developer tool)
471
+ bench/ field benchmark, heuristic rates and holdout (not shipped)
472
+ METHOD.md why each verdict and default, measured rates, limits, framework mapping
473
+ ```
@@ -0,0 +1,62 @@
1
+ /** A categorical label. Compared exactly: 1 and "1" are different labels. */
2
+ export type Label = string | number | boolean;
3
+ export type AgreementStatistic = 'percent-agreement' | 'cohen-kappa' | 'gwet-ac1';
4
+ export type UndefinedAgreementReason =
5
+ /** The list of pairs was empty. */
6
+ 'no-pairs'
7
+ /** Cohen's kappa: chance agreement is exactly 1 (both raters used one and the same single label for every item). */
8
+ | 'expected-agreement-is-1'
9
+ /** Gwet's AC1: fewer than two categories, so the chance model has nothing to spread over. */
10
+ | 'single-category';
11
+ interface AgreementBase {
12
+ statistic: AgreementStatistic;
13
+ /** Number of pairs used. */
14
+ n: number;
15
+ /** Pairs where the two labels are equal. */
16
+ agreements: number;
17
+ /** Distinct labels seen across both raters; for gwetAC1 the category count Q actually used. */
18
+ categories: number;
19
+ }
20
+ export interface DefinedAgreement extends AgreementBase {
21
+ status: 'defined';
22
+ value: number;
23
+ }
24
+ export interface UndefinedAgreement extends AgreementBase {
25
+ status: 'undefined';
26
+ value: null;
27
+ reason: UndefinedAgreementReason;
28
+ }
29
+ export type AgreementResult = DefinedAgreement | UndefinedAgreement;
30
+ export interface GwetOptions {
31
+ /**
32
+ * Number of categories the label scale allows (Q in Gwet's formula), when
33
+ * some categories may be unused in the sample. Must be a whole number at
34
+ * least as large as the number of distinct labels seen. Default: the
35
+ * number of distinct labels seen.
36
+ */
37
+ categories?: number | undefined;
38
+ }
39
+ /** Most distinct labels in one set of pairs. Agreement is about a handful of categories; this only keeps the label table from growing without bound. */
40
+ export declare const MAX_DISTINCT_LABELS = 1000000;
41
+ /**
42
+ * Share of pairs where the two raters gave the same label: agreements / n.
43
+ * With no pairs the result is `undefined` with reason "no-pairs".
44
+ * Throws TypeError / RangeError for malformed pairs.
45
+ */
46
+ export declare function percentAgreement(pairs: readonly (readonly [Label, Label])[]): AgreementResult;
47
+ /**
48
+ * Cohen's kappa for two raters: (po - pe) / (1 - pe), with pe the chance
49
+ * agreement implied by each rater's own label frequencies. Computed from
50
+ * integer counts (BigInt), so there is no cancellation error.
51
+ * Undefined when pe is exactly 1, or when there are no pairs.
52
+ */
53
+ export declare function cohenKappa(pairs: readonly (readonly [Label, Label])[]): AgreementResult;
54
+ /**
55
+ * Gwet's AC1 for two raters: (po - pe) / (1 - pe) with
56
+ * pe = sum_k pi_k (1 - pi_k) / (Q - 1) and pi_k the mean share of label k
57
+ * across both raters. Computed from integer counts (BigInt).
58
+ * Undefined with fewer than two categories (Q < 2) or no pairs.
59
+ * Throws RangeError if options.categories is below the number of labels seen.
60
+ */
61
+ export declare function gwetAC1(pairs: readonly (readonly [Label, Label])[], options?: GwetOptions): AgreementResult;
62
+ export {};