@mark1russell7/rigor-statistics 0.1.9 → 0.1.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +70 -0
- package/build/index.d.ts +78 -1
- package/build/index.d.ts.map +1 -1
- package/build/index.js +312 -3
- package/build/index.js.map +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -117,10 +117,14 @@ The default expansion has one product for each pair of levels of two dimensions.
|
|
|
117
117
|
| `pairedPermutation(candidate, baseline, options)` | The paired permutation test (random sign flips). It uses the size of each difference. |
|
|
118
118
|
| `wilson(k, n, z)` | The Wilson interval of a proportion. |
|
|
119
119
|
| `medianInterval(values, level)` | The distribution-free interval of a median, from two order statistics. It gives `low`, `high`, the `rank` of `low` and the `coverage`. |
|
|
120
|
+
| `quantileInterval(values, p, level)` | The distribution-free interval of a quantile, from two order statistics. A side with no rank is infinite. The values must be independent draws. |
|
|
121
|
+
| `quantileSamples(p, level)` | The fewest independent values that give a quantile an upper bound. At the level 0.95: 72 values for p95, 368 for p99 and 3,688 for p99.9. |
|
|
120
122
|
| `binomialInterval(n, p, level)`, `negativeBinomialInterval(successes, p, level)` | The central interval of a binomial count, and of a count of failures before `successes` successes. Each tail outside holds `(1 - level) / 2` or less. |
|
|
121
123
|
| `holm(pValues, level)` | The Holm correction of a family of tests. |
|
|
122
124
|
| `spearman`, `kendallTau`, `ranks` | The rank correlations, with mean ranks for ties. |
|
|
123
125
|
| `compareRatios`, `noiseFloor`, `noiseVerdict` | The comparison with a baseline, the noise floor of the 5-seed rule with a margin of practical effect, and the state of a difference over the seeds. |
|
|
126
|
+
| `modeCount(values, { minShare, valleyRatio })` | The count of the modes of a sample, from a kernel density estimate. |
|
|
127
|
+
| `smoothLawScores(values)`, `bestSmoothLaw(values)` | The mean log density of the uniform, normal, exponential, log-normal and Pareto laws at their fits, and the best of them. |
|
|
124
128
|
| `brier`, `calibration` | The Brier score of predictions, and the hit rate of each stated level. |
|
|
125
129
|
| `fitResponse(data, options)` | A record of the format `response-records/1`, with one model for each measure and held-out folds. |
|
|
126
130
|
| `parseResponseRecord(value)` | The examination of unknown data. It gives the record, or one fault with a path for each error. |
|
|
@@ -132,6 +136,72 @@ The default expansion has one product for each pair of levels of two dimensions.
|
|
|
132
136
|
|
|
133
137
|
Each function throws a RangeError for an empty list, for a NaN and for a parameter out of its range. Thus a missing value cannot make a quiet NaN in a record.
|
|
134
138
|
|
|
139
|
+
### The limits of the shape of a sample
|
|
140
|
+
|
|
141
|
+
`modeCount` has two reads. A sample with 24 or fewer distinct values, and 4 or more copies of each value on average, gets a direct read. That read uses the counts of the values. A kernel estimate reads each other sample.
|
|
142
|
+
|
|
143
|
+
The count is the same for `x` and `-x`. Two humps join in the order of the depth of their valleys, not in the order of their positions. A test holds this on seeded random laws.
|
|
144
|
+
|
|
145
|
+
The direct read does not change with the size of the sample, so one discrete law gives one result. A test holds this table, with a gap of 100 between the groups.
|
|
146
|
+
|
|
147
|
+
| Groups of equal values | 10 values each | 50 values each | 200 values each | 1,000 values each |
|
|
148
|
+
| --- | --- | --- | --- | --- |
|
|
149
|
+
| 2 | 2 | 2 | 2 | 2 |
|
|
150
|
+
| 3 | 3 | 3 | 3 | 3 |
|
|
151
|
+
| 6 | 6 | 6 | 6 | 6 |
|
|
152
|
+
| 10 | 10 | 10 | 10 | 10 |
|
|
153
|
+
| 12 | 12 | 12 | 12 | 12 |
|
|
154
|
+
| 16 | 16 | 16 | 16 | 16 |
|
|
155
|
+
| 24 | 24 | 24 | 24 | 24 |
|
|
156
|
+
| 25 | 1 | 1 | 1 | 1 |
|
|
157
|
+
|
|
158
|
+
- Flat counts give one mode for each value. So a rounded uniform law with 24 or fewer steps gives one mode for each value, and two close values are two modes.
|
|
159
|
+
- Counts with one rise and one fall give one mode. This read needs 4 or more values. A normal law at a step of 2 SD has 3 values, so it gives 3.
|
|
160
|
+
- A value under 1 percent of the sample is out of this read.
|
|
161
|
+
- Under 1,000 values the noise of a sample can move the count of a rounded uniform law with 10 or more steps by 1.
|
|
162
|
+
|
|
163
|
+
The kernel joins more than approximately 24 evenly spaced clusters, because its grid has 96 points. A tight cluster also needs more than approximately `n ** 0.8 / 8` values in a sample of `n` values. A cluster with an SD over 5 percent of the gap to the next cluster can join it. Under these limits a test holds the count at 20 clusters. A step of rounding is no gap between clusters. So the width of the kernel stays over the median gap of the distinct values of a tight sample.
|
|
164
|
+
|
|
165
|
+
The default counts the modes that hold 10 percent of the mass. A caller who needs all modes passes a smaller `minShare`.
|
|
166
|
+
|
|
167
|
+
A small smooth sample can give a false extra mode. This table gives the share of 500 seeded samples of each law with a count of 1. A test holds it.
|
|
168
|
+
|
|
169
|
+
| Values | uniform | normal | exponential | log-normal | Pareto |
|
|
170
|
+
| --- | --- | --- | --- | --- | --- |
|
|
171
|
+
| 10 | 0.54 | 0.51 | 0.54 | 0.55 | 0.50 |
|
|
172
|
+
| 30 | 0.80 | 0.81 | 0.76 | 0.70 | 0.77 |
|
|
173
|
+
| 100 | 0.99 | 0.98 | 0.88 | 0.92 | 0.91 |
|
|
174
|
+
| 300 | 1.00 | 1.00 | 0.98 | 0.99 | 0.99 |
|
|
175
|
+
| 1,000 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
|
|
176
|
+
|
|
177
|
+
A discrete law with one mode can also give a false count. This table gives the share of 1,000 seeded samples with a count that is not 1. A test holds it.
|
|
178
|
+
|
|
179
|
+
| Law | 100 values | 300 values | 1,000 values |
|
|
180
|
+
| --- | --- | --- | --- |
|
|
181
|
+
| Rounded normal, SD 4 | 0.003 | 0.001 | 0.001 |
|
|
182
|
+
| Poisson, mean 3 | 0.006 | 0.001 | 0.000 |
|
|
183
|
+
| Poisson, mean 10 | 0.001 | 0.000 | 0.000 |
|
|
184
|
+
| Binomial, 10 trials, p 0.5 | 0.008 | 0.009 | 0.001 |
|
|
185
|
+
| Binomial, 100 trials, p 0.5 | 0.001 | 0.000 | 0.000 |
|
|
186
|
+
| Poisson, mean 1 | 0.000 | 0.000 | 0.000 |
|
|
187
|
+
| Geometric, p 0.1 | 0.213 | 0.028 | 0.000 |
|
|
188
|
+
|
|
189
|
+
- Do not trust one count at 100 values. Use the table to find the risk.
|
|
190
|
+
- In the first five rows a false count is rare, and most of them are large. A bell passes the flat test, so each of its values counts as a mode.
|
|
191
|
+
- The kernel reads a geometric law with p 0.1 at these sizes. A false count from it is one extra mode.
|
|
192
|
+
|
|
193
|
+
`bestSmoothLaw` is not reliable under 300 values. The laws with a fitted end have a benefit at small sizes. This table gives the share of correct results for 500 seeded samples of each law. A test holds it.
|
|
194
|
+
|
|
195
|
+
| Values | uniform | normal | exponential | log-normal | Pareto |
|
|
196
|
+
| --- | --- | --- | --- | --- | --- |
|
|
197
|
+
| 10 | 0.96 | 0.01 | 0.45 | 0.02 | 0.69 |
|
|
198
|
+
| 30 | 1.00 | 0.30 | 0.90 | 0.25 | 0.87 |
|
|
199
|
+
| 100 | 1.00 | 0.68 | 0.99 | 0.71 | 0.99 |
|
|
200
|
+
| 300 | 1.00 | 0.85 | 1.00 | 0.99 | 1.00 |
|
|
201
|
+
| 1,000 | 1.00 | 0.97 | 1.00 | 1.00 | 1.00 |
|
|
202
|
+
|
|
203
|
+
The samples use a normal law with a mean of 100 and an SD of 10, where the normal and log-normal laws are close.
|
|
204
|
+
|
|
135
205
|
## Moves from the copies
|
|
136
206
|
|
|
137
207
|
The tests prove that these functions give the same numbers as the copies in the engines, to the bit:
|
package/build/index.d.ts
CHANGED
|
@@ -140,6 +140,56 @@ export declare function bootstrapMedian(values: readonly number[], options?: Boo
|
|
|
140
140
|
* are in the order of their first row.
|
|
141
141
|
*/
|
|
142
142
|
export declare function bootstrapClusters<T>(rows: readonly T[], cluster: (row: T) => string, statistic: (rows: readonly T[]) => number, options?: BootstrapOptions): Interval;
|
|
143
|
+
/** The smooth laws that `smoothLawScores` compares. Each law has 2 fitted numbers. */
|
|
144
|
+
type SmoothLaw = "uniform" | "normal" | "exponential" | "log-normal" | "pareto";
|
|
145
|
+
/** The value of one law at its fit. */
|
|
146
|
+
interface SmoothLawScore {
|
|
147
|
+
readonly law: SmoothLaw;
|
|
148
|
+
/** The mean log density of the values at the fit of the law. A larger value is a better fit. */
|
|
149
|
+
readonly meanLogDensity: number;
|
|
150
|
+
}
|
|
151
|
+
/** The options of `modeCount`. */
|
|
152
|
+
interface ModeOptions {
|
|
153
|
+
/**
|
|
154
|
+
* The smallest share of the mass that makes a mode. The default is `0.1`. Half of the mass of the largest mode also
|
|
155
|
+
* makes a mode, so many equal modes each count.
|
|
156
|
+
*/
|
|
157
|
+
readonly minShare?: number;
|
|
158
|
+
/** The valley between two peaks of the kernel is under this share of the lower peak. The default is `0.5`. */
|
|
159
|
+
readonly valleyRatio?: number;
|
|
160
|
+
}
|
|
161
|
+
/**
|
|
162
|
+
* This function counts the modes of a sample. A mode counts when it holds `minShare` of the mass. The default counts the
|
|
163
|
+
* modes that hold 10 percent, and a caller who needs all modes passes a smaller value. The result is at least 1. The
|
|
164
|
+
* result is the same for `x` and `-x`.
|
|
165
|
+
*
|
|
166
|
+
* A sample with 24 or fewer distinct values, and 4 or more copies of each value on average, gets a direct read.
|
|
167
|
+
* That read uses the counts of the values, and it does not change with the size of the sample.
|
|
168
|
+
* It gives one mode for each value of a flat discrete law. So even groups give the count of the groups, and so does a
|
|
169
|
+
* rounded uniform law with 24 or fewer steps. It gives one mode for each hump of a rounded smooth law.
|
|
170
|
+
* The direct read of a sample with 3 or fewer values gives one mode for each value that holds the share.
|
|
171
|
+
*
|
|
172
|
+
* A kernel estimate on a grid reads each other sample. The grid has 96 points, so the kernel joins more than
|
|
173
|
+
* approximately 24 evenly spaced clusters. A tight cluster also needs more than approximately `n ** 0.8 / 8` values for
|
|
174
|
+
* a sample of `n` values. A cluster with an SD over 5 percent of the gap to the next cluster can join it.
|
|
175
|
+
*
|
|
176
|
+
* The function throws a RangeError for an empty list or for a value that is not finite.
|
|
177
|
+
*/
|
|
178
|
+
export declare function modeCount(values: readonly number[], options?: ModeOptions): number;
|
|
179
|
+
/**
|
|
180
|
+
* This function gives the mean log density of the values under each smooth law, at the closed-form fit of that law.
|
|
181
|
+
* Each law has 2 fitted numbers, so the function adds no penalty. The uniform, the shifted exponential and the Pareto
|
|
182
|
+
* law take the smallest value as their lower limit. This fitted end helps them at small sizes. All values are exact
|
|
183
|
+
* maximum likelihoods.
|
|
184
|
+
* The function throws a RangeError for an empty list, for a value that is not positive and finite, or for equal values.
|
|
185
|
+
*/
|
|
186
|
+
export declare function smoothLawScores(values: readonly number[]): readonly SmoothLawScore[];
|
|
187
|
+
/**
|
|
188
|
+
* This function gives the smooth law with the highest likelihood. Under 300 values the result is not reliable: the
|
|
189
|
+
* normal and the log-normal law lose to the laws with a fitted end. The README gives the measured shares of correct
|
|
190
|
+
* results. The function throws as `smoothLawScores` does.
|
|
191
|
+
*/
|
|
192
|
+
export declare function bestSmoothLaw(values: readonly number[]): SmoothLaw;
|
|
143
193
|
/**
|
|
144
194
|
* The ridge solve of a least-squares problem from its normal equations. The caller gives the matrix `X'X` and the
|
|
145
195
|
* vector `X'y`. Thus it can make them from sparse rows. It can also remove the sums of one fold of a cross-validation.
|
|
@@ -763,6 +813,33 @@ interface MedianInterval {
|
|
|
763
813
|
* is less than `level`.
|
|
764
814
|
*/
|
|
765
815
|
export declare function medianInterval(values: readonly number[], level?: number): MedianInterval;
|
|
816
|
+
/** The distribution-free interval of a quantile, from two order statistics of the sample. */
|
|
817
|
+
interface QuantileInterval {
|
|
818
|
+
/** The value x_(lowRank), or `-Infinity` when the sample has no rank for a lower bound. */
|
|
819
|
+
readonly low: number;
|
|
820
|
+
/** The value x_(highRank), or `Infinity` when the sample has no rank for an upper bound. */
|
|
821
|
+
readonly high: number;
|
|
822
|
+
/** The rank of `low`, from 1, or 0 when `low` is `-Infinity`. */
|
|
823
|
+
readonly lowRank: number;
|
|
824
|
+
/** The rank of `high`, from 1, or n + 1 when `high` is `Infinity`. */
|
|
825
|
+
readonly highRank: number;
|
|
826
|
+
/** The chance that the interval holds the quantile of the population. */
|
|
827
|
+
readonly coverage: number;
|
|
828
|
+
}
|
|
829
|
+
/**
|
|
830
|
+
* This function gives the distribution-free interval of the quantile `p` of a sample. Each tail outside the interval holds
|
|
831
|
+
* (1 - level) / 2 or less. Let X ~ Binomial(n, p). The lower rank is the largest l with P(X <= l - 1) in the tail. The upper
|
|
832
|
+
* rank is the smallest u with P(X >= u) in the tail. The values must be independent draws, or the coverage is false.
|
|
833
|
+
*
|
|
834
|
+
* A side with no such rank is infinite. Thus a small sample gives no upper bound: refer to `quantileSamples`.
|
|
835
|
+
*/
|
|
836
|
+
export declare function quantileInterval(values: readonly number[], p: number, level?: number): QuantileInterval;
|
|
837
|
+
/**
|
|
838
|
+
* This function gives the fewest independent values that give the quantile `p` an upper bound at `level`: the smallest n with
|
|
839
|
+
* p^n <= (1 - level) / 2. With that n, the upper bound is the largest value. At the level 0.95, p95 needs 72 values, p99 needs
|
|
840
|
+
* 368 values and p99.9 needs 3,688 values.
|
|
841
|
+
*/
|
|
842
|
+
export declare function quantileSamples(p: number, level?: number): number;
|
|
766
843
|
/** A central interval of a count, from `low` to `high`, both included. */
|
|
767
844
|
interface CountInterval {
|
|
768
845
|
readonly low: number;
|
|
@@ -893,5 +970,5 @@ interface CalibrationLevel {
|
|
|
893
970
|
}
|
|
894
971
|
/** The hit rate of each stated level, from the lowest level. */
|
|
895
972
|
export declare function calibration(forecasts: readonly Forecast[]): CalibrationLevel[];
|
|
896
|
-
export type { BootstrapOptions, CalibrationLevel, CategoricalDimension, CompareOptions, Comparison, ContinuousDimension, Corrected, CountInterval, Effect, Forecast, Interval, LeastSquaresFit, LeastSquaresOptions, MedianInterval, NoiseOptions, NoiseState, NoiseVerdict, OrdinalDimension, ParsedResponseRecord, PermutationOptions, PermutationResult, Proportion, QuantileMethod, ResponseAbsentPoint, ResponseBand, ResponseCurvePoint, ResponseData, ResponseDimension, ResponseDimensionBase, ResponseFactor, ResponseFault, ResponseFitOptions, ResponseFold, ResponseFoldError, ResponseHeldOutError, ResponseInputValue, ResponseMeasure, ResponseModel, ResponseModelBase, ResponsePoint, ResponseReason, ResponseRecord, ResponseSlice, ResponseSliceOptions, ResponseSlicePoint, ResponseSurface, ResponseSurfaceAbsentPoint, ResponseSurfaceOptions, ResponseSurfacePoint, ResponseTableCell, ResponseValue, ResponseViewOptions, ResponseWhen, RidgeInput, RidgeInputBase, RidgeLevelInput, RidgeNumberInput, RidgeOptions, RidgeRankInput, RidgeResponseModel, RidgeTerm, Summary, TableResponseModel };
|
|
973
|
+
export type { BootstrapOptions, CalibrationLevel, CategoricalDimension, CompareOptions, Comparison, ContinuousDimension, Corrected, CountInterval, Effect, Forecast, Interval, LeastSquaresFit, LeastSquaresOptions, MedianInterval, ModeOptions, NoiseOptions, NoiseState, NoiseVerdict, OrdinalDimension, ParsedResponseRecord, PermutationOptions, PermutationResult, Proportion, QuantileInterval, QuantileMethod, ResponseAbsentPoint, ResponseBand, ResponseCurvePoint, ResponseData, ResponseDimension, ResponseDimensionBase, ResponseFactor, ResponseFault, ResponseFitOptions, ResponseFold, ResponseFoldError, ResponseHeldOutError, ResponseInputValue, ResponseMeasure, ResponseModel, ResponseModelBase, ResponsePoint, ResponseReason, ResponseRecord, ResponseSlice, ResponseSliceOptions, ResponseSlicePoint, ResponseSurface, ResponseSurfaceAbsentPoint, ResponseSurfaceOptions, ResponseSurfacePoint, ResponseTableCell, ResponseValue, ResponseViewOptions, ResponseWhen, RidgeInput, RidgeInputBase, RidgeLevelInput, RidgeNumberInput, RidgeOptions, RidgeRankInput, RidgeResponseModel, RidgeTerm, SmoothLaw, SmoothLawScore, Summary, TableResponseModel };
|
|
897
974
|
//# sourceMappingURL=index.d.ts.map
|
package/build/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","names":[],"sources":["../src/describe.ts","../src/regression.ts","../src/bootstrap.ts","../src/ridge.ts","../src/response.ts","../src/response-parse.ts","../src/response-fit.ts","../src/tests.ts","../src/rank.ts","../src/discrete.ts","../src/compare.ts"],"mappings":";;;;;;KAOY;;wBAmBI,OAAO;wBAIP,KAAK;;;;;wBAgBL,GAAG;;;;;;;wBAgBH,eAAe,iCAAiC,WAAW,SAAQ;;wBAkBnE,SAAS,2BAA2B,WAAW,SAAQ;;;;;wBASvD,OAAO;;UAQN;WACN;WACA;;WAEA;WACA;;WAEA;WACA;;WAEA;WACA;;WAEA;;wBAGK,UAAU,4BAA4B;;;;;;UC3GrC;;WAEN;;UAGM;;WAEN;;WAEA;;;;;WAKA;;WAEA;;WAEA;;;;;;;;;;;wBAkBK,aAAa,mCAAmC,sBAAsB,UAAS,sBAA2B;UC/BzG;;WAEN;;WAEA;;WAEA;;WAEA,SAAS;;;;;;WAMT;;UAGM;;WAEN;WACA;WACA;WACA;WACA;;WAEA;;WAEA;;;;;;wBAOK,UAAU,GAAG,eAAe,KAAK,YAAY,iBAAiB,gBAAgB,UAAS,mBAAwB;;;;;;;;wBAsB/G,aAAa,GAAG,GAC9B,YAAY,KACZ,YAAY,KACZ,YAAY,YAAY,KAAK,YAAY,gBACzC,UAAS,mBACR;;;;;;wBAwBa,qBAAqB,8BAA8B,gCAAgC,UAAS,mBAAwB;;;;;;;wBAUpH,sBACd,mCACA,sBACA,UAAS,mBAAmB,sBAC3B;;wBAmFa,gBAAgB,2BAA2B,UAAS,mBAAwB;;;;;;wBAS5E,kBAAkB,GAChC,eAAe,KACf,UAAU,KAAK,cACf,YAAY,eAAe,gBAC3B,UAAS,mBACR;;;;;;;
|
|
1
|
+
{"version":3,"file":"index.d.ts","names":[],"sources":["../src/describe.ts","../src/regression.ts","../src/bootstrap.ts","../src/distribution.ts","../src/ridge.ts","../src/response.ts","../src/response-parse.ts","../src/response-fit.ts","../src/tests.ts","../src/rank.ts","../src/discrete.ts","../src/compare.ts"],"mappings":";;;;;;KAOY;;wBAmBI,OAAO;wBAIP,KAAK;;;;;wBAgBL,GAAG;;;;;;;wBAgBH,eAAe,iCAAiC,WAAW,SAAQ;;wBAkBnE,SAAS,2BAA2B,WAAW,SAAQ;;;;;wBASvD,OAAO;;UAQN;WACN;WACA;;WAEA;WACA;;WAEA;WACA;;WAEA;WACA;;WAEA;;wBAGK,UAAU,4BAA4B;;;;;;UC3GrC;;WAEN;;UAGM;;WAEN;;WAEA;;;;;WAKA;;WAEA;;WAEA;;;;;;;;;;;wBAkBK,aAAa,mCAAmC,sBAAsB,UAAS,sBAA2B;UC/BzG;;WAEN;;WAEA;;WAEA;;WAEA,SAAS;;;;;;WAMT;;UAGM;;WAEN;WACA;WACA;WACA;WACA;;WAEA;;WAEA;;;;;;wBAOK,UAAU,GAAG,eAAe,KAAK,YAAY,iBAAiB,gBAAgB,UAAS,mBAAwB;;;;;;;;wBAsB/G,aAAa,GAAG,GAC9B,YAAY,KACZ,YAAY,KACZ,YAAY,YAAY,KAAK,YAAY,gBACzC,UAAS,mBACR;;;;;;wBAwBa,qBAAqB,8BAA8B,gCAAgC,UAAS,mBAAwB;;;;;;;wBAUpH,sBACd,mCACA,sBACA,UAAS,mBAAmB,sBAC3B;;wBAmFa,gBAAgB,2BAA2B,UAAS,mBAAwB;;;;;;wBAS5E,kBAAkB,GAChC,eAAe,KACf,UAAU,KAAK,cACf,YAAY,eAAe,gBAC3B,UAAS,mBACR;;KCxMS;;UAGK;WACN,KAAK;;WAEL;;;UAIM;;;;;WAKN;;WAEA;;;;;;;;;;;;;;;;;;;wBA0MK,UAAU,2BAA2B,UAAS;;;;;;;;wBAqB9C,gBAAgB,qCAAqC;;;;;;wBAwBrD,cAAc,4BAA4B;;;;;;;UCxQzC;;WAEN;;;;;;;;;;;;wBAoBK,WAAW,MAAM,mBAAmB,QAAQ,mBAAmB,gBAAgB,UAAS;;qBCd3F;;KAGD;;KAGA,qBAAqB;;UAGhB;;WAEN;WACA,iBAAiB;;;UAIX;;WAEN;;WAEA;;WAEA;;WAEA,OAAO;;;UAID,4BAA4B;WAClC;;WAEA;;WAEA;;;UAIM,yBAAyB;WAC/B;WACA;;WAEA;;;UAIM,6BAA6B;WACnC;WACA;;;KAIC,oBAAoB,sBAAsB,mBAAmB;;UAGxD;;WAEN;WACA;;WAEA;;WAEA;;;;;WAKA;;;UAIM;;WAEN;;WAEA;;;UAIM;WACN;;WAEA,QAAQ,SAAS,eAAe;;WAEhC,UAAU,SAAS;;WAEnB;;WAEA,SAAS,SAAS;;;KAIjB;WAA4B;WAA4B;;WAA6B;WAA4B,OAAO;;;UAGnH;WACN;;WAEA;;;UAIM,yBAAyB;WAC/B;WACA;WACA;WACA;;;UAIM,uBAAuB;WAC7B;WACA;WACA;WACA;;;UAIM,wBAAwB;WAC9B;WACA;;;KAIC,aAAa,mBAAmB,iBAAiB;;UAG5C;WACN,kBAAkB;WAClB;;;UAIM;;WAEN;;WAEA;;WAEA;;;UAIM,2BAA2B;WACjC;;WAEA;WACA,iBAAiB;WACjB;WACA,gBAAgB;;;UAIV;;WAEN,aAAa;;WAEb;;WAEA;;;UAIM,2BAA2B;WACjC;WACA;;WAEA;WACA,gBAAgB;;WAEhB;;;KAIC,gBAAgB,qBAAqB;;UAGhC;;WAEN;;WAEA;;WAEA;;WAEA;;;UAIM;;WAEN;;WAEA,QAAQ,SAAS,eAAe;;;;;WAKhC,cAAc,SAAS,eAAe,SAAS;;;UAIzC;WACN,eAAe;WACf;WACA,qBAAqB;WACrB,mBAAmB;;WAEnB,mBAAmB;WACnB,iBAAiB;;;UAIX,uBAAuB;WAC7B,iBAAiB;WACjB,gBAAgB;;;UAIV;;WAEN;;WAEA;;WAEA;;WAEA;;WAEA;;WAEA;;;UAIM;;WAEN,OAAO,SAAS,eAAe;;WAE/B;;;;;WAKA;;WAEA;;;UAIM,6BAA6B;;WAEnC;;;UAIM,+BAA+B;;WAErC;;;;;;UAOM;WACN;WACA;WACA;;;;;;;WAOA;;WAEA;;;UAIM,2BAA2B;;WAEjC,IAAI;;;UAIE;WACN;;WAEA,IAAI;WACJ;WACA;;;UAIM;WACN;;WAEA,IAAI;;WAEJ;WACA;;;UAIM;WACN;WACA;;WAEA,MAAM,SAAS,eAAe;WAC9B,gBAAgB;WAChB,iBAAiB;;WAEjB,iBAAiB;WACjB,OAAO;;WAEP;;;UAIM;WACN;;WAEA,cAAc,eAAe;WAC7B;WACA;;;UAIM;WACN;;WAEA,cAAc,eAAe;;WAE7B;WACA;;;UAIM;WACN;WACA;;WAEA,MAAM,SAAS,eAAe;WAC9B,YAAY;WACZ,YAAY;;WAEZ,0BAA0B;WAC1B,iBAAiB;;WAEjB,iBAAiB;WACjB,OAAO;;WAEP;;;wBAuBK,UAAU,WAAW,mBAAmB,QAAQ,SAAS;;;;;;;;;;;wBA4FzD,gBAAgB,OAAO,eAAe,QAAQ,SAAS,eAAe;;;;;wBAQtE,aAAa,QAAQ,KAAK,0BAA0B,kBAAkB;;;;;;;;;;;;wBAqCtE,eAAe,QAAQ,gBAAgB,iBAAiB,yBAAyB;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;wBAgPjF,cAAc,QAAQ,gBAAgB,iBAAiB,SAAS,uBAAuB;;;;;;;wBAsBvF,gBAAgB,QAAQ,gBAAgB,iBAAiB,SAAS,yBAAyB;;;;;;;;wBA4B3F,cAAc,QAAQ,KAAK,+BAA+B,0CAA0C;;qBC3yBvG;;UAGI;;WAEN;;WAEA;;;KAIC;WAAkC;WAAmB,QAAQ;;WAA8B;WAAoB,iBAAiB;;;;;;;;;;;;;;;;;;;;;;;;;;wBAgf5H,oBAAoB,iBAAiB;;UC/dpC;;WAEN;;WAEA;;WAEA;;WAEA;;;;;;;;;WASA;;WAEA,2BAA2B;;WAE3B;;WAEA;;WAEA;;;;;WAKA;;;qBAIE;;qBAGA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;wBAoTG,YAAY,MAAM,cAAc,UAAS,qBAA0B;;;;;;wBCvVnE,SAAS,gBAAgB,eAAe;;;;;wBAaxC,cAAc,WAAW,gBAAc;;qBAO1C;;UAGI;WACN;WACA;;;;;;wBAOK,OAAO,WAAW,WAAW,aAAkB;;UAc9C;;WAEN;;WAEA;;WAEA;;;;;;;wBAQK,KAAK,4BAA4B,iBAAe;UAa/C;;WAEN;;WAEA;;WAEA,aAAa;;UAGP;;WAEN;;WAEA;WACA;;;;;;wBAOK,kBAAkB,8BAA8B,6BAA6B,UAAS,qBAA0B;;wBCtHhH,MAAM;;;;;wBAgBN,SAAS,sBAAsB;;qBAmBlC;;;;;;wBAQG,WAAW,sBAAsB,sBAAsB;;UC1BtD;WACN;WACA;;WAEA;;WAEA;;;;;;;;;;wBAWK,eAAe,2BAA2B,iBAAe;;UAqBxD;;WAEN;;WAEA;;WAEA;;WAEA;;WAEA;;;;;;;;;wBAUK,iBAAiB,2BAA2B,WAAW,iBAAe;;;;;;wBAsCtE,gBAAgB,WAAW;;UAW1B;WACN;WACA;;WAEA;;;;;;;wBAQK,iBAAiB,WAAW,WAAW,iBAAe;;;;;;wBA2BtD,yBAAyB,mBAAmB,WAAW,iBAAe;UCvKrE;;WAEN;;WAEA;;;;;;WAMA;;;;;;wBAgBK,WAAW,2BAA2B,UAAS;;;;;KASnD;;UAGK;;WAEN;WACA;;WAEA;;WAEA;WACA;;WAEA;WACA,OAAO;;;;;;;;wBASF,aAAa,2BAA2B,UAAS;WAA0B;IAAqC;UAW/G;;WAEN;;WAEA;;WAEA;;;UAIM;WACN;WACA;;WAEA;;WAEA;;UAGM;;WAEN;WACA;;WAEA;;WAEA;;WAEA;;WAEA,QAAQ;;WAER;;WAEA;;WAEA;WACA,cAAc;;WAEd;WACA,cAAc;;;;;;;;wBAST,cAAc,2BAA2B,UAAS,iBAAsB;;UA6CvE;WACN;WACA;;;wBASK,MAAM,oBAAoB;;UAQzB;WACN;WACA;WACA;;WAEA;;WAEA,UAAU;;;wBAIL,YAAY,oBAAoB,aAAa"}
|
package/build/index.js
CHANGED
|
@@ -363,6 +363,263 @@ function bootstrapClusters(rows, cluster, statistic, options = {}) {
|
|
|
363
363
|
estimate
|
|
364
364
|
};
|
|
365
365
|
}
|
|
366
|
+
// src/distribution.ts
|
|
367
|
+
/**
|
|
368
|
+
* The shape of a sample: the count of its modes, and the smooth law with the highest likelihood. Use them to read the
|
|
369
|
+
* law that a generator really emits, from its values and not from its label.
|
|
370
|
+
*/
|
|
371
|
+
/** The points of the grid of the density estimate. */
|
|
372
|
+
const GRID = 96;
|
|
373
|
+
/** The width of the kernel, in the factor of Silverman's rule. The value 0.9 of that rule hides close modes. */
|
|
374
|
+
const KERNEL_FACTOR = .45;
|
|
375
|
+
/** The share of the values in a span of neighbours, as a factor of `n ** -0.2`. */
|
|
376
|
+
const SPAN_FACTOR = .1;
|
|
377
|
+
/** The quantile of the spans. A low quantile stays in the clusters when most spans cross a gap. */
|
|
378
|
+
const SPAN_QUANTILE = .25;
|
|
379
|
+
/**
|
|
380
|
+
* The tight width replaces the width from the quartiles under this ratio. In seeded smooth samples the ratio stays over
|
|
381
|
+
* 0.2, except for a heavy tail, and in tight clusters it stays under 0.02.
|
|
382
|
+
*/
|
|
383
|
+
const TIGHT_RATIO = .1;
|
|
384
|
+
/** The kernels stop at this count of widths. */
|
|
385
|
+
const KERNEL_REACH = 3;
|
|
386
|
+
const HALF_LOG_2PI_E = .5 * Math.log(2 * Math.PI * Math.E);
|
|
387
|
+
/** Two values closer than this share of the range are one value, so the noise of a float makes no new value. */
|
|
388
|
+
const SAME_VALUE = 1e-9;
|
|
389
|
+
/** The most distinct values of a direct read of their counts. The grid resolves no more than this count of clusters. */
|
|
390
|
+
const FEW_VALUES = GRID / 4;
|
|
391
|
+
/** The mean count of each value in a direct read. Under it the values are a small sample, and the kernel reads them. */
|
|
392
|
+
const MIN_REPEATS = 4;
|
|
393
|
+
/** The fewest values that show the shape of a law. A sample with fewer values gives one mode for each value. */
|
|
394
|
+
const MIN_BELL_VALUES = 4;
|
|
395
|
+
/** The ratio of the highest third of the counts to the lowest third, under which the counts are flat. */
|
|
396
|
+
const FLAT_RATIO = 3;
|
|
397
|
+
/** The ratio of the means of the three parts of the list of counts, under which the counts have no shape. */
|
|
398
|
+
const SHAPE_RATIO = 1.75;
|
|
399
|
+
/** A value with under this share of the sample is a tail. A rare value does not change the shape of the counts. */
|
|
400
|
+
const RARE_VALUE = .01;
|
|
401
|
+
/** The fewest values of a hump of a rounded law. When most humps are narrower, the values are a discrete law. */
|
|
402
|
+
const MIN_HUMP_VALUES = 3;
|
|
403
|
+
/**
|
|
404
|
+
* This function gives the distances of the values from one end of the sample, in order. It takes the end that gives the
|
|
405
|
+
* smaller list, so `x` and `-x` give one list and one count.
|
|
406
|
+
*/
|
|
407
|
+
function fromOneEnd(values) {
|
|
408
|
+
const order = sorted(values);
|
|
409
|
+
const low = order[0];
|
|
410
|
+
const high = order[order.length - 1];
|
|
411
|
+
const rising = order.map((x) => x - low);
|
|
412
|
+
const falling = order.map((x) => high - x).reverse();
|
|
413
|
+
const first = rising.findIndex((x, i) => x !== falling[i]);
|
|
414
|
+
return first >= 0 && falling[first] < rising[first] ? falling : rising;
|
|
415
|
+
}
|
|
416
|
+
/**
|
|
417
|
+
* This function gives the width of the kernel. The width from the quartiles follows the distance between clusters, so
|
|
418
|
+
* it hides many tight clusters. A low quantile of the spans of neighbouring values follows the width of a cluster.
|
|
419
|
+
* The function uses the tight width only when it is far under the other width, so a smooth sample keeps its width.
|
|
420
|
+
* The tight width stays over the grid step. No width goes under the median gap of the distinct values of a tight
|
|
421
|
+
* sample. A step of rounding is no gap between clusters, so a rounded smooth sample keeps one mode at each size.
|
|
422
|
+
*/
|
|
423
|
+
function kernelWidth(values, low, high) {
|
|
424
|
+
const count = values.length;
|
|
425
|
+
const order = sorted(values);
|
|
426
|
+
const scale = Math.min(quantile(values, .75) - quantile(values, .25), high - low) / 1.34;
|
|
427
|
+
const wide = KERNEL_FACTOR * Math.max(scale, (high - low) / GRID) * count ** -.2;
|
|
428
|
+
const neighbours = Math.min(count - 1, Math.ceil(SPAN_FACTOR * count ** .8));
|
|
429
|
+
const tight = quantile(order.slice(neighbours).map((x, i) => x - order[i]), SPAN_QUANTILE);
|
|
430
|
+
if (tight >= TIGHT_RATIO * wide) return wide;
|
|
431
|
+
const distinct = order.filter((x, i) => i === 0 || x !== order[i - 1]);
|
|
432
|
+
const resolution = quantile(distinct.slice(1).map((x, i) => x - distinct[i]), .5);
|
|
433
|
+
return Math.max(resolution, Math.min(wide, Math.max(tight, (high - low) / GRID)));
|
|
434
|
+
}
|
|
435
|
+
/** This function gives the Gaussian kernel density of the values on a grid. */
|
|
436
|
+
function densityGrid(values, low, high) {
|
|
437
|
+
const width = kernelWidth(values, low, high);
|
|
438
|
+
const start = low - KERNEL_REACH * width;
|
|
439
|
+
const step = (high - low + 6 * width) / 95;
|
|
440
|
+
const density = new Float64Array(GRID);
|
|
441
|
+
for (const x of values) {
|
|
442
|
+
const first = Math.max(0, Math.floor((x - KERNEL_REACH * width - start) / step));
|
|
443
|
+
const last = Math.min(95, Math.ceil((x + KERNEL_REACH * width - start) / step));
|
|
444
|
+
for (let i = first; i <= last; i++) {
|
|
445
|
+
const u = (start + i * step - x) / width;
|
|
446
|
+
density[i] += Math.exp(-.5 * u * u);
|
|
447
|
+
}
|
|
448
|
+
}
|
|
449
|
+
return density;
|
|
450
|
+
}
|
|
451
|
+
/**
|
|
452
|
+
* This function gives the humps of a series. It first gives each rise and fall one hump. Then it joins the two neighbours
|
|
453
|
+
* with the closest valley, and again, until each valley has a closeness under the limit. The result has no direction,
|
|
454
|
+
* because the order of the joins comes from the depth of the valleys and not from their position.
|
|
455
|
+
*/
|
|
456
|
+
function humpsOf(series, closeness, limit) {
|
|
457
|
+
const humps = [];
|
|
458
|
+
const valleys = [];
|
|
459
|
+
let index = 0;
|
|
460
|
+
while (index < series.length) {
|
|
461
|
+
let top = index;
|
|
462
|
+
while (top + 1 < series.length && series[top + 1] >= series[top]) top++;
|
|
463
|
+
let end = top;
|
|
464
|
+
while (end + 1 < series.length && series[end + 1] < series[end]) end++;
|
|
465
|
+
let mass = 0;
|
|
466
|
+
for (let i = index; i <= end; i++) mass += series[i];
|
|
467
|
+
humps.push({
|
|
468
|
+
top: series[top],
|
|
469
|
+
mass,
|
|
470
|
+
points: end + 1 - index
|
|
471
|
+
});
|
|
472
|
+
if (end + 1 < series.length) valleys.push(series[end]);
|
|
473
|
+
index = end + 1;
|
|
474
|
+
}
|
|
475
|
+
while (valleys.length > 0) {
|
|
476
|
+
const scores = valleys.map((valley, i) => closeness(Math.min(humps[i].top, humps[i + 1].top), valley));
|
|
477
|
+
const closest = scores.indexOf(Math.max(...scores));
|
|
478
|
+
if (scores[closest] < limit) break;
|
|
479
|
+
const [left, right] = [humps[closest], humps[closest + 1]];
|
|
480
|
+
humps.splice(closest, 2, {
|
|
481
|
+
top: Math.max(left.top, right.top),
|
|
482
|
+
mass: left.mass + right.mass,
|
|
483
|
+
points: left.points + right.points
|
|
484
|
+
});
|
|
485
|
+
valleys.splice(closest, 1);
|
|
486
|
+
}
|
|
487
|
+
return humps;
|
|
488
|
+
}
|
|
489
|
+
/** This function gives the count of each distinct value, in the order of the values. */
|
|
490
|
+
function valueCounts(values, low, high) {
|
|
491
|
+
const tie = SAME_VALUE * (high - low);
|
|
492
|
+
const counts = [];
|
|
493
|
+
let last = -Infinity;
|
|
494
|
+
for (const x of sorted(values)) {
|
|
495
|
+
if (x - last > tie) {
|
|
496
|
+
counts.push(0);
|
|
497
|
+
last = x;
|
|
498
|
+
}
|
|
499
|
+
counts[counts.length - 1]++;
|
|
500
|
+
}
|
|
501
|
+
return counts;
|
|
502
|
+
}
|
|
503
|
+
/**
|
|
504
|
+
* This function tells that the counts are flat. The highest third of the counts stays under `FLAT_RATIO` times the
|
|
505
|
+
* lowest third, so a list of heavy and light values is not flat. The means of the first, the middle and the last third
|
|
506
|
+
* of the list stay under `SHAPE_RATIO` of each other. The noise of a sample cancels in those means, and a bell does not.
|
|
507
|
+
*/
|
|
508
|
+
function flatCounts(counts) {
|
|
509
|
+
const order = [...counts].sort((a, b) => a - b);
|
|
510
|
+
const third = Math.ceil(order.length / 3);
|
|
511
|
+
if (mean(order.slice(-third)) >= FLAT_RATIO * mean(order.slice(0, third))) return false;
|
|
512
|
+
const parts = [
|
|
513
|
+
counts.slice(0, third),
|
|
514
|
+
counts.slice(third, -third),
|
|
515
|
+
counts.slice(-third)
|
|
516
|
+
].filter((part) => part.length > 0).map((part) => mean(part));
|
|
517
|
+
return Math.max(...parts) < SHAPE_RATIO * Math.min(...parts);
|
|
518
|
+
}
|
|
519
|
+
/** This function gives the closeness of a count to a valley: the negative of its height over the noise of a sample. */
|
|
520
|
+
const countCloseness = (lower, valley) => -(lower - valley) / Math.sqrt(lower + valley);
|
|
521
|
+
/**
|
|
522
|
+
* This function counts the modes of a sample with few distinct values, from the counts of those values. The result does
|
|
523
|
+
* not change with the size of the sample. Flat counts are a discrete law with one mode for each value, so the noise of
|
|
524
|
+
* a sample does not take a value out. Counts with one rise and one fall are a rounded smooth law with one mode.
|
|
525
|
+
*/
|
|
526
|
+
function fewValueModes(allCounts, total, minShare) {
|
|
527
|
+
const counts = allCounts.filter((count) => count / total >= RARE_VALUE);
|
|
528
|
+
const each = counts.map((count) => count / total);
|
|
529
|
+
const overLimit = (masses, limit) => masses.filter((share) => share >= limit).length;
|
|
530
|
+
if (counts.length < MIN_BELL_VALUES) return overLimit(each, Math.min(minShare, Math.max(...each) / 2));
|
|
531
|
+
if (flatCounts(counts)) return overLimit(each, Math.min(minShare, quantile(each, .5) / FLAT_RATIO));
|
|
532
|
+
const humps = humpsOf(counts, countCloseness, -3);
|
|
533
|
+
const masses = 2 * humps.filter((hump) => hump.points < MIN_HUMP_VALUES).length > humps.length ? each : humps.map((hump) => hump.mass / total);
|
|
534
|
+
return overLimit(masses, Math.min(minShare, Math.max(...masses) / 2));
|
|
535
|
+
}
|
|
536
|
+
/**
|
|
537
|
+
* This function gives the mass of each mode from a kernel estimate on a grid. Two peaks stay apart when the valley
|
|
538
|
+
* between them is under `valleyRatio` of the lower peak.
|
|
539
|
+
*/
|
|
540
|
+
function kernelMasses(values, low, high, valleyRatio) {
|
|
541
|
+
const density = densityGrid(values, low, high);
|
|
542
|
+
const total = density.reduce((sum, d) => sum + d, 0);
|
|
543
|
+
return humpsOf(density, (lower, valley) => valley / lower, valleyRatio).map((hump) => hump.mass / total);
|
|
544
|
+
}
|
|
545
|
+
/**
|
|
546
|
+
* This function counts the modes of a sample. A mode counts when it holds `minShare` of the mass. The default counts the
|
|
547
|
+
* modes that hold 10 percent, and a caller who needs all modes passes a smaller value. The result is at least 1. The
|
|
548
|
+
* result is the same for `x` and `-x`.
|
|
549
|
+
*
|
|
550
|
+
* A sample with 24 or fewer distinct values, and 4 or more copies of each value on average, gets a direct read.
|
|
551
|
+
* That read uses the counts of the values, and it does not change with the size of the sample.
|
|
552
|
+
* It gives one mode for each value of a flat discrete law. So even groups give the count of the groups, and so does a
|
|
553
|
+
* rounded uniform law with 24 or fewer steps. It gives one mode for each hump of a rounded smooth law.
|
|
554
|
+
* The direct read of a sample with 3 or fewer values gives one mode for each value that holds the share.
|
|
555
|
+
*
|
|
556
|
+
* A kernel estimate on a grid reads each other sample. The grid has 96 points, so the kernel joins more than
|
|
557
|
+
* approximately 24 evenly spaced clusters. A tight cluster also needs more than approximately `n ** 0.8 / 8` values for
|
|
558
|
+
* a sample of `n` values. A cluster with an SD over 5 percent of the gap to the next cluster can join it.
|
|
559
|
+
*
|
|
560
|
+
* The function throws a RangeError for an empty list or for a value that is not finite.
|
|
561
|
+
*/
|
|
562
|
+
function modeCount(values, options = {}) {
|
|
563
|
+
const { minShare = .1, valleyRatio = .5 } = options;
|
|
564
|
+
if (values.length === 0) throw new RangeError("modeCount: the list is empty.");
|
|
565
|
+
for (const v of values) if (!Number.isFinite(v)) throw new RangeError("modeCount: each value must be finite.");
|
|
566
|
+
const distances = fromOneEnd(values);
|
|
567
|
+
const high = distances[distances.length - 1];
|
|
568
|
+
if (high === 0) return 1;
|
|
569
|
+
const counts = valueCounts(distances, 0, high);
|
|
570
|
+
if (counts.length <= FEW_VALUES && values.length >= MIN_REPEATS * counts.length) return fewValueModes(counts, values.length, minShare);
|
|
571
|
+
const masses = kernelMasses(distances, 0, high, valleyRatio);
|
|
572
|
+
const limit = Math.min(minShare, Math.max(...masses) / 2);
|
|
573
|
+
return masses.filter((share) => share >= limit).length;
|
|
574
|
+
}
|
|
575
|
+
/**
|
|
576
|
+
* This function gives the mean log density of the values under each smooth law, at the closed-form fit of that law.
|
|
577
|
+
* Each law has 2 fitted numbers, so the function adds no penalty. The uniform, the shifted exponential and the Pareto
|
|
578
|
+
* law take the smallest value as their lower limit. This fitted end helps them at small sizes. All values are exact
|
|
579
|
+
* maximum likelihoods.
|
|
580
|
+
* The function throws a RangeError for an empty list, for a value that is not positive and finite, or for equal values.
|
|
581
|
+
*/
|
|
582
|
+
function smoothLawScores(values) {
|
|
583
|
+
if (values.length === 0) throw new RangeError("smoothLawScores: the list is empty.");
|
|
584
|
+
for (const v of values) if (!(v > 0 && Number.isFinite(v))) throw new RangeError("smoothLawScores: each value must be positive and finite.");
|
|
585
|
+
const logs = values.map((v) => Math.log(v));
|
|
586
|
+
const low = quantile(values, 0);
|
|
587
|
+
const high = quantile(values, 1);
|
|
588
|
+
if (low === high) throw new RangeError("smoothLawScores: the values are equal, so no law has a density.");
|
|
589
|
+
const meanLog = mean(logs);
|
|
590
|
+
const toExact = .5 * Math.log(values.length / (values.length - 1));
|
|
591
|
+
const alpha = 1 / (meanLog - Math.log(low));
|
|
592
|
+
return [
|
|
593
|
+
{
|
|
594
|
+
law: "uniform",
|
|
595
|
+
meanLogDensity: -Math.log(high - low)
|
|
596
|
+
},
|
|
597
|
+
{
|
|
598
|
+
law: "normal",
|
|
599
|
+
meanLogDensity: -Math.log(sd(values)) - HALF_LOG_2PI_E + toExact
|
|
600
|
+
},
|
|
601
|
+
{
|
|
602
|
+
law: "exponential",
|
|
603
|
+
meanLogDensity: -Math.log(mean(values) - low) - 1
|
|
604
|
+
},
|
|
605
|
+
{
|
|
606
|
+
law: "log-normal",
|
|
607
|
+
meanLogDensity: -Math.log(sd(logs)) - HALF_LOG_2PI_E - meanLog + toExact
|
|
608
|
+
},
|
|
609
|
+
{
|
|
610
|
+
law: "pareto",
|
|
611
|
+
meanLogDensity: Math.log(alpha) + alpha * Math.log(low) - (alpha + 1) * meanLog
|
|
612
|
+
}
|
|
613
|
+
];
|
|
614
|
+
}
|
|
615
|
+
/**
|
|
616
|
+
* This function gives the smooth law with the highest likelihood. Under 300 values the result is not reliable: the
|
|
617
|
+
* normal and the log-normal law lose to the laws with a fitted end. The README gives the measured shares of correct
|
|
618
|
+
* results. The function throws as `smoothLawScores` does.
|
|
619
|
+
*/
|
|
620
|
+
function bestSmoothLaw(values) {
|
|
621
|
+
return smoothLawScores(values).reduce((best, score) => score.meanLogDensity > best.meanLogDensity ? score : best).law;
|
|
622
|
+
}
|
|
366
623
|
// src/ridge.ts
|
|
367
624
|
/**
|
|
368
625
|
* The relative tolerance of the Cholesky pivots. A pivot is the squared length of the part of a column that is
|
|
@@ -2129,8 +2386,8 @@ function kendallTau(a, b, seed = 1) {
|
|
|
2129
2386
|
}
|
|
2130
2387
|
// src/discrete.ts
|
|
2131
2388
|
/**
|
|
2132
|
-
* The intervals that come from a discrete distribution. The interval of a median comes from the order
|
|
2133
|
-
* The central interval of a count comes from a binomial or a negative binomial distribution.
|
|
2389
|
+
* The intervals that come from a discrete distribution. The interval of a median or of a quantile comes from the order
|
|
2390
|
+
* statistics. The central interval of a count comes from a binomial or a negative binomial distribution.
|
|
2134
2391
|
*/
|
|
2135
2392
|
function whole(name, value) {
|
|
2136
2393
|
if (!Number.isSafeInteger(value) || value < 0) throw new RangeError(`${name}: ${value} is not a whole number of 0 or more.`);
|
|
@@ -2184,6 +2441,58 @@ function medianInterval(values, level = .95) {
|
|
|
2184
2441
|
};
|
|
2185
2442
|
}
|
|
2186
2443
|
/**
|
|
2444
|
+
* This function gives the distribution-free interval of the quantile `p` of a sample. Each tail outside the interval holds
|
|
2445
|
+
* (1 - level) / 2 or less. Let X ~ Binomial(n, p). The lower rank is the largest l with P(X <= l - 1) in the tail. The upper
|
|
2446
|
+
* rank is the smallest u with P(X >= u) in the tail. The values must be independent draws, or the coverage is false.
|
|
2447
|
+
*
|
|
2448
|
+
* A side with no such rank is infinite. Thus a small sample gives no upper bound: refer to `quantileSamples`.
|
|
2449
|
+
*/
|
|
2450
|
+
function quantileInterval(values, p, level = .95) {
|
|
2451
|
+
if (values.length === 0) throw new RangeError("quantileInterval: the list is empty.");
|
|
2452
|
+
for (const v of values) if (Number.isNaN(v)) throw new RangeError("quantileInterval: the list has a NaN.");
|
|
2453
|
+
if (!(p > 0 && p < 1)) throw new RangeError(`quantileInterval: p is more than 0 and less than 1, not ${p}.`);
|
|
2454
|
+
const tail = levelOf("quantileInterval", level);
|
|
2455
|
+
const s = sorted(values);
|
|
2456
|
+
const n = s.length;
|
|
2457
|
+
const terms = binomialTerms(n, p);
|
|
2458
|
+
let lowRank = 0;
|
|
2459
|
+
let left = 0;
|
|
2460
|
+
let below = terms[0];
|
|
2461
|
+
while (lowRank < n && below <= tail) {
|
|
2462
|
+
lowRank += 1;
|
|
2463
|
+
left = below;
|
|
2464
|
+
below += terms[lowRank];
|
|
2465
|
+
}
|
|
2466
|
+
let highRank = n + 1;
|
|
2467
|
+
let right = 0;
|
|
2468
|
+
let above = terms[n];
|
|
2469
|
+
while (highRank > 1 && above <= tail) {
|
|
2470
|
+
highRank -= 1;
|
|
2471
|
+
right = above;
|
|
2472
|
+
above += terms[highRank - 1];
|
|
2473
|
+
}
|
|
2474
|
+
return {
|
|
2475
|
+
low: lowRank === 0 ? -Infinity : s[lowRank - 1],
|
|
2476
|
+
high: highRank === n + 1 ? Infinity : s[highRank - 1],
|
|
2477
|
+
lowRank,
|
|
2478
|
+
highRank,
|
|
2479
|
+
coverage: Math.max(0, 1 - left - right)
|
|
2480
|
+
};
|
|
2481
|
+
}
|
|
2482
|
+
/**
|
|
2483
|
+
* This function gives the fewest independent values that give the quantile `p` an upper bound at `level`: the smallest n with
|
|
2484
|
+
* p^n <= (1 - level) / 2. With that n, the upper bound is the largest value. At the level 0.95, p95 needs 72 values, p99 needs
|
|
2485
|
+
* 368 values and p99.9 needs 3,688 values.
|
|
2486
|
+
*/
|
|
2487
|
+
function quantileSamples(p, level = .95) {
|
|
2488
|
+
if (!(p > 0 && p < 1)) throw new RangeError(`quantileSamples: p is more than 0 and less than 1, not ${p}.`);
|
|
2489
|
+
const tail = levelOf("quantileSamples", level);
|
|
2490
|
+
const [logTail, logP] = [Math.log(tail), Math.log(p)];
|
|
2491
|
+
let n = Math.max(1, Math.ceil(logTail / logP));
|
|
2492
|
+
while (n > 1 && (n - 1) * logP <= logTail) n -= 1;
|
|
2493
|
+
return n;
|
|
2494
|
+
}
|
|
2495
|
+
/**
|
|
2187
2496
|
* The central interval of a count X ~ Binomial(n, p). `low` is the smallest k with P(X <= k) > (1 - level) / 2.
|
|
2188
2497
|
* `high` is the largest k with P(X >= k) > (1 - level) / 2. Each tail outside the interval holds (1 - level) / 2
|
|
2189
2498
|
* or less. The two tails are separate sums, so that a small upper tail keeps its digits.
|
|
@@ -2382,6 +2691,6 @@ function calibration(forecasts) {
|
|
|
2382
2691
|
interval: wilson(hits, n)
|
|
2383
2692
|
}));
|
|
2384
2693
|
}
|
|
2385
|
-
export { KENDALL_EXACT, RESPONSE_LAMBDAS, RESPONSE_MAX_POWER, RESPONSE_MAX_TERMS, RESPONSE_RECORDS_SCHEMA, Z95, binomialInterval, bootstrap, bootstrapClusters, bootstrapLeastSquares, bootstrapMedian, bootstrapMedianRatio, bootstrapTwo, brier, calibration, compareRatios, fitResponse, heldOutError, heldOutErrorAt, holm, kendallTau, leastSquares, mean, median, medianInterval, negativeBinomialInterval, noiseFloor, noiseVerdict, pairedPermutation, parseResponseRecord, predictResponse, quantile, quantileSorted, ranks, responseCells, ridgeSolve, sd, signTest, signTestPower, sliceResponse, sorted, spearman, summarize, surfaceResponse, whenHolds, wilson };
|
|
2694
|
+
export { KENDALL_EXACT, RESPONSE_LAMBDAS, RESPONSE_MAX_POWER, RESPONSE_MAX_TERMS, RESPONSE_RECORDS_SCHEMA, Z95, bestSmoothLaw, binomialInterval, bootstrap, bootstrapClusters, bootstrapLeastSquares, bootstrapMedian, bootstrapMedianRatio, bootstrapTwo, brier, calibration, compareRatios, fitResponse, heldOutError, heldOutErrorAt, holm, kendallTau, leastSquares, mean, median, medianInterval, modeCount, negativeBinomialInterval, noiseFloor, noiseVerdict, pairedPermutation, parseResponseRecord, predictResponse, quantile, quantileInterval, quantileSamples, quantileSorted, ranks, responseCells, ridgeSolve, sd, signTest, signTestPower, sliceResponse, smoothLawScores, sorted, spearman, summarize, surfaceResponse, whenHolds, wilson };
|
|
2386
2695
|
|
|
2387
2696
|
//# sourceMappingURL=index.js.map
|