pond-ts 0.24.0 → 0.26.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +70 -1
- package/dist/batch/aggregate-columns.js +18 -10
- package/dist/batch/operators/column-builders.d.ts +31 -0
- package/dist/batch/operators/column-builders.js +103 -0
- package/dist/batch/time-series.d.ts +8 -0
- package/dist/batch/time-series.js +79 -66
- package/dist/batch/validate.js +7 -1
- package/dist/column.js +91 -21
- package/dist/columnar/chunked-column.d.ts +10 -0
- package/dist/columnar/chunked-column.js +29 -3
- package/dist/columnar/column.d.ts +19 -1
- package/dist/columnar/column.js +40 -5
- package/dist/reducers/avg.js +40 -13
- package/dist/reducers/count.js +30 -4
- package/dist/reducers/index.js +34 -2
- package/dist/reducers/max.js +37 -10
- package/dist/reducers/min.js +37 -23
- package/dist/reducers/percentile.d.ts +12 -28
- package/dist/reducers/percentile.js +49 -43
- package/dist/reducers/stdev.js +115 -29
- package/dist/reducers/sum.js +29 -5
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -7,7 +7,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
|
7
7
|
file covers both packages. Pre-1.0: minor bumps may include new features and
|
|
8
8
|
type-level changes; patch bumps are strictly additive.
|
|
9
9
|
|
|
10
|
-
[Unreleased]: https://github.com/pjm17971/pond-ts/compare/v0.
|
|
10
|
+
[Unreleased]: https://github.com/pjm17971/pond-ts/compare/v0.26.0...HEAD
|
|
11
|
+
[0.26.0]: https://github.com/pjm17971/pond-ts/compare/v0.25.0...v0.26.0
|
|
12
|
+
[0.25.0]: https://github.com/pjm17971/pond-ts/compare/v0.24.0...v0.25.0
|
|
11
13
|
[0.24.0]: https://github.com/pjm17971/pond-ts/compare/v0.23.0...v0.24.0
|
|
12
14
|
[0.23.0]: https://github.com/pjm17971/pond-ts/compare/v0.22.0...v0.23.0
|
|
13
15
|
[0.22.0]: https://github.com/pjm17971/pond-ts/compare/v0.21.0...v0.22.0
|
|
@@ -18,6 +20,73 @@ type-level changes; patch bumps are strictly additive.
|
|
|
18
20
|
|
|
19
21
|
## [Unreleased]
|
|
20
22
|
|
|
23
|
+
## [0.26.0] — 2026-06-15
|
|
24
|
+
|
|
25
|
+
### Changed
|
|
26
|
+
|
|
27
|
+
- **`rolling(...)` now builds its output columns directly instead of
|
|
28
|
+
materializing events.** The rolling family was the last batch operator still
|
|
29
|
+
assembling a row per event and re-validating/re-packing it through the
|
|
30
|
+
constructor; it now reads the key axis and source values straight off the
|
|
31
|
+
columnar store and writes the result columns via trusted construction. The
|
|
32
|
+
result is unchanged for the common (scalar) cases. Measured: `rolling` with
|
|
33
|
+
`avg`/`sum` ~2.2–2.7× faster; rolling `stdev` on 100k events ~3.3–7.3×
|
|
34
|
+
(a 1-event window 45.7 ms → 6.3 ms); partitioned rolling ~1.8×.
|
|
35
|
+
`baseline` / `outliers` (which delegate to `rolling`) inherit the speedup.
|
|
36
|
+
- **Behavior note — `array` columns:** an identity-comparing reducer (`keep`,
|
|
37
|
+
or a custom reducer using `===` on the cell) on an `array`-kind source
|
|
38
|
+
column now compares the value stored in the column, not the original object
|
|
39
|
+
reference passed at construction. Two rows given the *same* array object
|
|
40
|
+
therefore read as distinct. Scalar columns (number / string / boolean) are
|
|
41
|
+
unaffected. A non-finite or wrong-kind reducer result is still rejected with
|
|
42
|
+
a `ValidationError`, exactly as the constructor's intake did.
|
|
43
|
+
|
|
44
|
+
## [0.25.0] — 2026-06-15
|
|
45
|
+
|
|
46
|
+
### Changed
|
|
47
|
+
|
|
48
|
+
- **Reducers now treat non-finite numerics (`NaN` / `±Infinity`) as missing —
|
|
49
|
+
they are skipped — uniformly across every built-in reducer and all four
|
|
50
|
+
execution paths (`reduce`, the columnar fast path, `aggregate`/bucket, and
|
|
51
|
+
`rolling`/live).** Previously the paths disagreed on non-finite input: e.g.
|
|
52
|
+
`min`/`max` returned a position-dependent wrong extreme on the batch/columnar
|
|
53
|
+
paths but the true extreme on aggregate/rolling; `sum`/`avg` propagated
|
|
54
|
+
`NaN`. Non-finite can't enter via the row API (intake rejects it) — it only
|
|
55
|
+
arises inside computed columns (`cumulative` overflow, `diff`/`rate`
|
|
56
|
+
overflow, `collapse`, trusted construction) — so this only changes results
|
|
57
|
+
for those degenerate values, and makes every path agree. The three-layer
|
|
58
|
+
contract: **intake** stays strict (rejects non-finite), **computed writers**
|
|
59
|
+
stay permissive (pack honest non-finite), **reducers** are robust (skip it).
|
|
60
|
+
A standing parity-matrix test now pins all paths together. See
|
|
61
|
+
`docs/notes/reducer-nan-policy.md`. This also resolves the `aggregate('stdev')`
|
|
62
|
+
divergence class and the `min`/`max` NaN-laundering bug.
|
|
63
|
+
- Internal: `Float64Column` gained an `allFinite` fast-path flag (data-derived
|
|
64
|
+
at construction, conservative-by-default) so reducers skip the per-element
|
|
65
|
+
finite check on provably-finite columns — keeping the policy's cost off the
|
|
66
|
+
hot path (min/max/count stay at their pre-policy speed).
|
|
67
|
+
|
|
68
|
+
### Fixed
|
|
69
|
+
|
|
70
|
+
- **`rolling(window, { x: 'stdev' })` is now numerically stable.** It was the
|
|
71
|
+
last batch stdev path still on the one-pass `Σx²/n − mean²`, which cancels
|
|
72
|
+
catastrophically on near-equal large values (`[1e10, 1e10+1, …]` → `0`
|
|
73
|
+
instead of ≈1.118, or a negative variance → `NaN`) and drifts on trending
|
|
74
|
+
data (cumulative distance, elevation). It now uses Welford's online variance
|
|
75
|
+
with an order-independent **delete** — deviation-space, so no cancellation,
|
|
76
|
+
and removal **by value**, which keeps it correct under the live layer's
|
|
77
|
+
`reorder`-mode eviction (a positional/FIFO remove would have broken it; the
|
|
78
|
+
documented "stdev is reorder-safe" contract is preserved). Rolling-stdev
|
|
79
|
+
values shift in the last ULPs (now correct); the path stays O(1) and within
|
|
80
|
+
run-noise of the old one-pass, and a single-element window now reports exactly
|
|
81
|
+
`0` at any magnitude. Like any subtractive sliding variance, evicting an
|
|
82
|
+
outlier far outside the residual spread loses precision — negligible until the
|
|
83
|
+
evicted point is ~1e7–1e8× the residual stdev, far beyond realistic data.
|
|
84
|
+
- A standing differential-fuzz parity suite now pins every built-in reducer's
|
|
85
|
+
execution paths (columnar fast path vs `bucket` vs `rolling`, and the FIFO
|
|
86
|
+
sliding window vs a from-scratch recompute) against silent drift across
|
|
87
|
+
randomized magnitudes and window sizes — the class of bug behind the stdev
|
|
88
|
+
and `min`/`max` divergences.
|
|
89
|
+
|
|
21
90
|
## [0.24.0] — 2026-06-14
|
|
22
91
|
|
|
23
92
|
### Changed
|
|
@@ -123,26 +123,34 @@ export function tryAggregateColumnarTimeKeyed(begins, getColumn, buckets, column
|
|
|
123
123
|
reduced[p] = plan.reduce(plan.column.sliceByRange(start, scan));
|
|
124
124
|
}
|
|
125
125
|
else if (plan.which === 'first') {
|
|
126
|
-
// First defined cell in [start, scan); scans past missing cells
|
|
126
|
+
// First defined cell in [start, scan); scans past missing cells and
|
|
127
|
+
// past non-finite numeric cells (reducer non-finite policy,
|
|
128
|
+
// docs/notes/reducer-nan-policy.md — a NaN/±Inf numeric is "not a
|
|
129
|
+
// contributor", matching the row path's `defined` filter).
|
|
127
130
|
let value;
|
|
128
131
|
for (let i = start; i < scan; i += 1) {
|
|
129
132
|
const cell = plan.column.read(i);
|
|
130
|
-
if (cell
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
133
|
+
if (cell === undefined)
|
|
134
|
+
continue;
|
|
135
|
+
if (typeof cell === 'number' && !Number.isFinite(cell))
|
|
136
|
+
continue;
|
|
137
|
+
value = cell;
|
|
138
|
+
break;
|
|
134
139
|
}
|
|
135
140
|
reduced[p] = value;
|
|
136
141
|
}
|
|
137
142
|
else {
|
|
138
|
-
// Last defined cell in [start, scan); scans backward past missing
|
|
143
|
+
// Last defined cell in [start, scan); scans backward past missing and
|
|
144
|
+
// past non-finite numeric cells (see the 'first' branch above).
|
|
139
145
|
let value;
|
|
140
146
|
for (let i = scan - 1; i >= start; i -= 1) {
|
|
141
147
|
const cell = plan.column.read(i);
|
|
142
|
-
if (cell
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
148
|
+
if (cell === undefined)
|
|
149
|
+
continue;
|
|
150
|
+
if (typeof cell === 'number' && !Number.isFinite(cell))
|
|
151
|
+
continue;
|
|
152
|
+
value = cell;
|
|
153
|
+
break;
|
|
146
154
|
}
|
|
147
155
|
reduced[p] = value;
|
|
148
156
|
}
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
import { type Column } from '../../columnar/index.js';
|
|
2
|
+
/**
|
|
3
|
+
* Build a typed {@link Column} from a per-row value array, dispatching on the
|
|
4
|
+
* column's `kind`. An `undefined` cell becomes missing (its validity bit is
|
|
5
|
+
* left unset, so `read(i)` returns `undefined`). Numeric arrays reject
|
|
6
|
+
* non-finite values at construction (`float64ColumnFromArray`) — packed
|
|
7
|
+
* numeric columns stay NaN-free.
|
|
8
|
+
*
|
|
9
|
+
* This is the shared form of the kind→builder dispatch that `fill`
|
|
10
|
+
* (`buildFilledColumn`), `map` (its inline builder), and `collapse`
|
|
11
|
+
* (`buildScalarColumn`) each carry a local copy of — flagged there as a
|
|
12
|
+
* follow-up to converge. Columnar `rolling` is the fourth caller and uses
|
|
13
|
+
* this one directly; retrofitting the other three onto it is a separate
|
|
14
|
+
* (optional) cleanup.
|
|
15
|
+
*/
|
|
16
|
+
export declare function columnFromValuesByKind(kind: string, values: unknown[]): Column;
|
|
17
|
+
/**
|
|
18
|
+
* Throw if any *defined* value fails the column's `kind` contract, mirroring
|
|
19
|
+
* the constructor's strict intake (`validate.ts`): a `number` must be finite;
|
|
20
|
+
* `string` / `boolean` must match `typeof`; an `array` must be a real array
|
|
21
|
+
* whose elements are each a finite number, string, or boolean. `undefined`
|
|
22
|
+
* (missing) values are allowed.
|
|
23
|
+
*
|
|
24
|
+
* Computed-writer paths that assemble via trusted construction bypass intake,
|
|
25
|
+
* and {@link columnFromValuesByKind}'s `*FromArray` builders silently coerce a
|
|
26
|
+
* kind mismatch to a *missing* cell — so a caller that must preserve the intake
|
|
27
|
+
* rejection (e.g. columnar `rolling`, matching `mapColumns`) calls this first.
|
|
28
|
+
* `label` is prefixed to the thrown message.
|
|
29
|
+
*/
|
|
30
|
+
export declare function assertColumnValuesMatchKind(kind: string, values: ReadonlyArray<unknown>, label: string): void;
|
|
31
|
+
//# sourceMappingURL=column-builders.d.ts.map
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
import { arrayColumnFromArray, booleanColumnFromArray, float64ColumnFromArray, stringColumnFromArray, } from '../../columnar/index.js';
|
|
2
|
+
import { ValidationError } from '../../core/errors.js';
|
|
3
|
+
/**
|
|
4
|
+
* Build a typed {@link Column} from a per-row value array, dispatching on the
|
|
5
|
+
* column's `kind`. An `undefined` cell becomes missing (its validity bit is
|
|
6
|
+
* left unset, so `read(i)` returns `undefined`). Numeric arrays reject
|
|
7
|
+
* non-finite values at construction (`float64ColumnFromArray`) — packed
|
|
8
|
+
* numeric columns stay NaN-free.
|
|
9
|
+
*
|
|
10
|
+
* This is the shared form of the kind→builder dispatch that `fill`
|
|
11
|
+
* (`buildFilledColumn`), `map` (its inline builder), and `collapse`
|
|
12
|
+
* (`buildScalarColumn`) each carry a local copy of — flagged there as a
|
|
13
|
+
* follow-up to converge. Columnar `rolling` is the fourth caller and uses
|
|
14
|
+
* this one directly; retrofitting the other three onto it is a separate
|
|
15
|
+
* (optional) cleanup.
|
|
16
|
+
*/
|
|
17
|
+
export function columnFromValuesByKind(kind, values) {
|
|
18
|
+
switch (kind) {
|
|
19
|
+
case 'number':
|
|
20
|
+
return float64ColumnFromArray(values);
|
|
21
|
+
case 'string':
|
|
22
|
+
return stringColumnFromArray(values);
|
|
23
|
+
case 'boolean':
|
|
24
|
+
return booleanColumnFromArray(values);
|
|
25
|
+
case 'array':
|
|
26
|
+
// ArrayValue cells; the cast is the kind dispatch's trust point.
|
|
27
|
+
return arrayColumnFromArray(values);
|
|
28
|
+
default:
|
|
29
|
+
throw new TypeError(`columnFromValuesByKind: unsupported kind '${kind}'`);
|
|
30
|
+
}
|
|
31
|
+
}
|
|
32
|
+
/**
|
|
33
|
+
* Throw if any *defined* value fails the column's `kind` contract, mirroring
|
|
34
|
+
* the constructor's strict intake (`validate.ts`): a `number` must be finite;
|
|
35
|
+
* `string` / `boolean` must match `typeof`; an `array` must be a real array
|
|
36
|
+
* whose elements are each a finite number, string, or boolean. `undefined`
|
|
37
|
+
* (missing) values are allowed.
|
|
38
|
+
*
|
|
39
|
+
* Computed-writer paths that assemble via trusted construction bypass intake,
|
|
40
|
+
* and {@link columnFromValuesByKind}'s `*FromArray` builders silently coerce a
|
|
41
|
+
* kind mismatch to a *missing* cell — so a caller that must preserve the intake
|
|
42
|
+
* rejection (e.g. columnar `rolling`, matching `mapColumns`) calls this first.
|
|
43
|
+
* `label` is prefixed to the thrown message.
|
|
44
|
+
*/
|
|
45
|
+
export function assertColumnValuesMatchKind(kind, values, label) {
|
|
46
|
+
for (let i = 0; i < values.length; i += 1) {
|
|
47
|
+
const v = values[i];
|
|
48
|
+
if (v === undefined)
|
|
49
|
+
continue; // missing cell — allowed
|
|
50
|
+
let ok;
|
|
51
|
+
switch (kind) {
|
|
52
|
+
case 'number':
|
|
53
|
+
ok = typeof v === 'number' && Number.isFinite(v);
|
|
54
|
+
break;
|
|
55
|
+
case 'string':
|
|
56
|
+
ok = typeof v === 'string';
|
|
57
|
+
break;
|
|
58
|
+
case 'boolean':
|
|
59
|
+
ok = typeof v === 'boolean';
|
|
60
|
+
break;
|
|
61
|
+
case 'array': {
|
|
62
|
+
if (!Array.isArray(v)) {
|
|
63
|
+
ok = false;
|
|
64
|
+
break;
|
|
65
|
+
}
|
|
66
|
+
// Indexed loop, NOT `.every` (which skips holes): a sparse array's
|
|
67
|
+
// hole reads as `undefined`, an invalid element that intake's indexed
|
|
68
|
+
// scan rejects. Match it exactly so sparse arrays throw here rather
|
|
69
|
+
// than getting silently coerced to a missing cell by the builder.
|
|
70
|
+
ok = true;
|
|
71
|
+
for (let j = 0; j < v.length; j += 1) {
|
|
72
|
+
const el = v[j];
|
|
73
|
+
if (!((typeof el === 'number' && Number.isFinite(el)) ||
|
|
74
|
+
typeof el === 'string' ||
|
|
75
|
+
typeof el === 'boolean')) {
|
|
76
|
+
ok = false;
|
|
77
|
+
break;
|
|
78
|
+
}
|
|
79
|
+
}
|
|
80
|
+
break;
|
|
81
|
+
}
|
|
82
|
+
default:
|
|
83
|
+
ok = false;
|
|
84
|
+
}
|
|
85
|
+
if (!ok) {
|
|
86
|
+
// `ValidationError` (not `RangeError`) so the failure class matches the
|
|
87
|
+
// constructor's strict intake — a caller that catches `ValidationError`
|
|
88
|
+
// for bad user data sees columnar-rolling rejections identically. The
|
|
89
|
+
// value is stringified defensively: some invalid values (e.g. an array
|
|
90
|
+
// containing a `Symbol`) throw from `String(...)`, which would mask the
|
|
91
|
+
// `ValidationError` with a `TypeError`.
|
|
92
|
+
let shown;
|
|
93
|
+
try {
|
|
94
|
+
shown = String(v);
|
|
95
|
+
}
|
|
96
|
+
catch {
|
|
97
|
+
shown = `<unstringifiable ${typeof v}>`;
|
|
98
|
+
}
|
|
99
|
+
throw new ValidationError(`${label}: result ${shown} is not a valid '${kind}' value`);
|
|
100
|
+
}
|
|
101
|
+
}
|
|
102
|
+
}
|
|
103
|
+
//# sourceMappingURL=column-builders.js.map
|
|
@@ -900,6 +900,14 @@ export declare class TimeSeries<S extends SeriesSchema> {
|
|
|
900
900
|
* series carrying multiple entities (host, region, device id), use
|
|
901
901
|
* `series.partitionBy(col).rolling(...).collect()` to scope per
|
|
902
902
|
* entity. See {@link TimeSeries.partitionBy}.
|
|
903
|
+
*
|
|
904
|
+
* **`array`-column identity:** the window reads values from the
|
|
905
|
+
* columnar store, so on an `array`-kind source column an identity-
|
|
906
|
+
* comparing reducer (`keep`, or a custom reducer using `===` on the
|
|
907
|
+
* cell) compares the stored cell, not the original object reference
|
|
908
|
+
* from construction. Two rows that were given the *same* array object
|
|
909
|
+
* therefore read as distinct values here. Scalar columns
|
|
910
|
+
* (number / string / boolean) are unaffected (value semantics).
|
|
903
911
|
*/
|
|
904
912
|
rolling<const Mapping extends ValidatedAggregateMap<S, Mapping>>(window: DurationInput, mapping: Mapping, options?: {
|
|
905
913
|
alignment?: RollingAlignment;
|
|
@@ -5,6 +5,7 @@ import { fillOp } from './operators/fill.js';
|
|
|
5
5
|
import { mapOp } from './operators/map.js';
|
|
6
6
|
import { shiftOp } from './operators/shift.js';
|
|
7
7
|
import { collapseOp } from './operators/collapse.js';
|
|
8
|
+
import { assertColumnValuesMatchKind, columnFromValuesByKind, } from './operators/column-builders.js';
|
|
8
9
|
import { BoundedSequence } from '../sequence/bounded-sequence.js';
|
|
9
10
|
import { Interval } from '../core/interval.js';
|
|
10
11
|
import { Time } from '../core/time.js';
|
|
@@ -12,10 +13,10 @@ import { TimeRange } from '../core/time-range.js';
|
|
|
12
13
|
import { compareEventKeys } from '../core/temporal.js';
|
|
13
14
|
import { PartitionedTimeSeries } from './partitioned-time-series.js';
|
|
14
15
|
import { Sequence } from '../sequence/sequence.js';
|
|
15
|
-
import { IntervalKeyColumn, TimeKeyColumn, TimeRangeKeyColumn, float64ColumnFromArray, stringColumnFromArray, withColumnsRenamed, withColumnsSelected, withKeyColumn, withRowRange, withRowSelection, } from '../columnar/index.js';
|
|
16
|
+
import { ColumnarStore, IntervalKeyColumn, TimeKeyColumn, TimeRangeKeyColumn, float64ColumnFromArray, stringColumnFromArray, withColumnsRenamed, withColumnsSelected, withKeyColumn, withRowRange, withRowSelection, } from '../columnar/index.js';
|
|
16
17
|
import { SeriesStore } from '../live/series-store.js';
|
|
17
18
|
import { parseDuration } from '../core/duration.js';
|
|
18
|
-
import { resolveReducer, } from '../reducers/index.js';
|
|
19
|
+
import { resolveReducer, bucketStateFor, rollingStateFor, } from '../reducers/index.js';
|
|
19
20
|
// JSON ↔ typed-row primitives live in `./json.js`. Both `TimeSeries`
|
|
20
21
|
// and `LiveSeries` reach for them; extracted to break the import cycle
|
|
21
22
|
// that would otherwise form (Event needs them, TimeSeries imports
|
|
@@ -274,7 +275,11 @@ function bucketOverlapsHalfOpen(bucket, event) {
|
|
|
274
275
|
return event.begin() < bucket.end() && bucket.begin() < event.end();
|
|
275
276
|
}
|
|
276
277
|
function aggregateValues(operation, values) {
|
|
277
|
-
|
|
278
|
+
// Non-finite numerics (NaN / ±Inf) are treated as missing — excluded from
|
|
279
|
+
// both `defined` and `numeric` so every built-in reducer skips them exactly
|
|
280
|
+
// as it skips a missing cell. See docs/notes/reducer-nan-policy.md.
|
|
281
|
+
const defined = values.filter((value) => value !== undefined &&
|
|
282
|
+
(typeof value !== 'number' || Number.isFinite(value)));
|
|
278
283
|
const numeric = defined.filter((value) => typeof value === 'number');
|
|
279
284
|
return resolveReducer(operation).reduce(defined, numeric);
|
|
280
285
|
}
|
|
@@ -315,7 +320,9 @@ function applyAggregateReducer(reducer, values) {
|
|
|
315
320
|
// LivePartitionedSyncRolling) share the same normalisation. See the
|
|
316
321
|
// import below.
|
|
317
322
|
function createAggregateBucketState(operation) {
|
|
318
|
-
|
|
323
|
+
// Delegates to the shared factory so the non-finite skip policy
|
|
324
|
+
// (docs/notes/reducer-nan-policy.md) applies uniformly to batch + live.
|
|
325
|
+
return bucketStateFor(operation);
|
|
319
326
|
}
|
|
320
327
|
/**
|
|
321
328
|
* Resolve the output column kind for an `arrayAggregate` call. Numeric
|
|
@@ -336,7 +343,9 @@ function resolveArrayAggregateKind(reducer, explicitKind) {
|
|
|
336
343
|
return 'string';
|
|
337
344
|
}
|
|
338
345
|
function createRollingReducerState(operation) {
|
|
339
|
-
|
|
346
|
+
// Delegates to the shared factory so the non-finite skip policy
|
|
347
|
+
// (docs/notes/reducer-nan-policy.md) applies uniformly to batch + live.
|
|
348
|
+
return rollingStateFor(operation);
|
|
340
349
|
}
|
|
341
350
|
function duplicateValueColumnNames(schemas) {
|
|
342
351
|
const counts = new Map();
|
|
@@ -1971,33 +1980,34 @@ export class TimeSeries {
|
|
|
1971
1980
|
this.schema[0],
|
|
1972
1981
|
...resultColumnDefs,
|
|
1973
1982
|
]);
|
|
1983
|
+
// Columnar output path (3C): read the key + source columns straight off
|
|
1984
|
+
// the store and build the result columns directly — no `this.events`
|
|
1985
|
+
// materialization, no per-row `Event`, no row re-validation/re-pack.
|
|
1986
|
+
const store = this.#store.store;
|
|
1987
|
+
const rowCount = store.length;
|
|
1988
|
+
const sourceCols = columnSpecs.map((spec) => store.columns.get(spec.source));
|
|
1974
1989
|
const reducerStates = columnSpecs.map((spec) => isBuiltInAggregateReducer(spec.reducer)
|
|
1975
1990
|
? createRollingReducerState(spec.reducer)
|
|
1976
1991
|
: null);
|
|
1977
|
-
|
|
1978
|
-
const
|
|
1992
|
+
// Begin axis from the key buffer (replaces `this.events.map(e => e.begin())`).
|
|
1993
|
+
const beginTimes = store.keys.begin;
|
|
1994
|
+
// One value accumulator per output column; the result columns are built
|
|
1995
|
+
// from these and assembled via trusted construction below.
|
|
1996
|
+
const outValues = columnSpecs.map(() => new Array(rowCount));
|
|
1979
1997
|
let windowStart = 0;
|
|
1980
1998
|
let windowEnd = 0;
|
|
1981
1999
|
const addEvent = (index) => {
|
|
1982
|
-
const event = this.events[index];
|
|
1983
|
-
const data = event.data();
|
|
1984
2000
|
for (let i = 0; i < reducerStates.length; i++) {
|
|
1985
2001
|
const state = reducerStates[i];
|
|
1986
|
-
if (state)
|
|
1987
|
-
|
|
1988
|
-
state.add(index, data[spec.source]);
|
|
1989
|
-
}
|
|
2002
|
+
if (state)
|
|
2003
|
+
state.add(index, sourceCols[i].read(index));
|
|
1990
2004
|
}
|
|
1991
2005
|
};
|
|
1992
2006
|
const removeEvent = (index) => {
|
|
1993
|
-
const event = this.events[index];
|
|
1994
|
-
const data = event.data();
|
|
1995
2007
|
for (let i = 0; i < reducerStates.length; i++) {
|
|
1996
2008
|
const state = reducerStates[i];
|
|
1997
|
-
if (state)
|
|
1998
|
-
|
|
1999
|
-
state.remove(index, data[spec.source]);
|
|
2000
|
-
}
|
|
2009
|
+
if (state)
|
|
2010
|
+
state.remove(index, sourceCols[i].read(index));
|
|
2001
2011
|
}
|
|
2002
2012
|
};
|
|
2003
2013
|
const snapshotWindow = () => {
|
|
@@ -2007,25 +2017,33 @@ export class TimeSeries {
|
|
|
2007
2017
|
const state = reducerStates[i];
|
|
2008
2018
|
if (state)
|
|
2009
2019
|
return state.snapshot();
|
|
2010
|
-
|
|
2011
|
-
|
|
2012
|
-
|
|
2013
|
-
|
|
2014
|
-
|
|
2015
|
-
|
|
2020
|
+
// Custom reducer: read the window's source values directly off the
|
|
2021
|
+
// column in arrival order — identical value list to the old
|
|
2022
|
+
// `this.events.slice(windowStart, windowEnd)` path.
|
|
2023
|
+
const col = sourceCols[i];
|
|
2024
|
+
const values = [];
|
|
2025
|
+
for (let j = windowStart; j < windowEnd; j++)
|
|
2026
|
+
values.push(col.read(j));
|
|
2016
2027
|
return applyAggregateReducer(spec.reducer, values);
|
|
2017
2028
|
});
|
|
2018
2029
|
};
|
|
2030
|
+
// Scatter one window's aggregated values across every row of an equal-key
|
|
2031
|
+
// group (replaces the per-row frozen `[key, ...aggregated]` tuple write).
|
|
2032
|
+
const writeGroup = (groupStart, groupEnd, aggregated) => {
|
|
2033
|
+
for (let index = groupStart; index < groupEnd; index++) {
|
|
2034
|
+
for (let c = 0; c < outValues.length; c++) {
|
|
2035
|
+
outValues[c][index] = aggregated[c];
|
|
2036
|
+
}
|
|
2037
|
+
}
|
|
2038
|
+
};
|
|
2019
2039
|
if (alignment === 'trailing') {
|
|
2020
|
-
for (let groupStart = 0; groupStart <
|
|
2040
|
+
for (let groupStart = 0; groupStart < rowCount;) {
|
|
2021
2041
|
const anchor = beginTimes[groupStart];
|
|
2022
2042
|
let groupEnd = groupStart + 1;
|
|
2023
|
-
while (groupEnd <
|
|
2024
|
-
beginTimes[groupEnd] === anchor) {
|
|
2043
|
+
while (groupEnd < rowCount && beginTimes[groupEnd] === anchor) {
|
|
2025
2044
|
groupEnd += 1;
|
|
2026
2045
|
}
|
|
2027
|
-
while (windowEnd <
|
|
2028
|
-
beginTimes[windowEnd] <= anchor) {
|
|
2046
|
+
while (windowEnd < rowCount && beginTimes[windowEnd] <= anchor) {
|
|
2029
2047
|
addEvent(windowEnd);
|
|
2030
2048
|
windowEnd += 1;
|
|
2031
2049
|
}
|
|
@@ -2035,22 +2053,15 @@ export class TimeSeries {
|
|
|
2035
2053
|
removeEvent(windowStart);
|
|
2036
2054
|
windowStart += 1;
|
|
2037
2055
|
}
|
|
2038
|
-
|
|
2039
|
-
for (let index = groupStart; index < groupEnd; index++) {
|
|
2040
|
-
resultRows[index] = Object.freeze([
|
|
2041
|
-
this.events[index].key(),
|
|
2042
|
-
...aggregated,
|
|
2043
|
-
]);
|
|
2044
|
-
}
|
|
2056
|
+
writeGroup(groupStart, groupEnd, snapshotWindow());
|
|
2045
2057
|
groupStart = groupEnd;
|
|
2046
2058
|
}
|
|
2047
2059
|
}
|
|
2048
2060
|
else if (alignment === 'leading') {
|
|
2049
|
-
for (let groupStart = 0; groupStart <
|
|
2061
|
+
for (let groupStart = 0; groupStart < rowCount;) {
|
|
2050
2062
|
const anchor = beginTimes[groupStart];
|
|
2051
2063
|
let groupEnd = groupStart + 1;
|
|
2052
|
-
while (groupEnd <
|
|
2053
|
-
beginTimes[groupEnd] === anchor) {
|
|
2064
|
+
while (groupEnd < rowCount && beginTimes[groupEnd] === anchor) {
|
|
2054
2065
|
groupEnd += 1;
|
|
2055
2066
|
}
|
|
2056
2067
|
const lowerBound = anchor;
|
|
@@ -2060,28 +2071,20 @@ export class TimeSeries {
|
|
|
2060
2071
|
windowStart += 1;
|
|
2061
2072
|
}
|
|
2062
2073
|
const upperBound = anchor + windowMs;
|
|
2063
|
-
while (windowEnd <
|
|
2064
|
-
beginTimes[windowEnd] < upperBound) {
|
|
2074
|
+
while (windowEnd < rowCount && beginTimes[windowEnd] < upperBound) {
|
|
2065
2075
|
addEvent(windowEnd);
|
|
2066
2076
|
windowEnd += 1;
|
|
2067
2077
|
}
|
|
2068
|
-
|
|
2069
|
-
for (let index = groupStart; index < groupEnd; index++) {
|
|
2070
|
-
resultRows[index] = Object.freeze([
|
|
2071
|
-
this.events[index].key(),
|
|
2072
|
-
...aggregated,
|
|
2073
|
-
]);
|
|
2074
|
-
}
|
|
2078
|
+
writeGroup(groupStart, groupEnd, snapshotWindow());
|
|
2075
2079
|
groupStart = groupEnd;
|
|
2076
2080
|
}
|
|
2077
2081
|
}
|
|
2078
2082
|
else {
|
|
2079
2083
|
const halfWindow = windowMs / 2;
|
|
2080
|
-
for (let groupStart = 0; groupStart <
|
|
2084
|
+
for (let groupStart = 0; groupStart < rowCount;) {
|
|
2081
2085
|
const anchor = beginTimes[groupStart];
|
|
2082
2086
|
let groupEnd = groupStart + 1;
|
|
2083
|
-
while (groupEnd <
|
|
2084
|
-
beginTimes[groupEnd] === anchor) {
|
|
2087
|
+
while (groupEnd < rowCount && beginTimes[groupEnd] === anchor) {
|
|
2085
2088
|
groupEnd += 1;
|
|
2086
2089
|
}
|
|
2087
2090
|
const lowerBound = anchor - halfWindow;
|
|
@@ -2091,26 +2094,36 @@ export class TimeSeries {
|
|
|
2091
2094
|
windowStart += 1;
|
|
2092
2095
|
}
|
|
2093
2096
|
const upperBound = anchor + halfWindow;
|
|
2094
|
-
while (windowEnd <
|
|
2095
|
-
beginTimes[windowEnd] < upperBound) {
|
|
2097
|
+
while (windowEnd < rowCount && beginTimes[windowEnd] < upperBound) {
|
|
2096
2098
|
addEvent(windowEnd);
|
|
2097
2099
|
windowEnd += 1;
|
|
2098
2100
|
}
|
|
2099
|
-
|
|
2100
|
-
for (let index = groupStart; index < groupEnd; index++) {
|
|
2101
|
-
resultRows[index] = Object.freeze([
|
|
2102
|
-
this.events[index].key(),
|
|
2103
|
-
...aggregated,
|
|
2104
|
-
]);
|
|
2105
|
-
}
|
|
2101
|
+
writeGroup(groupStart, groupEnd, snapshotWindow());
|
|
2106
2102
|
groupStart = groupEnd;
|
|
2107
2103
|
}
|
|
2108
2104
|
}
|
|
2109
|
-
|
|
2110
|
-
|
|
2111
|
-
|
|
2112
|
-
|
|
2113
|
-
|
|
2105
|
+
// Assemble the result columns from the per-column accumulators and wrap via
|
|
2106
|
+
// trusted construction — the key column passes through zero-copy, and there
|
|
2107
|
+
// is no row re-validation / re-pack (the win this path exists for).
|
|
2108
|
+
const outColumns = new Map();
|
|
2109
|
+
for (let c = 0; c < columnSpecs.length; c++) {
|
|
2110
|
+
const kind = resultColumnDefs[c].kind;
|
|
2111
|
+
const values = outValues[c];
|
|
2112
|
+
// The old event-based path re-packed via the constructor's strict intake,
|
|
2113
|
+
// which rejected any defined result that didn't match the declared output
|
|
2114
|
+
// kind — a non-finite number (a `sum` overflow), or a wrong-typed value
|
|
2115
|
+
// (a custom reducer / `kind` override producing a string for a number
|
|
2116
|
+
// column, etc.). Trusted construction skips intake, and the `*FromArray`
|
|
2117
|
+
// builders silently coerce a kind mismatch to a *missing* cell — so
|
|
2118
|
+
// re-assert the same contract here (same value rules AND the same
|
|
2119
|
+
// `ValidationError` class as intake), keeping packed columns clean.
|
|
2120
|
+
// Missing cells (`undefined`, e.g. the minSamples warm-up) are
|
|
2121
|
+
// unaffected. (#225 Codex finding.)
|
|
2122
|
+
assertColumnValuesMatchKind(kind, values, `rolling column '${columnSpecs[c].output}'`);
|
|
2123
|
+
outColumns.set(columnSpecs[c].output, columnFromValuesByKind(kind, values));
|
|
2124
|
+
}
|
|
2125
|
+
const outStore = ColumnarStore.fromTrustedStore(resultSchema, store.keys, outColumns);
|
|
2126
|
+
return TimeSeries.#fromTrustedStore(this.name, resultSchema, outStore);
|
|
2114
2127
|
}
|
|
2115
2128
|
/**
|
|
2116
2129
|
* Example: `series.smooth("value", "ema", { alpha: 0.2 })`.
|
package/dist/batch/validate.js
CHANGED
|
@@ -416,7 +416,13 @@ export function validateAndNormalizeColumnar(input) {
|
|
|
416
416
|
let column;
|
|
417
417
|
switch (kind) {
|
|
418
418
|
case 'number':
|
|
419
|
-
|
|
419
|
+
// Strict intake: `assertCellKind` (kind 'number') already
|
|
420
|
+
// rejected every non-finite cell with a `ValidationError`
|
|
421
|
+
// before it reached `numberBufs`, so a surviving column is
|
|
422
|
+
// provably all-finite → `allFinite: true` (lets reducers skip
|
|
423
|
+
// the per-element finite guard). See the reducer non-finite
|
|
424
|
+
// policy + `Float64Column.allFinite`'s safety contract.
|
|
425
|
+
column = new Float64Column(numberBufs[c], length, validity, true);
|
|
420
426
|
break;
|
|
421
427
|
case 'boolean':
|
|
422
428
|
column = new BooleanColumn(booleanBufs[c], length, validity);
|