pond-ts 0.53.1 → 0.55.0

package/README.md

@@ -152,27 +152,31 @@ const points = series.sample({ reservoir: { size: 500 } }).toRows();
 
 ## Performance
 
-pond-ts is faster on **every** comparable operation, with no regressions —
-a **~17x** geometric-mean speedup across the measurable ops, plus a handful
-of transforms (`select` / `rename`) that are **effectively instant** (O(1)
-column rebinds, below the timer's resolution). The advantage grows with data
-size.
-
-| Category          | Speedup (N=16k)                                      | Notes                                         |
-| ----------------- | ---------------------------------------------------- | --------------------------------------------- |
-| **Rate**          | ~120x                                                | Single columnar walk vs Pipeline              |
-| **Fill**          | 77–87x                                               | Single columnar pass vs Pipeline per strategy |
-| **Aggregation**   | 57–82x                                               | O(N+B) bucketing vs O(N×B) Pipeline           |
-| **Statistics**    | 18–80x                                               | Typed-array reduce vs ImmutableJS iteration   |
-| **Alignment**     | 42x                                                  | Forward cursor vs repeated binary search      |
-| **Construction**  | 13x                                                  | Columnar intake vs ImmutableJS wrapping       |
-| **Chained**       | 8x                                                   | Derived constructors vs per-step Pipeline     |
-| **Transforms**    | `select`/`rename` instant; `collapse` 30x; `map` ~4x | Column reshapes vs Pipeline                   |
-| **Event access**  | 6x                                                   | Array indexing vs ImmutableJS `get()`         |
-| **Serialization** | 4x                                                   | Lightweight columnar representation           |
-
-See the [full benchmark results](website/docs/reference/benchmarks.mdx)
-for detailed numbers. Run locally:
+Measured against three reference points (snapshot 2026-07-30; full tables,
+methodology, and every losing number in the
+[benchmark reference](website/docs/reference/benchmarks.mdx)):
+
+- **vs pondjs** (the predecessor): faster on all 54 measurable shared
+  operations — geometric mean **20.7×**, `aggregate` up to 453×,
+  `median` 157×, and `select` / `rename` effectively instant (O(1)
+  column rebinds).
+- **vs pandas** (500k-bar workload): roughly **even** — ahead on `ema`,
+  `mean`, `median` / `percentile`; behind ~1.1–1.9× on the rolling
+  studies. A five-study strategy pass runs 1.28× slower than pandas'
+  Cython kernels.
+- **vs polars, single-threaded**: **ahead on composite studies**
+  (`bollinger` 0.53×, strategy stack 0.85×, `ema` 0.32× — lower is
+  pond-ts faster), behind 4–9× on whole-column reductions (the SIMD gap;
+  half-closed already by blocked summation) and on raw ingest, where
+  pond-ts front-loads validation the dataframe engines defer.
+- **vs polars on all 10 cores**: behind ~3.5× on the strategy stack —
+  pond-ts has no parallelism today; a measured 2.42× worker-thread path
+  is on the roadmap (`[PND-PROCPAR]`).
+
+The honest one-line version: the rewrite beat its predecessor by an order
+of magnitude, holds its own per-core against the native engines on the
+composite queries that dominate real workloads, and knows exactly where
+it is behind. Run locally:
 
 ```sh
 npm run build && node packages/core/bench/vs-pondjs.cjs

package/dist/batch/aggregate-columns.d.ts

@@ -1,5 +1,6 @@
-import type { Column } from '../columnar/index.js';
-import { Float64Column } from '../columnar/index.js';
+import type { Column, ColumnSchema } from '../columnar/index.js';
+import { ColumnarStore, Float64Column } from '../columnar/index.js';
+import type { Interval } from '../core/interval.js';
 import type { AggregateMap, AggregateOutputMap, AggregateReducer, RollingAlignment, ScalarKind, SeriesSchema } from '../schema/index.js';
 /**
  * Normalised column spec used by both batch and live aggregation paths.
@@ -35,10 +36,48 @@ export type AggregateColumnSpec = {
  * same `mapping` shape produces the same schema.
  */
 export declare function normalizeAggregateColumns<S extends SeriesSchema>(schema: S, mapping: AggregateMap<S> | AggregateOutputMap<S>): AggregateColumnSpec[];
-export declare function tryAggregateColumnarTimeKeyed<B extends {
-    begin(): number;
-    end(): number;
-}>(begins: Float64Array, getColumn: (name: string) => Column | undefined, buckets: ReadonlyArray<B>, columns: ReadonlyArray<AggregateColumnSpec>): Array<ReadonlyArray<unknown>> | null;
+/**
+ * **Columnar output for time-keyed `aggregate()` — [PND-IVLCOL].**
+ *
+ * Same gate and same bucket walk as the reduction fast path above, but
+ * the result is assembled as a `ColumnarStore` instead of a row array.
+ *
+ * **Why this exists.** `aggregate`'s output is interval-keyed, and the
+ * only construction door for an interval-keyed series used to be row
+ * intake — so the columnar fast path computed each bucket's answer in
+ * typed arrays, boxed it into a frozen `[Interval, …]` row, and then
+ * `new TimeSeries({ rows })` walked those rows straight back into
+ * columns. Measured on a 1M-row series at 16,667 buckets × 4 columns,
+ * that round trip cost **2.52 ms of a 5.01 ms call** — more than the
+ * reduction it was packaging. Writing the columns directly costs
+ * **0.09 ms** (spike: `spikes/columnar-wasm/bench/interval-columnar.mjs`).
+ *
+ * **Behaviour is identical, deliberately.** Trusted construction skips
+ * row intake, so every check row intake performed is performed here:
+ *
+ * - Cell kinds go through the *same* `assertCellKind` (shared, not
+ *   copied), so a reducer that overflows to `Infinity` still raises the
+ *   same `ValidationError` with the same message rather than silently
+ *   landing a non-finite cell in a column flagged `allFinite`.
+ * - Numeric columns are stamped `allFinite: true` on the same
+ *   justification the row path uses — every surviving cell was
+ *   finite-checked above. Dropping that flag would be safe but would
+ *   quietly deoptimise every downstream reduction of an aggregate
+ *   result.
+ * - Missing cells (an empty bucket, or an all-missing one) set no
+ *   validity bit, so they read back as `undefined` — **not** `NaN`. That
+ *   is the sentinel decision this change forces, resolved in favour of
+ *   preserving current behaviour exactly.
+ *
+ * The one row-intake check deliberately skipped is the non-decreasing
+ * key scan: `BoundedSequence` already validates its intervals as sorted,
+ * non-overlapping, and positive-duration at construction, so re-deriving
+ * it per bucket would be checking the same fact twice.
+ *
+ * Returns `null` for the same disqualifying mappings as before; the
+ * caller falls back to the unchanged row path.
+ */
+export declare function tryAggregateColumnarStore(begins: Float64Array, getColumn: (name: string) => Column | undefined, buckets: ReadonlyArray<Interval>, columns: ReadonlyArray<AggregateColumnSpec>, resultSchema: ColumnSchema): ColumnarStore | null;
 /**
  * **Columnar fast path for count-window `rolling()`** — the N-bar window
  * financial studies compose on (SMA / Bollinger / rolling stats). When every
@@ -57,7 +96,7 @@ export declare function tryAggregateColumnarTimeKeyed<B extends {
  * qualify: a custom-function reducer, a non-`'number'` output kind
  * (`unique` / `samples` / `keep` over a non-number source / an explicit
  * `kind` override), or a non-numeric / chunked / missing source column.
- * All-or-nothing per call, matching {@link tryAggregateColumnarTimeKeyed}.
+ * All-or-nothing per call, matching {@link tryAggregateColumnarStore}.
  *
  * Window shape replicates the generic count sweep exactly: rows are the unit
  * (`count` is a bar count — no equal-key grouping), `lo` / `hi` are monotonic