@pond-ts/process 0.54.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Peter Murphy
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,345 @@
+# @pond-ts/process
+
+> **Experimental.** Pre-1.0, and the API is expected to move as friction
+> reports land — pin an exact version. The design iterates in the open
+> against
+> [RFC #543](https://github.com/pond-ts/pond/blob/main/docs/rfcs/process.md);
+> the roadmap and the measurements behind every design call are in
+> [PND_PROCESS_PLAN.md](https://github.com/pond-ts/pond/blob/main/docs/plans/PND_PROCESS_PLAN.md),
+> reproducible from
+> [`scripts/`](https://github.com/pond-ts/pond/tree/main/packages/process/scripts).
+
+**Computations as data over
+[pond-ts](https://www.npmjs.com/package/pond-ts).** A processing graph
+authored fluently in application code — or composed as JSON by a saved
+view or a tool-calling model — resolves against a declared op vocabulary
+and runs over a bound `TimeSeries`, with content-addressed caching,
+provenance, and per-node timings on every response. Underneath sits a
+small pull-based evaluation engine: nodes with typed ports, memoized
+results, and change propagation that stops as soon as a value stops
+changing.
+
+**Docs: [pond-ts.org/docs/process](https://pond-ts.org/docs/process/)** —
+tutorial, fluent authoring, the request/response contract, hosts and
+sources, caching and budgets.
+
+```sh
+npm install @pond-ts/process pond-ts
+```
+
+`pond-ts` is a peer dependency.
+
+## When _not_ to use this
+
+Chaining is pond's mental model and stays the right default:
+
+```ts
+const out = series
+  .rolling('5m', { cpu: 'avg' })
+  .aggregate(Sequence.every('1h'), {
+    cpu: 'max',
+  });
+```
+
+That is clearer than any graph, and pond's design notes deliberately
+[resist operator-graph vocabulary](https://github.com/pond-ts/pond/blob/main/docs/rfcs/streaming.md) for the
+core API — you do not submit a job graph to a runtime.
+
+This package exists for the case chaining genuinely cannot express: when
+**the pipeline itself is data**. Assembled at runtime from config,
+reshaped by a user in an editor, or fanning one expensive computation out
+to several consumers that each want a different slice. If your pipeline is
+known when you write the code, chain it and skip this package.
+
+## Fluent plans
+
+Bind authoring to a registry when application code constructs the graph.
+The registry's literal type supplies the operation methods, params, secondary
+input roles, and multi-output suffixes:
+
+```ts
+const graph = process(registry, 'ACME_5m').as('bands_and_stretch');
+const close = graph.column('close');
+
+const bands = close.bollinger({
+  as: 'bands',
+  period: 20,
+  stdDev: 2,
+});
+
+const width = bands.output('Upper').subtract({
+  as: 'width',
+  right: bands.output('Lower'),
+});
+
+const request = graph.outputs({
+  bands: bands.columns(),
+  upper: bands.output('Upper').columns(),
+  latestWidth: width.last(),
+});
+
+const result = host.run(request);
+```
+
+`period: '20'`, `bands.output('Banana')`, or an omitted `right` input are
+compile errors. The result is nevertheless plain slot-plan data: the fluent
+surface has no resolver or cache of its own, and a model-composed JSON request
+lands on the same nodes.
+
+Use `plan(from).add(...)` when the operation itself is dynamic and cannot be
+known to TypeScript.
+
+## Remote sources
+
+A graph may name an opaque asynchronous source instead of a preloaded dataset.
+The request carries its name and params; URLs, credentials, and loader code stay
+on the host:
+
+```ts
+const marketBars = defineSource({
+  name: 'market.bars',
+  async load(
+    {
+      symbol,
+      interval,
+    }: {
+      symbol: string;
+      interval: '1m' | '5m' | '1h';
+    },
+    { previous },
+  ) {
+    const response = await fetch(
+      `${MARKET_API}/bars?symbol=${symbol}&interval=${interval}`,
+      {
+        headers: {
+          Authorization: `Bearer ${MARKET_TOKEN}`,
+          ...(previous && { 'If-None-Match': previous.revision }),
+        },
+      },
+    );
+    if (response.status === 304) return previous!;
+    return {
+      value: TimeSeries.fromJSON(await response.json()),
+      revision: response.headers.get('etag')!,
+    };
+  },
+});
+
+const sources = createSourceRegistry().define(marketBars);
+const host = createHost({ registry, sources, units });
+
+const graph = process(
+  registry,
+  marketBars.ref({ symbol: 'ACME', interval: '5m' }),
+);
+const close = graph.column('close');
+const average = close.sma({ as: 'average', period: 20 });
+
+const result = await host.runAsync(
+  graph.outputs({ average: average.columns(), latest: average.last() }),
+);
+```
+
+The canonical source reference chooses a long-lived bound graph. The loader's
+`revision` decides freshness: an equal revision preserves every cached node; a
+new revision updates the source in place and lets ordinary graph invalidation
+run. Use synchronous `host.run()` for string-keyed datasets added with
+`host.add()`.
+
+## Worker pool (Node)
+
+`@pond-ts/process/pool` runs **whole requests** across worker threads, each
+holding a long-lived `Host`. It scales throughput under concurrent load; it
+does not make one request faster.
+
+```ts
+// setup.mjs — imported by BOTH isolates, because a registry is functions
+// and functions do not survive structured clone.
+export default function setup() {
+  return { registry, datasets: { px: series } };
+}
+```
+
+```ts
+import { HostPool } from '@pond-ts/process/pool';
+
+const pool = await HostPool.start({
+  setup: new URL('./setup.mjs', import.meta.url),
+  size: 4,
+});
+const result = await pool.run({ from: 'px', process: plan, select });
+await pool.close();
+```
+
+Requests run with `assemble: false` — the pool answers `columns` (which cross
+as transferable buffers) and the caller assembles a `TimeSeries` if it wants
+one. Pass an `affinity` key as the second argument to pin related requests to
+one worker, so its warm nodes get reused.
+
+**When it pays — it is about cache-hit rate, not request size.** Measured
+at 32 requests over 8 workers (`node packages/process/scripts/perf-pool.mjs`):
+**3.1–4.0× on distinct requests** at every size from 0.5 ms to 10 ms each, and
+**~0.01× on repeated ones**. In-process, a re-asked question is a memo hit that
+returns the same column for nothing; a pool copies and ships every answer
+however cheap it was, and each worker warms its own graph. Pooling and caching
+compete rather than compose.
+
+**Check your ops before you reach for the pool.** The same rolling mean writing
+a `Float64Array` instead of `new Array(n)` runs **482 ms single-threaded where
+the boxed version needs 632 ms across eight workers** — fixing the op beat
+adding eight cores. Boxing also parallelises worse, contending on memory
+bandwidth and per-isolate GC. A high pool speedup can be a symptom of a slow
+op.
+
+## Quick start
+
+```ts
+import { source, derive } from '@pond-ts/process';
+
+const raw = source<TimeSeries<Schema>>();
+
+const hourly = derive({ s: raw.out.value }, ({ s }) =>
+  s.aggregate(Sequence.every('1h'), { cpu: 'avg' }),
+);
+const peak = derive({ s: hourly.out.value }, ({ s }) => s.column('cpu').max());
+
+raw.set(series);
+peak.out.value.get(); // aggregates once, caches
+peak.out.value.get(); // cache hit — nothing recomputes
+```
+
+## How evaluation works
+
+Two mechanisms doing two different jobs:
+
+- **Dirty marking (push).** Setting a source marks everything downstream
+  as "revalidate before answering," cutting off at nodes already marked.
+  A change costs O(affected nodes) regardless of graph size.
+- **Version stamps (pull).** Each outlet's version increments only when a
+  recomputed value _actually differs_. A dirty node whose input versions
+  all match skips `compute` entirely.
+
+The second is the point. A source change that produces an identical
+downstream value stops the cascade there, so expensive transforms below it
+never run:
+
+```ts
+const level = source<number>();
+const bucket = derive({ x: level.out.value }, ({ x }) => Math.floor(x / 10), {
+  equals: (a, b) => a === b,
+});
+const expensive = derive({ b: bucket.out.value }, ({ b }) => heavyWork(b));
+
+level.set(11);
+expensive.out.value.get(); // computes
+level.set(13); // different input, same bucket
+expensive.out.value.get(); // cache hit — heavyWork never re-runs
+```
+
+Equality defaults to `Object.is`, which is right for immutable pond values
+— a transform that changed something returns a new instance. Supply
+`equals` where a node produces scalars or small records. Note that a
+`true` from `equals` also **keeps the old value** and discards the new
+one, so compare everything a consumer can observe, not just an id.
+
+## Types are enforced, not documented
+
+Ports are typed fields, so mismatches are compile errors:
+
+```ts
+text.out.value.connect(add.in.a); // ✗ Outlet<string> → Inlet<number>
+add.in.nope; // ✗ 'nope' is not a declared input
+```
+
+## Live sources
+
+`fromLive` binds a pond `LiveSeries` / `LiveView` into a graph. Events
+**invalidate**; they don't snapshot. A burst of 10,000 events costs one
+dirty mark each and exactly one `toTimeSeries()` at the next pull:
+
+```ts
+const feed = fromLive(liveSeries);
+const hourly = derive({ s: feed.out.value }, ({ s }) =>
+  s.aggregate(Sequence.every('1h'), { cpu: 'avg' }),
+);
+
+// ... 10k events arrive ...
+hourly.out.value.get(); // one snapshot, one aggregate
+feed.dispose(); // unsubscribe
+```
+
+That keeps the layer on the right side of pond's split: incremental
+per-event work stays in the live layer, and the graph composes whole-value
+batch transforms over snapshots.
+
+### There is no partial invalidation — bind the aggregation instead
+
+A dirty node recomputes from a _whole_ snapshot. The pipeline above
+therefore re-aggregates every retained event on every pull, even though
+only the tail moved. The graph does not track which rows changed, and
+deliberately doesn't try to: pond's live layer already does incremental
+per-event computation, and reimplementing it behind ports would be a
+second engine to keep correct.
+
+So push the windowed work down and bind _its_ output:
+
+```ts
+const feed = fromLive(live.aggregate(Sequence.every('1h'), { cpu: 'avg' }));
+const peak = derive({ s: feed.out.value }, ({ s }) => s.column('cpu').max());
+```
+
+`LiveAggregation` maintains its buckets per event, so a pull materializes
+bucket count instead of event count. At 200k events through a 50k-event
+buffer, pulling every 1k events: **9.05 ms/pull re-aggregating the buffer
+vs 0.04 ms/pull off the live aggregation — 235×**, and the gap widens with
+buffer size (O(retained events) vs O(buckets)).
+
+**Read the tradeoff before switching.** A live aggregation exposes _closed_
+buckets only. Data is the clock, so the newest bucket is invisible until an
+event crosses its end — two hours of minute data ending at 1h59m reads as
+one row this way and two by re-aggregating the buffer. If the currently
+filling bucket must be on screen, stay on the buffer and pay for it, or use
+a `Trigger` so buckets close on a schedule you control.
+
+## Multi-output nodes
+
+`derive` covers single-output nodes. `defineNode` declares a reusable node
+type, with as many outputs as you like — all computed in one pass:
+
+```ts
+const Extent = defineNode({
+  kind: 'extent',
+  inputs: { series: port<TimeSeries<Schema>>() },
+  outputs: { min: port<number>(), max: port<number>() },
+  compute: ({ series }) => ({
+    min: series.column('cpu').min(),
+    max: series.column('cpu').max(),
+  }),
+});
+
+const extent = Extent();
+hourly.out.value.connect(extent.in.series);
+extent.out.min.get(); // computes both outputs once
+```
+
+## Inspecting a graph
+
+`Graph` is a read-only view over already-wired nodes — evaluation never
+consults it.
+
+```ts
+const graph = Graph.from(peak); // discovers every reachable node
+graph.order(); // dependency order
+graph.toJSON(); // structure: nodes, ports, edges
+```
+
+`toJSON()` is a description, not a serialization — there is no `fromJSON`.
+Rebuilding a graph needs a `kind` → factory registry and per-node config in
+the dump; neither exists yet.
+
+## Errors
+
+A node caches the error its `compute` threw and rethrows it without
+re-running until an input changes, so a broken node stays cheap to poll.
+`node.error` exposes it. Cycles are rejected by `connect()`, so the graph
+is acyclic by construction and evaluation never guards against recursion.

package/dist/cjs-fallback.cjs

@@ -0,0 +1,15 @@
+'use strict';
+
+// @pond-ts/process ships as ES modules only. This stub is the `require` target in
+// the package's `exports` map so that CommonJS consumers get a clear,
+// actionable error instead of Node's cryptic `ERR_PACKAGE_PATH_NOT_EXPORTED`.
+//
+// It is copied verbatim into `dist/` during `prepack` (see package.json) so it
+// rides along in the published tarball; the source of truth lives at the
+// package root and is never touched by `tsc`.
+
+throw new Error(
+  '@pond-ts/process is an ES module package and cannot be loaded with require(). ' +
+    "Use `import { source } from '@pond-ts/process'` instead, or a dynamic " +
+    "`await import('@pond-ts/process')` from CommonJS. See https://nodejs.org/api/esm.html.",
+);

package/dist/column.d.ts

@@ -0,0 +1,197 @@
+/**
+ * Column-valued node support — [PND-PROCCOL].
+ *
+ * An op computes a study by calling into the corpus, which returns a
+ * `TimeSeries` whose new column is already **packed** (a `Float64Array`
+ * plus a validity bitmap). The obvious adapter then unpacks that into a
+ * boxed `Array<number | undefined>` to use as the node's value. Keeping
+ * the `Column` instead is worth doing for **memory and sizeability**: 20
+ * SMAs over 500k rows cost 271 MB of GC-managed heap boxed versus 42 MB
+ * as columns (rss 466 MB vs 353 MB — the bytes move to `arrayBuffers`
+ * rather than vanish).
+ *
+ * It is **not** worth doing for read throughput, which measurement
+ * contradicted: folding a max over 500k cells took 0.91 ms over the boxed
+ * array, 0.96 ms walking the buffer plus validity bits, and 4.27 ms via
+ * `Column.scan()` — whose per-cell callback costs more than either. A
+ * reduction on a hot path should walk `toFloat64Array()` and the bits
+ * directly rather than call `scan`.
+ *
+ * What that needs, and pond does not expose, is here:
+ *
+ * - {@link columnBytes} — a value's retained size, so a cache budget can
+ *   be expressed in bytes rather than entries ([PND-PROCCACHE]).
+ * - {@link packColumn} — build a packed column from loose values, for an
+ *   op whose kernel hands back an array.
+ * - {@link appendColumn} — put a column back onto a series for the
+ *   renderer path, avoiding the round trip where the column is gapless.
+ * - {@link columnBuffers} / {@link columnFromBuffers} — the buffer pair a
+ *   column *is*, for moving one across an isolate boundary
+ *   ([PND-PROCPAR]).
+ */
+import { Float64Column, TimeSeries } from 'pond-ts';
+import type { Column, SeriesSchema } from 'pond-ts';
+/**
+ * Retained size of a column value, in bytes.
+ *
+ * Approximate by construction — it counts the backing buffers a packed
+ * column owns, which is what a cache budget is actually trying to bound,
+ * and ignores per-object overhead. A chunked column reports the sum of
+ * its chunks. A column whose kind this cannot size returns `0` rather
+ * than guessing, so a budget treats it as free instead of evicting on a
+ * fabricated number.
+ *
+ * Bytes rather than entries is the point: a 500k-row result and a scalar
+ * both count as one entry, and only one of them matters.
+ */
+export declare function columnBytes(column: Column): number;
+/**
+ * Packs loose values into a `Float64Column`.
+ *
+ * For an op whose kernel returns an array — the financial kernels are
+ * array-out (`rollingValues`, `emaValues`) — this confines the boxing to
+ * one pass, instead of the boxed array being retained as the node value.
+ *
+ * `NaN` is treated as **missing**, matching what a study's warm-up
+ * means, and never packed as a defined cell: core's reducer non-finite
+ * policy requires that a column flagged `allFinite` contain no `NaN`,
+ * and a wrongly-set flag makes reducers take an unguarded path and
+ * silently return a wrong result. `allFinite` is therefore computed by
+ * inspecting every defined cell, not assumed.
+ */
+export declare function packColumn(values: ArrayLike<number | undefined | null>): Float64Column;
+/**
+ * Appends a column to a series under `name` — the renderer path, where a
+ * caller genuinely wants one `TimeSeries` carrying several studies.
+ *
+ * **A gapless column round-trips without boxing**, via the column's own
+ * `Float64Array`. A column *with* gaps cannot: core's `withColumn` takes
+ * values, not a column, and rejects a non-finite cell, so a warm-up has
+ * to be expressed as `undefined` in a boxed array. Core appends columns
+ * directly internally (`withColumnAppended`) but does not expose it;
+ * until it does, a gapped column pays one boxing pass here.
+ *
+ * Since most studies have a warm-up, that fallback is the common case —
+ * which is exactly why assembly should be requested rather than assumed
+ * ([PND-PROCTERM]). A facts-only request never calls this.
+ */
+export declare function appendColumn<S extends SeriesSchema>(series: TimeSeries<S>, name: string, column: Column): TimeSeries<SeriesSchema>;
+/**
+ * The buffer pair a packed numeric column **is** — for moving one across
+ * an isolate boundary ([PND-PROCPAR]).
+ *
+ * `RunResult.columns` is already described as the wire-shaped answer: a
+ * `Float64Array` plus a validity bitmap. This makes that literal, so a
+ * worker can hand a result back as two transferable buffers rather than
+ * 500k boxed values (48.6 ms per answer boxed vs 0.5 ms transferred).
+ *
+ * **The buffers are copies, deliberately.** A column's own buffers are
+ * shared with the node's memo, and transferring a buffer *detaches* it in
+ * the sending isolate — which would silently empty the cache the worker
+ * exists to keep warm. One `slice()` per surfaced column is the price of
+ * the cache staying valid, and it is still the cheap direction.
+ *
+ * Falls back to `undefined` for a column this cannot express (chunked
+ * storage, or a non-numeric kind), so a caller can box that one rather
+ * than the pool failing over a column it did not need packed.
+ */
+export interface ColumnBuffers {
+    readonly length: number;
+    readonly values: Float64Array;
+    /** Absent ⇒ every cell defined (the framework's no-bitmap convention). */
+    readonly bits?: Uint8Array;
+    readonly definedCount: number;
+    readonly allFinite: boolean;
+}
+export declare function columnBuffers(column: Column): ColumnBuffers | undefined;
+/** Rebuilds a column from {@link columnBuffers}, adopting both buffers. */
+export declare function columnFromBuffers(wire: ColumnBuffers): Float64Column;
+/**
+ * A **zero-copy** read view over a packed numeric column — [PND-PROCCOL].
+ *
+ * The counterpart to {@link columnBuffers}, which copies because its
+ * result crosses a thread. This one borrows: `values` and `bits` are
+ * `subarray`s of the column's own storage, valid only while the memo
+ * that produced the column holds it. Read, never retain.
+ *
+ * `undefined` for anything not packed numeric — a string column, or a
+ * value that arrived boxed — so a caller keeps its slow path.
+ *
+ * ## Why a view rather than `scan`
+ *
+ * Measured folding a max over 500k cells:
+ *
+ * | read path                     | ms   |
+ * | ----------------------------- | ---- |
+ * | boxed `(number\|undefined)[]` | 0.91 |
+ * | `Column.scan()`               | 4.27 |
+ * | buffer + bitmap               | 0.96 |
+ *
+ * The columnar form is **not faster to read** — a buffer walk only
+ * reaches parity with the boxed array, and `scan` is 4.7× slower than
+ * either because it takes a callback per cell. The case for it is that
+ * it does not *allocate*: no `Array` of 500k boxed slots per version,
+ * and nothing at all for a fold like `latest`, which reads one cell.
+ * Core's design principles recommend `scan` as the columnar read path,
+ * which is worth revisiting on this evidence.
+ */
+export interface ColumnView {
+    readonly length: number;
+    /** Column storage. A cell is meaningful only where {@link ColumnView.defined}. */
+    readonly values: Float64Array;
+    /** Absent ⇒ every cell is defined. */
+    readonly bits?: Uint8Array;
+    readonly definedCount: number;
+    defined(i: number): boolean;
+    at(i: number): number | undefined;
+}
+export declare function columnView(column: Column): ColumnView | undefined;
+/**
+ * A writable output buffer for a ranged recompute — [PND-PROCRANGE].
+ *
+ * The rows an op keeps unchanged are already here, values **and**
+ * validity, copied as blocks. The op fills `[from, to)` and nothing else.
+ *
+ * This exists because carrying a prefix forward correctly is harder than
+ * it looks and the obvious shortcut is silently wrong. Packed storage
+ * holds `0` at a missing cell, not `NaN`, so copying only `values` turns
+ * every warm-up gap in the prefix into a defined zero — measured at 1,875
+ * wrong cells on a 5-study pass, caught by a from-scratch comparison
+ * rather than by any type. Validity has to move with the values, and
+ * doing that per cell is the `O(n)` walk the whole ticket exists to
+ * remove.
+ */
+export interface RangeOutput {
+    /** Length `to`. `[0, from)` already carries the previous output. */
+    readonly values: Float64Array;
+    /**
+     * Validity bits, LSB-first, already carrying `[0, from)`.
+     *
+     * Exposed so an op can write a run of cells as bytes rather than
+     * through {@link RangeOutput.set} per cell; most ops want `set`.
+     */
+    readonly bits: Uint8Array;
+    /** Marks a cell present, with its value. */
+    set(i: number, v: number): void;
+    /**
+     * Marks a cell missing.
+     *
+     * **Only needed to undo a {@link RangeOutput.set} made in the same
+     * pass.** Cells in `[from, to)` start unset and an unwritten cell
+     * seals as missing, so clearing one the op never set does nothing —
+     * which is why an empty `clear` passed the whole suite until a
+     * set-then-clear case was written (Layer 2, PR #573).
+     */
+    clear(i: number): void;
+}
+/** Bytes needed for `length` validity bits. */
+export declare function validityByteCount(length: number): number;
+/**
+ * Prepares a {@link RangeOutput} of `length`, carrying `[0, keep)` from
+ * `prior` — a `Float64Array.set` for the values and a byte-wise copy for
+ * the bitmap, with the straddling byte masked.
+ */
+export declare function prepareRange(length: number, keep: number, prior: ColumnView | undefined): RangeOutput;
+/** Seals a {@link RangeOutput} into a column, counting validity once. */
+export declare function sealRange(out: RangeOutput, length: number): Float64Column;
+//# sourceMappingURL=column.d.ts.map