@zakkster/lite-stream 1.0.0

package/ROADMAP.md

@@ -0,0 +1,191 @@
+# @zakkster/lite-stream Roadmap
+
+Features deliberately deferred from 1.0 with the rationale. Each is gated
+on a specific triggering signal from a real consumer rather than
+speculative inclusion.
+
+## Triggering signals
+
+This roadmap exists because of the same discipline that drove
+`lite-signal`'s and `lite-statechart`'s development: ship the minimum that
+solves the actual problem, defer everything else until a concrete consumer
+demonstrates the need with a real code site. Nothing in this file is
+"planned for 1.x" -- everything is "considered, deferred, waiting for a
+triggering signal."
+
+---
+
+## 1.1.0 candidates
+
+### Concurrent mapping (`mapLimit`)
+
+```js
+const live = fromAsyncIterable(urls, {
+    transform: async (url, ctx) => {
+        return await fetch(url, { signal: ctx.signal }).then((r) => r.json());
+    },
+    concurrency: 4                      // up to 4 in-flight transforms
+});
+```
+
+A consumer-side pattern where each yielded value needs an async
+transformation, and the user wants bounded concurrency. Currently doable
+by composing async generators externally, but if multiple consumers (helix
+batch enrichment, EBS request fan-out) end up writing the same pattern,
+add it.
+
+**Triggering signal:** Two or more consumers in the ecosystem writing
+identical "transform-with-concurrency-limit" wrappers around
+`fromAsyncIterable`.
+
+---
+
+### Reverse-direction buffering modes
+
+`toAsyncIterable` currently only does drop-oldest on overflow. Some
+consumers might want:
+
+- `overflow: "drop-newest"` -- reject incoming when full
+- `overflow: "throw"` -- iterator throws on overflow
+- `overflow: "block"` -- producer-side signal back-channel (requires
+  consumer cooperation; significant API change)
+
+**Why deferred from 1.0:** Drop-oldest is the only mode any real consumer
+in the roadmap actually wants. Adding alternatives speculatively grows the
+API surface for no measurable benefit.
+
+**Triggering signal:** A consumer demonstrating that drop-oldest causes
+observable correctness issues for their use case.
+
+---
+
+### `mergeIterables(iterA, iterB, ...)`
+
+Merge multiple async iterators into one signal, interleaving values as
+they arrive. Useful for "watch both helix and pubsub for follower
+changes" patterns.
+
+**Why deferred from 1.0:** Doable today by composing iterators externally
+(an async generator that races multiple sources). Built-in support cleans
+up the cancellation story (one AbortSignal cancels all sources) but adds
+complexity.
+
+**Triggering signal:** First Twitch consumer that needs to merge >= 2
+async sources into a single reactive surface with shared cancellation.
+
+---
+
+### `splitSignal(sig, predicate) -> [matchingSig, restSig]`
+
+The dual of `mergeIterables`: split a signal's value stream into two based
+on a predicate, exposing both halves as new signals.
+
+**Why deferred from 1.0:** No concrete consumer in the roadmap needs this.
+Speculative.
+
+**Triggering signal:** A consumer demonstrating a real need to fan-out a
+single async source into two reactive surfaces.
+
+---
+
+## 1.2.0+ candidates (smaller, lower priority)
+
+- **`fromEventTarget(target, eventName)`** -- a wrapper that turns DOM
+  EventTarget events into an async iterable feeding a signal. Doable
+  today via async generators, but ergonomically common. Add if the
+  pattern recurs across browser-facing consumers (e.g. window resize
+  observers, pointer event streams).
+
+- **`backpressure: "block"` for pipeToSignal** -- await the target
+  signal's effects before pulling the next value. Requires effect-
+  completion semantics from lite-signal, which don't exist today.
+
+- **`opts.equals` for diff-mode signals** -- by default the underlying
+  signal uses lite-signal's default `Object.is`. Allow opt-in to a custom
+  equality predicate for buffer-mode snapshot arrays (where Object.is
+  always reports them as distinct).
+
+- **Diagnostic surface** -- `live.stats() -> { yields, drops, errors,
+  startedAt, settledAt }` for telemetry. Useful but no consumer asking
+  for it yet.
+
+---
+
+## Deferred indefinitely
+
+### Full reactive stream combinators
+
+`map`, `filter`, `take`, `skip`, `debounce`, `throttle`, etc. -- the
+ReactiveX / RxJS surface area.
+
+**Why:** Once you have async iterators, all of these are five-line async
+generators. Adding them to lite-stream would create two ways to do the
+same thing, where the second way (combinator API) hides the source code
+and makes the cancellation model harder to reason about. The async
+generator form is more inspectable and composes naturally with all of
+JS's existing async tooling.
+
+If a consumer wants `debounce`, they write:
+
+```js
+async function* debounce(source, ms, signal) {
+    let pending;
+    for await (const v of source) {
+        if (pending) clearTimeout(pending);
+        pending = await new Promise((r) => {
+            setTimeout(() => r(v), ms);
+            signal?.addEventListener("abort", () => r(undefined));
+        });
+        if (pending !== undefined) yield pending;
+    }
+}
+```
+
+Not as pretty as `source.pipe(debounce(500))`, but inspectable, debuggable,
+and composes with everything else in the JS async ecosystem.
+
+---
+
+### `propagate: "always"` mode
+
+A mode where every yielded value triggers an effect run even if
+`Object.is`-equal to the previous. Currently the underlying signal's
+default equality (`Object.is`) is sufficient for all known consumers --
+our wrapper state object is fresh per yield, so dedup never kicks in.
+
+**Why deferred:** No consumer wants force-propagate semantics in
+practice; the wrapper state shape already guarantees per-yield
+propagation.
+
+---
+
+## Non-goals
+
+These will never ship in `lite-stream`:
+
+- **A standalone scheduler.** Microtask / setImmediate decisions stay
+  inside the pump. The library does not expose a scheduling API.
+- **A retry/backoff policy.** Consumers compose with `lite-await`'s
+  `withTimeout`, `withAbort`, and a plain `for` loop. Every Twitch SDK
+  endpoint has different retry semantics; a single policy would be
+  opinionated wrong for half of them.
+- **WebSocket / EventSource adapters.** Those belong in the SDK that
+  consumes them (lite-twitch-ebs, lite-twitch-pubsub), not in lite-stream.
+- **Concurrent multi-iterator orchestration beyond `mergeIterables`** --
+  use `lite-await`'s `allOf` / `anyOf` / `raceOf` for that.
+
+---
+
+## Versioning policy
+
+`@zakkster/lite-stream` follows strict semver. 1.x is a stable API; only
+additive changes (new exports, new opts fields, new state-shape fields)
+ship in minor releases. Breaking changes wait for 2.0. Patch releases are
+bug fixes only.
+
+Each addition will land with:
+
+1. A concrete triggering consumer in the `@zakkster/lite-*` ecosystem
+2. Tests covering the full lifecycle (happy + all three cleanup paths)
+3. A bench measuring throughput + retained heap
+4. A line item in CHANGELOG.md with a hot-path-cost note

package/Stream.d.ts

@@ -0,0 +1,137 @@
+/**
+ * @zakkster/lite-stream -- zero-GC bridge between async iterators and
+ * `@zakkster/lite-signal`.
+ *
+ * Public type surface for the JavaScript implementation in `Stream.js`.
+ */
+
+import { Signal, Computed } from "@zakkster/lite-signal";
+
+// ---------------------------------------------------------------------------
+// State shapes (discriminated by mode)
+// ---------------------------------------------------------------------------
+
+/**
+ * Per-yield state for `fromAsyncIterable` in "latest" mode. The signal value
+ * is the most recent yielded item; on completion or error, `done: true` is
+ * set and `value` is preserved.
+ */
+export interface LatestState<T> {
+    readonly value:  T | undefined;
+    readonly count:  number;
+    readonly done:   boolean;
+    readonly error:  unknown;
+}
+
+/**
+ * Per-yield state for `fromAsyncIterable` in "buffer" mode. The signal value
+ * is a bounded array of recent yields (newest last). On overflow, the OLDEST
+ * value is dropped and `droppedCount` is incremented. Snapshot is a fresh
+ * array per update; the user-facing array is never mutated in place.
+ */
+export interface BufferState<T> {
+    readonly values:        ReadonlyArray<T>;
+    readonly count:         number;
+    readonly droppedCount:  number;
+    readonly done:          boolean;
+    readonly error:         unknown;
+}
+
+// ---------------------------------------------------------------------------
+// Options
+// ---------------------------------------------------------------------------
+
+export interface FromAsyncIterableLatestOptions<T> {
+    mode?: "latest";
+    initial?: T;
+    signal?: AbortSignal;
+    onError?: (err: unknown) => void;
+    onDone?: () => void;
+}
+
+export interface FromAsyncIterableBufferOptions<T> {
+    mode: "buffer";
+    /**
+     * Required when mode === "buffer". Positive integer. Unbounded buffering
+     * is explicitly rejected as a memory bug pretending to be a feature; pick
+     * a deliberate ceiling and observe `droppedCount` to know when you're
+     * losing data.
+     */
+    maxBuffer: number;
+    initial?: T;
+    signal?: AbortSignal;
+    onError?: (err: unknown) => void;
+    onDone?: () => void;
+}
+
+export interface PipeToSignalOptions<T> {
+    signal?: AbortSignal;
+    onError?: (err: unknown) => void;
+    onDone?: () => void;
+    transform?: (value: T) => T;
+}
+
+export interface ToAsyncIterableOptions {
+    signal?: AbortSignal;
+    /** Default `true`: yield the current signal value first. */
+    emitInitial?: boolean;
+    /** Default `1024`. Positive integer. */
+    maxBuffer?: number;
+}
+
+// ---------------------------------------------------------------------------
+// Public functions
+// ---------------------------------------------------------------------------
+
+/**
+ * Drive a signal from an async iterator. Returns a Signal whose value is a
+ * tagged state object reflecting the iterator's lifecycle.
+ *
+ * **Termination** (cleanup is structural on all three paths):
+ * 1. Iterator natural completion (`{ done: true }`) -> signal state goes done.
+ * 2. Iterator throws -> signal state goes error.
+ * 3. `opts.signal` aborts -> the iterator's `return()` is called best-effort.
+ *
+ * **Modes:**
+ * - `"latest"` (default): signal value is `{ value, count, done, error }`.
+ *   Cheapest path; allocates one wrapper object per yield.
+ * - `"buffer"`: signal value is `{ values, count, droppedCount, done, error }`.
+ *   `maxBuffer` REQUIRED. Overflow drops oldest and increments `droppedCount`.
+ */
+export function fromAsyncIterable<T>(
+    source: AsyncIterable<T> | AsyncIterator<T> | Iterable<T>,
+    opts?: FromAsyncIterableLatestOptions<T>
+): Signal<LatestState<T>>;
+export function fromAsyncIterable<T>(
+    source: AsyncIterable<T> | AsyncIterator<T> | Iterable<T>,
+    opts: FromAsyncIterableBufferOptions<T>
+): Signal<BufferState<T>>;
+
+/**
+ * Lower-level companion to `fromAsyncIterable`: pump an existing writable
+ * signal from an async iterator. The signal's value is replaced directly
+ * with each yielded value (no wrapper state). Returns a stop function that
+ * ends the pump and calls `iterator.return()`.
+ *
+ * Does NOT dispose the target signal; the caller owns its lifetime.
+ */
+export function pipeToSignal<T>(
+    source: AsyncIterable<T> | AsyncIterator<T>,
+    target: Signal<T>,
+    opts?: PipeToSignalOptions<T>
+): () => void;
+
+/**
+ * Yield signal changes as an async iterable. Each change resolves a pending
+ * `next()`. If the consumer is slower than the producer, values queue in a
+ * bounded ring (default size 1024); on overflow, OLDEST is dropped and
+ * `droppedCount` increments. Inspect via the iterable's `.droppedCount`
+ * property at any time during or after consumption.
+ *
+ * The iterator naturally completes when `opts.signal` aborts; consumer-side
+ * `break` triggers `iterator.return()` and cleans up the subscription.
+ */
+export function toAsyncIterable<T>(
+    sig: Signal<T> | Computed<T>,
+    opts?: ToAsyncIterableOptions
+): AsyncIterable<T> & { readonly droppedCount: number };