@soffinal/stream 0.2.1 → 0.2.3

package/package.json

@@ -1,8 +1,8 @@
 {
   "name": "@soffinal/stream",
   "module": "./dist/index.js",
-  "version": "0.2.1",
-  "description": "A reactive event streaming library for TypeScript/JavaScript",
+  "version": "0.2.3",
+  "description": "Multicast Event Pipelines with functional composition for TypeScript/JavaScript",
   "type": "module",
   "devDependencies": {
     "@types/bun": "latest"

package/src/transformers/filter.md

@@ -1,202 +1,326 @@
 # Filter Transformer
 
-## The Adaptive Gatekeeper
+The `filter` transformer selectively passes values through a stream based on predicate functions. It supports synchronous and asynchronous filtering, type guards, stateful operations, and multiple concurrency strategies.
 
-Traditional filtering is binary and stateless - a value either passes or doesn't. But real-world filtering often requires **memory, learning, and evolution**. The `filter` transformer embodies **Adaptive Reactive Programming** - where the gatekeeper remembers, and can even decide when to stop.
+## Quick Start
 
-## Design
+```typescript
+import { Stream, filter } from "@soffinal/stream";
 
-### Why State-First Architecture?
+const numbers = new Stream<number>();
 
-```typescript
-filter(initialState, (state, value) => [boolean, newState]);
+// Simple filtering
+const positives = numbers.pipe(filter((n) => n > 0));
+
+positives.listen(console.log);
+numbers.push(-1, 2, -3, 4); // Outputs: 2, 4
 ```
 
-**State comes first** because it's the foundation of adaptation. This isn't just filtering - it's **Adaptive gatekeeping** that evolves with each event.
+## Basic Usage
 
-### The Dual Return Pattern
+### Synchronous Filtering
 
 ```typescript
-return [shouldPass, newState]; // Continue with evolution
-return; // Terminate with wisdom
+// Basic predicate
+stream.pipe(filter((value) => value > 10));
+
+// Complex conditions
+stream.pipe(filter((user) => user.active && user.age >= 18));
+
+// Null/undefined filtering
+stream.pipe(filter((value) => value != null));
 ```
 
-**Two outcomes, infinite possibilities:**
+### Type Guards
+
+Filter supports TypeScript type guards for type narrowing:
+
+```typescript
+const mixed = new Stream<string | number>();
+
+// Type narrows to Stream<number>
+const numbers = mixed.pipe(filter((value): value is number => typeof value === "number"));
 
-- `[boolean, state]` - The filter learns and continues
-- `void` - The filter decides the stream has served its purpose
+// Type narrows to Stream<string>
+const strings = mixed.pipe(filter((value): value is string => typeof value === "string"));
+```
 
-This mirrors human decision-making: we either let something through (and remember why), or we decide we've seen enough.
+## Asynchronous Filtering
 
-### Argument Order
+### Sequential Processing (Default)
 
 ```typescript
-(state, value) => // State first, value second
+const stream = new Stream<string>();
+
+const validated = stream.pipe(
+  filter(async (email) => {
+    const isValid = await validateEmail(email);
+    return isValid;
+  })
+);
 ```
 
-**State precedes value** because context shapes perception. We don't judge events in isolation - we judge them based on what we've learned. The state is the accumulated transformations; the value is just the current moment.
+### Concurrent Strategies
+
+For expensive async predicates, choose a concurrency strategy:
 
-## The Adaptive Constraint System
+#### Concurrent Unordered
 
-### Level 1: Simple Gatekeeping
+Results emit as soon as they complete, potentially out of order:
 
 ```typescript
-// Traditional filtering - no memory, no learning
-stream.pipe(filter({}, (_, value) => [value > 0, {}]));
+const stream = new Stream<string>();
+
+const validated = stream.pipe(
+  filter(
+    async (url) => {
+      const isReachable = await checkURL(url);
+      return isReachable;
+    },
+    { strategy: "concurrent-unordered" }
+  )
+);
+
+// URLs may emit in different order based on response times
 ```
 
-Even "simple" filtering uses the adaptive architecture. The empty state `{}` represents a gatekeeper that doesn't need memory - but could develop it.
+#### Concurrent Ordered
 
-### Level 2: Memory-Based Filtering
+Parallel processing but maintains original order:
 
 ```typescript
-// The gatekeeper remembers and counts
-stream.pipe(
-  filter({ count: 0 }, (state, value) => {
-    const newCount = state.count + 1;
-    return [newCount % 3 === 0, { count: newCount }]; // Every 3rd passes
-  })
+const stream = new Stream<User>();
+
+const verified = stream.pipe(
+  filter(
+    async (user) => {
+      const isVerified = await verifyUser(user.id);
+      return isVerified;
+    },
+    { strategy: "concurrent-ordered" }
+  )
 );
+
+// Users always emit in original order despite varying verification times
 ```
 
-### Level 3: Termination
+## Stateful Filtering
+
+Maintain state across filter operations for complex logic:
+
+### Basic Stateful Filtering
 
 ```typescript
-// The gatekeeper knows when enough is enough
-stream.pipe(
-  filter({ seen: 0 }, (state, value) => {
-    if (state.seen >= 10) return; // Wisdom: we've seen enough
-    return [value > 0, { seen: state.seen + 1 }];
+const stream = new Stream<number>();
+
+// Take only first 5 items
+const limited = stream.pipe(
+  filter({ count: 0 }, (state, value) => {
+    if (state.count >= 5) return; // Terminate stream
+    return [true, { count: state.count + 1 }];
   })
 );
 ```
 
-**Stream termination** represents the ultimate adaptive behavior - knowing when to stop. This isn't just filtering; it's **stream lifecycle management**.
-
-### Level 4: Async
+### Advanced State Management
 
 ```typescript
-// The gatekeeper consults external validation
-stream.pipe(
-  filter({ cache: new Map() }, async (state, value) => {
-    if (state.cache.has(value)) {
-      return [state.cache.get(value), state]; // Remember previous decisions
+const stream = new Stream<string>();
+
+// Deduplicate values
+const unique = stream.pipe(
+  filter({ seen: new Set<string>() }, (state, value) => {
+    if (state.seen.has(value)) {
+      return [false, state]; // Skip duplicate
     }
 
-    const isValid = await validateAsync(value);
-    state.cache.set(value, isValid); // Learn for next time
-    return [isValid, state];
+    const newSeen = new Set(state.seen);
+    newSeen.add(value);
+    return [true, { seen: newSeen }];
   })
 );
 ```
 
-**Async filtering with memory** - the gatekeeper doesn't just validate, it **builds institutional knowledge**.
+### Complex Stateful Logic
+
+```typescript
+const events = new Stream<{ type: string; timestamp: number }>();
+
+// Rate limiting: max 5 events per second
+const rateLimited = events.pipe(
+  filter(
+    {
+      timestamps: [] as number[],
+      maxPerSecond: 5,
+    },
+    (state, event) => {
+      const now = event.timestamp;
+      const recent = state.timestamps.filter((t) => now - t < 1000);
+
+      if (recent.length >= state.maxPerSecond) {
+        return [false, { ...state, timestamps: recent }];
+      }
+
+      return [
+        true,
+        {
+          ...state,
+          timestamps: [...recent, now],
+        },
+      ];
+    }
+  )
+);
+```
 
-## Essential Copy-Paste Transformers
+## Stream Termination
 
-### simpleFilter - Gateway to Adaptation
+Filters can terminate streams by returning `undefined`:
 
 ```typescript
-// For users transitioning from traditional filtering
-const simpleFilter = <T>(predicate: (value: T) => boolean | Promise<boolean>) =>
-  filter<T, {}>({}, async (_, value) => {
-    const shouldPass = await predicate(value);
-    return [shouldPass, {}];
-  });
+const stream = new Stream<string>();
 
-// Usage: familiar syntax, adaptive foundation
-stream.pipe(simpleFilter((x) => x > 0));
-stream.pipe(simpleFilter(async (user) => await isValid(user)));
-```
+const untilStop = stream.pipe(
+  filter((value) => {
+    if (value === "STOP") return; // Terminates stream
+    return value.length > 0;
+  })
+);
 
-**Design choice**: `simpleFilter` is a **bridge**, not a replacement. It introduces users to the adaptive architecture while providing familiar syntax. The empty state `{}` is an invitation to evolution.
+stream.push("hello", "world", "STOP", "ignored");
+// Only "hello" and "world" are emitted
+```
 
-### take - The Counting Gatekeeper
+### Conditional Termination
 
 ```typescript
-const take = <T>(n: number) =>
-  filter<T, { count: number }>({ count: 0 }, (state, value) => {
-    if (state.count >= n) return; // Wisdom: we have enough
-    return [true, { count: state.count + 1 }];
-  });
+const numbers = new Stream<number>();
+
+const untilNegative = numbers.pipe(
+  filter({ sum: 0 }, (state, value) => {
+    const newSum = state.sum + value;
+    if (newSum < 0) return; // Terminate when sum goes negative
+
+    return [value > 0, { sum: newSum }];
+  })
+);
 ```
 
-### distinct - The Memory Gatekeeper
+## Performance Considerations
+
+### When to Use Concurrency
+
+- **Sequential**: Default choice, maintains order, lowest overhead
+- **Concurrent-unordered**: Use when order doesn't matter and predicates are expensive
+- **Concurrent-ordered**: Use when order matters but predicates are expensive
+
+### Memory Management
+
+Stateful filters maintain state objects. For large datasets:
 
 ```typescript
-const distinct = <T>() =>
-  filter<T, { seen: Set<T> }>({ seen: new Set() }, (state, value) => {
-    if (state.seen.has(value)) return [false, state];
-    state.seen.add(value);
-    return [true, state];
-  });
+// Good: Bounded state
+filter({ count: 0, limit: 1000 }, (state, value) => {
+  if (state.count >= state.limit) return;
+  return [predicate(value), { ...state, count: state.count + 1 }];
+});
+
+// Avoid: Unbounded state growth
+filter({ history: [] }, (state, value) => {
+  // This grows indefinitely!
+  return [true, { history: [...state.history, value] }];
+});
 ```
 
-### tap - The Observer Gatekeeper
+## Error Handling
 
-```typescript
-const tap = <T>(fn: (value: T) => void | Promise<void>) =>
-  filter<T, {}>({}, async (_, value) => {
-    await fn(value);
-    return [true, {}]; // Always pass through
-  });
+Errors in predicates will propagate and potentially terminate the stream:
 
-// Usage: Side effects without changing the stream
-stream.pipe(tap((value) => console.log("Saw:", value)));
-stream.pipe(tap(async (value) => await logToDatabase(value)));
+```typescript
+const stream = new Stream<number>();
+
+const safe = stream.pipe(
+  filter((value) => {
+    try {
+      return riskyPredicate(value);
+    } catch (error) {
+      console.error("Filter error:", error);
+      return false; // Skip problematic values
+    }
+  })
+);
 ```
 
-## The Termination
+## Common Patterns
 
-Stream termination isn't failure - it's **purposeful completion**. When a filter returns `void`, it's saying: "I have served my purpose, and this stream's journey ends here."
+### Throttling
 
 ```typescript
-// A filter that knows its mission
-const untilCondition = <T>(condition: (value: T) => boolean) =>
-  filter<T, {}>({}, (_, value) => {
-    if (condition(value)) return; // Mission complete
-    return [true, {}];
+const throttle = <T>(ms: number) =>
+  filter<T, { lastEmit: number }>({ lastEmit: 0 }, (state, value) => {
+    const now = Date.now();
+    if (now - state.lastEmit < ms) {
+      return [false, state];
+    }
+    return [true, { lastEmit: now }];
   });
+
+stream.pipe(throttle(1000)); // Max one value per second
 ```
 
-This represents a fundamental shift from infinite streams to **purpose-driven streams** that know when their work is done.
+### Sampling
 
-## Enhanced Pipe Integration
+```typescript
+const sample = <T>(n: number) =>
+  filter<T, { count: number }>({ count: 0 }, (state, value) => {
+    const shouldEmit = (state.count + 1) % n === 0;
+    return [shouldEmit, { count: state.count + 1 }];
+  });
+
+stream.pipe(sample(3)); // Every 3rd value
+```
 
-The new pipe architecture enables seamless integration:
+### Windowing
 
 ```typescript
-// Filter integrates with any transformer
-const result = stream
-  .pipe(filter({}, (_, v) => [v > 0, {}])) // Returns Stream<T>
-  .pipe(map({}, (_, v) => [v.toString(), {}])) // Returns Stream<string>
-  .pipe(toState("0")); // Returns State<string>
+const slidingWindow = <T>(size: number) =>
+  filter<T, { window: T[] }>({ window: [] }, (state, value) => {
+    const newWindow = [...state.window, value].slice(-size);
+    const shouldEmit = newWindow.length === size;
 
-// Complex filtering chains
-const processed = source
-  .pipe(
-    filter({ seen: 0 }, (state, v) => {
-      if (state.seen >= 100) return; // Terminate after 100
-      return [v > 0, { seen: state.seen + 1 }];
-    })
-  )
-  .pipe(tap((v) => console.log("Positive:", v)))
-  .pipe(
-    filter({ count: 0 }, (state, v) => {
-      return [state.count % 2 === 0, { count: state.count + 1 }]; // Every other
-    })
-  );
+    return [shouldEmit, { window: newWindow }];
+  });
+
+stream.pipe(slidingWindow(5)); // Emit when window is full
 ```
 
-**Note**: Filters compose naturally because they all speak the same language - **adaptive constraints** that can terminate, remember, and evolve.
+## Type Signatures
 
-**Design insight**: Filtering State creates **conditional reactivity** - the derived state only reacts to values that pass the adaptive constraints.
+```typescript
+// Simple predicate with optional concurrency
+filter<VALUE>(
+  predicate: (value: VALUE) => boolean | void | Promise<boolean | void>,
+  options?: { strategy: "sequential" | "concurrent-unordered" | "concurrent-ordered" }
+): (stream: Stream<VALUE>) => Stream<VALUE>
+
+// Type guard predicate (synchronous only)
+filter<VALUE, FILTERED extends VALUE>(
+  predicate: (value: VALUE) => value is FILTERED
+): (stream: Stream<VALUE>) => Stream<FILTERED>
+
+// Stateful predicate (always sequential)
+filter<VALUE, STATE>(
+  initialState: STATE,
+  predicate: (state: STATE, value: VALUE) => [boolean, STATE] | void
+): (stream: Stream<VALUE>) => Stream<VALUE>
+```
 
-## Conclusion
+## Best Practices
 
-The `filter` transformer isn't just about removing unwanted values - it's about **intelligent gatekeeping** that:
+1. **Choose the right strategy**: Use sequential for simple predicates, concurrent for expensive async operations
+2. **Manage state size**: Keep stateful filter state bounded to prevent memory leaks
+3. **Handle errors gracefully**: Wrap risky predicates in try-catch blocks
+4. **Use type guards**: Leverage TypeScript's type narrowing for better type safety
+5. **Consider termination**: Use `return undefined` to cleanly terminate streams when conditions are met
 
-- **Remembers** previous decisions (state)
-- **Learns** from patterns (adaptation)
-- **Evolves** behavior over time (constraints)
-- **Knows** when to stop (termination)
+The filter transformer is a powerful tool for stream processing that scales from simple synchronous predicates to complex stateful async operations with optimal performance characteristics.