lulz 1.0.3 → 2.0.0

package/README.md

@@ -1,6 +1,28 @@
 # lulz
 
-A reactive dataflow system inspired by FFmpeg filtergraph notation and Node-RED.
+A reactive dataflow system that makes coders happy. 🎉
+
+Inspired by FFmpeg filtergraph notation, Node-RED, and RxJS.
+
+```javascript
+import { flow, inject, debug } from 'lulz';
+
+const app = flow([
+  [inject({ payload: 'Hello!' }), debug({ name: 'out' })],
+]);
+
+app.start();
+```
+
+## Features
+
+- **EventEmitter API** - Natural `emit`/`on` for packet injection and interception
+- **Series by Default** - `['in', a, b, c, 'out']` processes sequentially
+- **Explicit Parallel** - `['in', [a, b, c], 'out']` fans out to all
+- **Auto-compose** - Embed flows within flows seamlessly
+- **Worker Pool** - CPU-bound tasks with Worker Threads/Web Workers
+- **RxJS Operators** - `map`, `filter`, `scan`, `combineLatest`, and more
+- **Node-RED Style** - `inject`, `debug`, `func`, `switch`, `template`
 
 ## Installation
 
@@ -11,275 +33,355 @@ npm install lulz
 ## Quick Start
 
 ```javascript
-import { flow, Inject, Debug } from 'lulz';
+import { flow, inject, debug, func } from 'lulz';
 
+// Create a flow
 const app = flow([
-  [Inject({ payload: 'Hello World!' }), Debug({ name: 'output' })],
+  [inject({ payload: 'Hello World!' }), 'input'],
+  ['input', func({ func: msg => ({ ...msg, payload: msg.payload.toUpperCase() }) }), 'output'],
+  ['output', debug({ name: 'result' })],
 ]);
 
+// Start producers
 app.start();
-```
 
-## Core Concepts
+// Or inject manually via EventEmitter API
+app.emit('input', { payload: 'Manual injection!' });
 
-### Flow Syntax
+// Listen to any pipe
+app.on('output', (packet) => {
+  console.log('Received:', packet.payload);
+});
+```
+
+## Processing Modes
 
-Each line in a flow is an array: `[source, ...transforms, destination]`
+### Series (Default)
 
-- **Source**: A string (pipe name) or producer function
-- **Transforms**: Functions that process packets
-- **Destination**: A string (pipe name) or consumer function
+Functions process sequentially. Output of one becomes input of next.
 
 ```javascript
-const app = flow([
-  [producer, 'pipeName'],          // Producer → pipe
-  ['pipeName', transform, 'out'],  // Pipe → transform → pipe
-  ['out', consumer],               // Pipe → consumer
-]);
+['input', validate, transform, save, 'output']
+//          ↓          ↓         ↓
+//       packet → packet → packet → output
 ```
 
-### Function Types
+### Parallel (Explicit)
 
-**Outer functions** (factories) are regular functions that return inner functions:
+Use `[]` or `parallel()` to fan out. All receive the same packet.
 
 ```javascript
-function myTransform(options) {           // Outer - receives config
-  return (send, packet) => {              // Inner - processes packets
-    send({ ...packet, modified: true });
-  };
-}
+['input', [processA, processB, processC], 'output']
+//              ↓         ↓         ↓
+//           packet    packet    packet
+//              ↓         ↓         ↓
+//              └─────────┴─────────┘
+//                        ↓
+//                 output (3 packets)
 ```
 
-**Inner functions** are arrow functions that do the actual processing:
+### Helper Functions
 
 ```javascript
-const passthrough = (send, packet) => send(packet);
+import { series, parallel } from 'lulz';
+
+// Explicit series (same as default, but clearer)
+['input', series(a, b, c), 'output']
+
+// Explicit parallel
+['input', parallel(a, b, c), 'output']
 ```
 
-The system auto-detects which is which using `fn.hasOwnProperty('prototype')`:
-- Regular functions have `prototype` → outer
-- Arrow functions don't → inner
+## EventEmitter API
 
-### Pre-configured vs Auto-configured
+Every flow is an EventEmitter. Pipes are events.
 
 ```javascript
-['input', myTransform, 'output']              // Auto-config: called with {}
-['input', myTransform({ option: 1 }), 'output'] // Pre-configured
+const app = flow([
+  ['data', transform, 'result'],
+]);
+
+// Inject packets
+app.emit('data', { payload: 42 });
+
+// Listen to pipes
+app.on('result', (packet) => {
+  console.log(packet.payload);
+});
+
+// Also works
+app.inject('data', { payload: 42 });
 ```
 
-## Processing Modes
+## Function Types
 
-### Fan-out (Parallel)
+### Outer Functions (Factories)
 
-All transforms receive the same packet simultaneously:
+Regular functions that return inner functions. Called with options.
 
 ```javascript
-['input', transformA, transformB, transformC, 'output']
-//         ↓           ↓           ↓
-//      packet      packet      packet
-//         ↓           ↓           ↓
-//         └───────────┴───────────┘
-//                     ↓
-//                  output (receives 3 packets)
+function myTransform(options) {
+  const { multiplier = 1 } = options;
+  return (send, packet) => {
+    send({ ...packet, payload: packet.payload * multiplier });
+  };
+}
+
+// Usage
+['input', myTransform({ multiplier: 2 }), 'output']  // Pre-configured
+['input', myTransform, 'output']                      // Auto-configured with {}
 ```
 
-### Series (Sequential)
+### Inner Functions (Processors)
 
-Use `[a, b, c]` syntax for sequential processing:
+Arrow functions that process packets. Used directly.
 
 ```javascript
-['input', [transformA, transformB, transformC], 'output']
-//                ↓
-//            packet
-//                ↓
-//           transformA
-//                ↓
-//           transformB
-//                ↓
-//           transformC
-//                ↓
-//             output (receives 1 packet)
+const double = (send, packet) => {
+  send({ ...packet, payload: packet.payload * 2 });
+};
+
+['input', double, 'output']
 ```
 
-### Mixed
+## Subflows
 
-Combine both in a single line:
+Create reusable flow components.
 
 ```javascript
-['input', validate, [enrich, format], notify, 'output']
-//           ↓              ↓            ↓
-//        fan-out        series       fan-out
+import { subflow, compose } from 'lulz';
+
+// Create reusable component
+const sanitizer = subflow([
+  ['in', func({ func: msg => ({
+    ...msg,
+    payload: String(msg.payload).trim().toLowerCase()
+  })}), 'out'],
+]);
+
+// Use via compose
+const pipeline = compose(sanitizer, validator, enricher);
+pipeline.emit('in', { payload: '  HELLO  ' });
+
+// Or embed in flow (auto-compose)
+const app = flow([
+  ['input', sanitizer, 'output'],
+]);
 ```
 
-## Built-in Nodes
+## Node-RED Style Nodes
 
-### Inject
+### inject
 
-Produces packets on schedule or trigger:
+Produce packets on schedule.
 
 ```javascript
-Inject({
-  payload: 'hello',        // or () => Date.now() for dynamic
+inject({
+  payload: 'hello',           // or () => Date.now()
   topic: 'greeting',
-  once: true,              // Emit once on start
-  onceDelay: 100,          // Delay before first emit (ms)
-  interval: 1000,          // Repeat interval (ms)
+  once: true,                  // Emit once on start
+  onceDelay: 100,             // Delay before first
+  interval: 1000,             // Repeat interval (ms)
 })
 ```
 
-### Debug
+### debug
 
-Logs packets:
+Log packets.
 
 ```javascript
-Debug({
+debug({
   name: 'my-debug',
   active: true,
-  complete: false,         // true = show full msg, false = payload only
-  logger: console.log,     // Custom logger function
+  complete: false,            // true = full packet
+  logger: console.log,
 })
 ```
 
-### Function
+### func
 
-Execute custom JavaScript:
+Execute custom code.
 
 ```javascript
-Function({
+func({
   func: (msg, context) => {
     return { ...msg, payload: msg.payload.toUpperCase() };
   },
 })
 ```
 
-Return `null` to drop the message.
-
-### Change
+### change
 
-Modify message properties:
+Modify properties.
 
 ```javascript
-Change({
+change({
   rules: [
     { type: 'set', prop: 'payload', to: 'new value' },
-    { type: 'set', prop: 'topic', to: (msg) => msg.payload.type },
     { type: 'change', prop: 'payload', from: /old/, to: 'new' },
-    { type: 'delete', prop: 'unwanted' },
-    { type: 'move', prop: 'payload', to: 'data' },
+    { type: 'delete', prop: 'temp' },
+    { type: 'move', prop: 'data', to: 'payload' },
   ]
 })
 ```
 
-### Switch
+### switchNode
 
-Route messages based on conditions:
+Route by conditions.
 
 ```javascript
-Switch({
+switchNode({
   property: 'payload',
   rules: [
-    { type: 'eq', value: 'hello' },
     { type: 'gt', value: 100 },
     { type: 'regex', value: /^test/ },
     { type: 'else' },
   ],
-  checkall: false,  // Stop at first match
 })
 ```
 
-Rule types: `eq`, `neq`, `lt`, `gt`, `lte`, `gte`, `regex`, `true`, `false`, `null`, `nnull`, `else`
+### template
 
-### Template
-
-Render mustache templates:
+Render templates.
 
 ```javascript
-Template({
-  template: 'Hello {{name}}, you have {{count}} messages!',
-  field: 'payload',  // Output field
+template({
+  template: 'Hello {{name}}!',
+  field: 'payload',
 })
 ```
 
-### Delay
+## RxJS-Style Operators
 
-Delay or rate-limit messages:
+### Transformation
 
 ```javascript
-Delay({
-  delay: 1000,    // Delay in ms
-  rate: 10,       // Or rate limit (msgs/sec)
-  drop: false,    // Drop vs queue when rate limited
-})
-```
+import { map, scan, pluck, pairwise, buffer } from 'lulz';
 
-### Split
+map({ fn: x => x * 2 })
+scan({ reducer: (acc, x) => acc + x, initial: 0 })
+pluck({ path: 'data.value' })
+pairwise()
+buffer({ count: 5 })
+```
 
-Split arrays/strings into sequences:
+### Filtering
 
 ```javascript
-Split({
-  property: 'payload',
-  delimiter: ',',  // For strings, or 'array' for arrays
-})
+import { filter, take, skip, distinct, distinctUntilChanged } from 'lulz';
+
+filter({ predicate: x => x > 0 })
+take({ count: 5 })
+skip({ count: 2 })
+distinct()
+distinctUntilChanged()
 ```
 
-### Join
+### Timing
 
-Join sequences back together:
+```javascript
+import { debounce, throttle, delay, timeout } from 'lulz';
+
+debounce({ time: 300 })
+throttle({ time: 1000 })
+delay({ time: 500 })
+timeout({ time: 5000 })
+```
+
+### Combination
 
 ```javascript
-Join({
-  count: 5,           // Emit after N messages
-  property: 'payload',
-  mode: 'manual',     // 'auto', 'manual', 'reduce'
-})
+import { combineLatest, merge, zip, withLatestFrom } from 'lulz';
+
+combineLatest({ pipes: ['temp', 'humidity'] })
+merge()
+zip({ sources: 2 })
 ```
 
-## Subflows
+## Worker Pool
 
-Create reusable flow components:
+Process CPU-bound tasks in parallel using Worker Threads.
 
 ```javascript
-const sanitizer = subflow([
-  ['in', Function({ func: (msg) => ({
-    ...msg,
-    payload: String(msg.payload).trim().toLowerCase()
-  })}), 'out'],
+import { taskQueue, worker, parallelMap } from 'lulz';
+
+// Standalone task queue
+const queue = taskQueue({
+  workers: 4,
+  handler: (data) => heavyComputation(data),
+});
+
+queue.on('result', ({ id, result }) => console.log(result));
+queue.submit({ data: 42 });
+await queue.drain();
+
+// In a flow
+const app = flow([
+  ['input', worker({
+    workers: 4,
+    handler: (data) => data * data,
+  }), 'output'],
 ]);
+```
 
-// Wire into main flow
-mainFlow.pipes['input'].connect(sanitizer._input);
-sanitizer._output.connect(mainFlow.pipes['process']);
+## API Reference
+
+### flow(graph, context?)
+
+Create a new flow.
+
+```javascript
+const app = flow([...], { username: 'alice' });
+
+app.start();     // Start producers
+app.stop();      // Stop producers
+app.emit(pipe, packet);   // Inject packet
+app.on(pipe, handler);    // Listen to pipe
+app.pipe(name);           // Get pipe node
 ```
 
-## API
+### subflow(graph, context?)
 
-### `flow(graph, context)`
+Create a reusable flow with `in`/`out` pipes.
 
-Create a new flow from a graph definition.
+### compose(...flows)
 
-Returns:
-- `start()` - Start all producers
-- `stop()` - Stop all producers
-- `inject(pipeName, packet)` - Inject a packet into a pipe
-- `getPipe(name)` - Get a pipe by name
-- `pipes` - Object of all pipes
+Connect flows in sequence.
 
-### `subflow(graph, context)`
+### series(...fns) / parallel(...fns)
 
-Create a subflow with `in` and `out` pipes.
+Explicit processing mode markers.
 
-### `compose(...flows)`
+## Project Structure
 
-Connect multiple flows in sequence.
+```
+lulz/
+├── index.js           # Main exports
+├── src/
+│   ├── flow.js        # Core engine
+│   ├── red-lib.js     # Node-RED style nodes
+│   ├── rx-lib.js      # RxJS operators
+│   ├── workers.js     # Worker pool
+│   └── utils.js       # Utilities
+├── test.js
+├── examples.js
+├── TODO.md            # Future operators
+└── README.md
+```
 
 ## Running
 
 ```bash
-npm test      # Run tests
-npm run examples  # Run examples
+npm test        # Run tests
+npm run examples    # Run examples
 ```
 
 ## License
 
 MIT
+
+## Links
+
+- [GitHub](https://github.com/catpea/lulz)
+- [Node-RED](https://nodered.org/)
+- [RxJS](https://rxjs.dev/)

package/TODO.md

@@ -0,0 +1,132 @@
+# lulz - TODO
+
+## RxJS Operators to Add Later
+
+These operators are less commonly used but might be useful in specific scenarios.
+
+### Combination Operators
+
+- [ ] `forkJoin` - Wait for all observables to complete, emit last values
+- [ ] `race` - First to emit wins
+- [ ] `combineAll` - Combine all inner observables
+- [ ] `concatAll` - Concat all inner observables
+- [ ] `mergeAll` - Merge all inner observables
+- [ ] `switchAll` - Switch to latest inner observable
+- [ ] `startWith` - Prepend values
+- [ ] `endWith` - Append values
+
+### Transformation Operators
+
+- [ ] `bufferCount` - Buffer by count
+- [ ] `bufferTime` - Buffer by time
+- [ ] `bufferToggle` - Buffer between open/close signals
+- [ ] `bufferWhen` - Buffer until closing selector
+- [ ] `concatMap` - Map and concat
+- [ ] `exhaustMap` - Ignore new while processing
+- [ ] `expand` - Recursively project
+- [ ] `groupBy` - Group by key
+- [ ] `mapTo` - Map to constant
+- [ ] `mergeMap` / `flatMap` - Map and merge
+- [ ] `mergeScan` - Scan with merge
+- [ ] `partition` - Split into two streams
+- [ ] `switchMap` - Switch to new on each emission
+- [ ] `windowCount` - Window by count
+- [ ] `windowTime` - Window by time
+- [ ] `windowToggle` - Window between signals
+- [ ] `windowWhen` - Window by closing selector
+
+### Filtering Operators
+
+- [ ] `audit` - Ignore during duration selector
+- [ ] `auditTime` - Ignore during time window
+- [ ] `debounceTime` - (alias for debounce with time)
+- [ ] `elementAt` - Emit only Nth element
+- [ ] `first` - Emit first (or first matching)
+- [ ] `ignoreElements` - Ignore all, only complete/error
+- [ ] `last` - Emit last (or last matching)
+- [ ] `sample` - Sample on signal
+- [ ] `sampleTime` - Sample at interval
+- [ ] `single` - Emit only if single match
+- [ ] `skipLast` - Skip last N
+- [ ] `skipUntil` - Skip until signal
+- [ ] `takeLast` - Take last N
+- [ ] `takeUntil` - Take until signal
+- [ ] `throttleTime` - (alias for throttle)
+
+### Utility Operators
+
+- [ ] `dematerialize` - Convert notification objects
+- [ ] `materialize` - Wrap in notification objects
+- [ ] `observeOn` - Schedule emissions
+- [ ] `subscribeOn` - Schedule subscription
+- [ ] `timeInterval` - Emit time between values
+- [ ] `finalize` - Execute on complete/error
+- [ ] `repeat` - Repeat N times
+- [ ] `repeatWhen` - Repeat based on notifier
+
+### Error Handling
+
+- [ ] `retryWhen` - Retry based on notifier
+- [ ] `onErrorResumeNext` - Continue with next on error
+
+### Multicasting
+
+- [ ] `multicast` - Share with subject
+- [ ] `publish` - Share via publish subject
+- [ ] `publishBehavior` - Share via behavior subject
+- [ ] `publishLast` - Share via async subject
+- [ ] `publishReplay` - Share via replay subject
+- [ ] `refCount` - Auto connect/disconnect
+- [ ] `shareReplay` - Share and replay
+
+## Node-RED Nodes to Add
+
+### Network
+
+- [ ] `http in` - HTTP endpoint
+- [ ] `http request` - HTTP client
+- [ ] `http response` - HTTP response
+- [ ] `websocket in` - WebSocket listener
+- [ ] `websocket out` - WebSocket sender
+- [ ] `tcp in` / `tcp out` / `tcp request` - TCP nodes
+- [ ] `udp in` / `udp out` - UDP nodes
+- [ ] `mqtt in` / `mqtt out` - MQTT nodes
+
+### Storage
+
+- [ ] `file in` - Read file
+- [ ] `file out` - Write file
+- [ ] `watch` - Watch file/directory
+
+### Parsers
+
+- [ ] `csv` - Parse/generate CSV
+- [ ] `html` - Parse HTML
+- [ ] `json` - Parse/stringify JSON
+- [ ] `xml` - Parse/generate XML
+- [ ] `yaml` - Parse/generate YAML
+
+### Sequence
+
+- [ ] `batch` - Batch messages
+- [ ] `sort` - Sort messages
+
+## Worker Enhancements
+
+- [ ] Browser Web Worker support
+- [ ] SharedArrayBuffer for zero-copy data transfer
+- [ ] Worker pool auto-scaling
+- [ ] Priority queue
+- [ ] Task cancellation
+- [ ] Progress reporting
+
+## Core Improvements
+
+- [ ] Better TypeScript definitions
+- [ ] Flow visualization / debugging
+- [ ] Hot reloading of flows
+- [ ] Persistence / checkpointing
+- [ ] Metrics / monitoring
+- [ ] Backpressure handling
+- [ ] Dead letter queue for failed messages
+- [ ] Circuit breaker pattern