lulz 2.0.2 → 2.0.3

package/README.md

@@ -352,6 +352,355 @@ Connect flows in sequence.
 
 Explicit processing mode markers.
 
+## Workers
+
+lulz includes a worker pool system for CPU-intensive tasks. It uses **Worker Threads** in Node.js (and Web Workers in browsers) so heavy computation doesn't block your main thread.
+
+### Why Workers?
+
+JavaScript is single-threaded. If you compute Fibonacci(45), your entire app freezes:
+
+```javascript
+// ❌ Bad: Blocks everything
+['input', (send, packet) => {
+  const result = fibonacci(packet.payload);  // 🧊 Frozen for 5 seconds
+  send({ ...packet, payload: result });
+}, 'output']
+```
+
+Workers run in separate threads:
+
+```javascript
+// ✅ Good: Non-blocking
+['input', worker({ handler: fibonacci }), 'output']
+// Main thread stays responsive while workers compute
+```
+
+---
+
+### taskQueue — Standalone Task Queue
+
+The foundation. An EventEmitter that manages a pool of workers.
+
+```javascript
+import { taskQueue } from 'lulz';
+
+// Create a queue with 4 workers
+const queue = taskQueue({
+  workers: 4,                      // Number of worker threads (default: CPU cores)
+  handler: (data) => data * data   // Function that runs in worker
+});
+
+// Listen for completed tasks
+queue.on('result', ({ id, result }) => {
+  console.log(`Task ${id} finished:`, result);
+});
+
+// Listen for errors (handler threw an exception)
+queue.on('error', ({ id, error }) => {
+  console.error(`Task ${id} failed:`, error);
+});
+
+// Listen for all tasks complete
+queue.on('drain', () => {
+  console.log('All tasks done!');
+});
+
+// Submit a single task
+queue.submit({ id: 'task-1', data: 42 });
+// → Task task-1 finished: 1764
+
+// Submit multiple tasks
+queue.submitAll([
+  { id: 'a', data: 10 },
+  { id: 'b', data: 20 },
+  { id: 'c', data: 30 },
+]);
+```
+
+#### Async Handlers
+
+Handlers can be async:
+
+```javascript
+const queue = taskQueue({
+  workers: 2,
+  handler: async (url) => {
+    const response = await fetch(url);
+    return response.json();
+  }
+});
+
+queue.submit({ data: 'https://api.example.com/data' });
+```
+
+#### Queue Control
+
+```javascript
+// Check queue status
+console.log(queue.stats());
+// → { pending: 5, running: 4, available: 0, totalSubmitted: 9, totalCompleted: 4 }
+
+// Wait for all tasks to complete
+await queue.drain();
+
+// Shut down all workers
+await queue.terminate();
+```
+
+---
+
+### worker — Flow Integration
+
+Use workers directly in your flows. Packets go in, get processed in worker threads, come out.
+
+```javascript
+import { flow, worker } from 'lulz';
+
+const app = flow([
+  ['numbers',
+    // This runs in a worker thread, not the main thread
+    worker({
+      workers: 4,
+      handler: (n) => {
+        // Heavy computation here
+        let sum = 0;
+        for (let i = 0; i < n * 1000000; i++) {
+          sum += Math.sqrt(i);
+        }
+        return sum;
+      }
+    }),
+    'results'
+  ],
+
+  ['results', debug({ name: 'computed' })],
+]);
+
+// Send numbers to process
+app.emit('numbers', { payload: 100 });
+app.emit('numbers', { payload: 200 });
+app.emit('numbers', { payload: 300 });
+// Results arrive as workers complete (may be out of order)
+```
+
+#### Preserves Packet Metadata
+
+The worker node keeps your packet's other properties intact:
+
+```javascript
+app.emit('numbers', {
+  payload: 100,
+  userId: 'alice',      // ← preserved
+  requestId: 'req-123'  // ← preserved
+});
+
+// Output packet:
+// { payload: 12345.67, userId: 'alice', requestId: 'req-123' }
+```
+
+---
+
+### parallelMap — Process Arrays
+
+When you have an array and want each item processed in parallel:
+
+```javascript
+import { flow, parallelMap } from 'lulz';
+
+const app = flow([
+  ['images',
+    parallelMap({
+      workers: 4,
+      fn: (image) => {
+        // Each image processed in its own worker
+        return {
+          ...image,
+          thumbnail: generateThumbnail(image),
+          compressed: compress(image)
+        };
+      }
+    }),
+    'processed'
+  ],
+]);
+
+// Send an array
+app.emit('images', {
+  payload: [image1, image2, image3, image4, image5]
+});
+
+// Receive complete array (order preserved!)
+// { payload: [processed1, processed2, processed3, processed4, processed5] }
+```
+
+Key difference from `worker`:
+- `worker`: Each packet = one task, results stream out
+- `parallelMap`: One packet with array = many tasks, waits for all, emits single array
+
+---
+
+### cpuTask — Quick Wrapper
+
+Shorthand when you just want to run a function in a worker:
+
+```javascript
+import { flow, cpuTask } from 'lulz';
+
+// Instead of this:
+worker({ handler: (n) => fibonacci(n) })
+
+// Write this:
+cpuTask((n) => fibonacci(n))
+```
+
+Example:
+
+```javascript
+const app = flow([
+  ['input', cpuTask(expensiveCalculation), 'output'],
+]);
+```
+
+It's just sugar for `worker({ handler: fn })` with default worker count.
+
+---
+
+### Patterns
+
+#### Pattern 1: Fan-Out Computation
+
+Process the same data multiple ways in parallel:
+
+```javascript
+const app = flow([
+  ['data', [
+    worker({ handler: analyzeWithMethodA }),
+    worker({ handler: analyzeWithMethodB }),
+    worker({ handler: analyzeWithMethodC }),
+  ], 'analyzed'],
+]);
+// All three analyses run simultaneously in different workers
+```
+
+#### Pattern 2: Pipeline with Mixed Threading
+
+Some steps in main thread, heavy steps in workers:
+
+```javascript
+const app = flow([
+  ['request',
+    validate,           // Fast: main thread
+    parseInput,         // Fast: main thread
+    worker({ handler: heavyTransform }),  // Slow: worker
+    formatOutput,       // Fast: main thread
+    'response'
+  ],
+]);
+```
+
+#### Pattern 3: Batch Processing
+
+Split → process in workers → join:
+
+```javascript
+const app = flow([
+  // Split array into individual items
+  ['batch', split(), 'item'],
+
+  // Process each in workers
+  ['item', worker({ handler: processOne }), 'processed'],
+
+  // Collect results (need custom collector)
+  ['processed', join({ count: expectedCount }), 'complete'],
+]);
+```
+
+Or just use `parallelMap` which does this for you:
+
+```javascript
+const app = flow([
+  ['batch', parallelMap({ fn: processOne }), 'complete'],
+]);
+```
+
+---
+
+### Error Handling
+
+Worker errors don't crash your app. They emit on the `'error'` event:
+
+```javascript
+const queue = taskQueue({
+  handler: (data) => {
+    if (data < 0) throw new Error('Negative not allowed');
+    return Math.sqrt(data);
+  }
+});
+
+queue.on('result', ({ id, result }) => {
+  console.log(`${id} = ${result}`);
+});
+
+queue.on('error', ({ id, error }) => {
+  console.log(`${id} failed: ${error}`);
+});
+
+queue.submit({ id: 'good', data: 16 });   // → good = 4
+queue.submit({ id: 'bad', data: -1 });    // → bad failed: Negative not allowed
+```
+
+In flows, errors become packet properties:
+
+```javascript
+['input', worker({ handler: riskyFunction }), 'output']
+
+// If handler throws, packet becomes:
+// { payload: ..., error: 'Error message' }
+```
+
+---
+
+### Configuration
+
+```javascript
+import { cpus } from 'os';
+
+taskQueue({
+  workers: cpus().length,  // Default: number of CPU cores
+  handler: fn,             // Required: function to run in worker
+})
+
+worker({
+  workers: 4,              // Default: number of CPU cores
+  handler: fn,             // Required: function to run in worker
+})
+
+parallelMap({
+  workers: 4,              // Default: number of CPU cores
+  fn: fn,                  // Required: function to run in worker
+})
+```
+
+---
+
+### When to Use Workers
+
+✅ **Use workers for:**
+- Mathematical computations (crypto, statistics, ML inference)
+- Image/video processing
+- Data parsing (large JSON, CSV)
+- Compression/decompression
+- Any task taking >50ms
+
+❌ **Don't use workers for:**
+- Simple transformations (`x * 2`)
+- I/O-bound tasks (use async/await instead)
+- Tasks needing DOM access (workers can't touch DOM)
+- Very small tasks (worker overhead > computation)
+
+The overhead of sending data to a worker and back is ~1-5ms. If your task takes less than that, just run it in the main thread.
+
 ## Project Structure
 
 ```

package/examples-queue.js

@@ -0,0 +1,419 @@
+/**
+ * lulz - Worker Queue Examples
+ * 
+ * Demonstrates:
+ *   - taskQueue: Standalone task queue with worker pool
+ *   - worker: Flow-integrated worker node
+ *   - parallelMap: Parallel array processing
+ *   - cpuTask: Quick wrapper for CPU-bound functions
+ */
+
+import { 
+  flow, 
+  inject, 
+  debug, 
+  func,
+  map,
+  split,
+  taskQueue, 
+  worker, 
+  parallelMap, 
+  cpuTask 
+} from './index.js';
+
+import { cpus } from 'os';
+
+console.log(`\n🖥️  System has ${cpus().length} CPU cores\n`);
+
+
+// ═══════════════════════════════════════════════════════════════
+// Example 1: Basic taskQueue
+// ═══════════════════════════════════════════════════════════════
+
+console.log('═══ Example 1: Basic taskQueue ═══\n');
+
+// Create a queue with a simple handler
+const queue1 = taskQueue({
+  workers: 2,
+  handler: (data) => {
+    // Simulate CPU work
+    let result = 0;
+    for (let i = 0; i < 1_000_000; i++) {
+      result += Math.sqrt(i);
+    }
+    return { input: data, computed: Math.round(result) };
+  }
+});
+
+// Listen for results
+queue1.on('result', ({ id, result }) => {
+  console.log(`[Queue1] Task ${id} completed:`, result);
+});
+
+queue1.on('error', ({ id, error }) => {
+  console.error(`[Queue1] Task ${id} failed:`, error);
+});
+
+queue1.on('drain', () => {
+  console.log('[Queue1] All tasks completed!\n');
+});
+
+// Submit tasks
+console.log('[Queue1] Submitting 5 tasks...');
+for (let i = 1; i <= 5; i++) {
+  queue1.submit({ id: i, data: i * 10 });
+}
+
+
+// ═══════════════════════════════════════════════════════════════
+// Example 2: taskQueue with async handler
+// ═══════════════════════════════════════════════════════════════
+
+setTimeout(() => {
+  console.log('═══ Example 2: Async Handler ═══\n');
+
+  const queue2 = taskQueue({
+    workers: 3,
+    handler: async (data) => {
+      // Simulate async work (API call, file I/O, etc.)
+      await new Promise(r => setTimeout(r, 100 + Math.random() * 200));
+      return `Processed: ${data.toUpperCase()}`;
+    }
+  });
+
+  const results = [];
+  
+  queue2.on('result', ({ id, result }) => {
+    results.push(result);
+    console.log(`[Queue2] Task ${id}: ${result}`);
+  });
+
+  queue2.on('drain', () => {
+    console.log(`[Queue2] All done! Results:`, results);
+    console.log();
+  });
+
+  // Submit multiple tasks
+  ['apple', 'banana', 'cherry', 'date', 'elderberry'].forEach((fruit, i) => {
+    queue2.submit({ id: i + 1, data: fruit });
+  });
+
+}, 1500);
+
+
+// ═══════════════════════════════════════════════════════════════
+// Example 3: taskQueue statistics and control
+// ═══════════════════════════════════════════════════════════════
+
+setTimeout(() => {
+  console.log('═══ Example 3: Queue Statistics ═══\n');
+
+  const queue3 = taskQueue({
+    workers: 2,
+    handler: async (data) => {
+      await new Promise(r => setTimeout(r, 150));
+      return data * 2;
+    }
+  });
+
+  queue3.on('result', ({ id, result }) => {
+    console.log(`[Queue3] Task ${id} = ${result}`);
+    console.log(`[Queue3] Stats:`, queue3.stats());
+  });
+
+  // Submit tasks and check stats
+  console.log('[Queue3] Initial stats:', queue3.stats());
+  
+  queue3.submitAll([
+    { id: 'a', data: 10 },
+    { id: 'b', data: 20 },
+    { id: 'c', data: 30 },
+    { id: 'd', data: 40 },
+  ]);
+
+  console.log('[Queue3] After submit:', queue3.stats());
+
+  // Wait for completion
+  queue3.drain().then(() => {
+    console.log('[Queue3] Final stats:', queue3.stats());
+    console.log();
+  });
+
+}, 3000);
+
+
+// ═══════════════════════════════════════════════════════════════
+// Example 4: worker() in a flow
+// ═══════════════════════════════════════════════════════════════
+
+setTimeout(() => {
+  console.log('═══ Example 4: worker() in Flow ═══\n');
+
+  // Heavy computation that runs in worker thread
+  const heavyComputation = (n) => {
+    let result = 0;
+    for (let i = 0; i < n * 100000; i++) {
+      result += Math.sin(i) * Math.cos(i);
+    }
+    return Math.round(result * 1000) / 1000;
+  };
+
+  const app = flow([
+    // Input numbers
+    ['numbers', 
+      // Process in worker threads (doesn't block main thread!)
+      worker({ 
+        workers: 2, 
+        handler: heavyComputation 
+      }), 
+      'computed'
+    ],
+    
+    // Log results
+    ['computed', debug({ name: 'result', complete: true })],
+  ]);
+
+  // Inject numbers to process
+  console.log('[Worker Flow] Processing numbers in parallel workers...');
+  [10, 20, 30, 40, 50].forEach((n, i) => {
+    setTimeout(() => {
+      console.log(`[Worker Flow] Submitting: ${n}`);
+      app.emit('numbers', { payload: n, id: i });
+    }, i * 100);
+  });
+
+}, 5000);
+
+
+// ═══════════════════════════════════════════════════════════════
+// Example 5: parallelMap for batch processing
+// ═══════════════════════════════════════════════════════════════
+
+setTimeout(() => {
+  console.log('\n═══ Example 5: parallelMap ═══\n');
+
+  // Process entire arrays in parallel
+  const batchProcessor = flow([
+    ['batch',
+      parallelMap({
+        workers: 4,
+        fn: (item) => {
+          // Each item processed in its own worker
+          return {
+            original: item,
+            squared: item * item,
+            sqrt: Math.sqrt(item),
+          };
+        }
+      }),
+      'processed'
+    ],
+    
+    ['processed', debug({ name: 'batch-result', complete: true })],
+  ]);
+
+  console.log('[parallelMap] Processing array [1, 4, 9, 16, 25]...');
+  batchProcessor.emit('batch', { payload: [1, 4, 9, 16, 25] });
+
+}, 7500);
+
+
+// ═══════════════════════════════════════════════════════════════
+// Example 6: cpuTask shorthand
+// ═══════════════════════════════════════════════════════════════
+
+setTimeout(() => {
+  console.log('\n═══ Example 6: cpuTask Shorthand ═══\n');
+
+  // cpuTask is a quick way to wrap CPU-intensive functions
+  const fibonacci = (n) => {
+    if (n <= 1) return n;
+    let a = 0, b = 1;
+    for (let i = 2; i <= n; i++) {
+      [a, b] = [b, a + b];
+    }
+    return b;
+  };
+
+  const app = flow([
+    ['input',
+      cpuTask(fibonacci),  // Runs in worker thread
+      debug({ name: 'fibonacci' })
+    ],
+  ]);
+
+  console.log('[cpuTask] Computing Fibonacci numbers...');
+  [10, 20, 30, 40, 45].forEach((n, i) => {
+    setTimeout(() => {
+      console.log(`[cpuTask] fib(${n}) = ...`);
+      app.emit('input', { payload: n });
+    }, i * 200);
+  });
+
+}, 9000);
+
+
+// ═══════════════════════════════════════════════════════════════
+// Example 7: Real-world pattern - Image Processing Pipeline
+// ═══════════════════════════════════════════════════════════════
+
+setTimeout(() => {
+  console.log('\n═══ Example 7: Image Processing Pipeline ═══\n');
+
+  // Simulated image processing functions
+  const processImage = (img) => {
+    // Simulate resize
+    const resized = { ...img, width: 800, height: 600 };
+    // Simulate compress
+    const compressed = { ...resized, size: Math.round(img.size * 0.3) };
+    // Simulate watermark
+    const final = { ...compressed, watermark: true };
+    return final;
+  };
+
+  const imageProcessor = flow([
+    // Images come in
+    ['upload', func({ func: (msg) => {
+      console.log(`[Image] Uploading: ${msg.payload.name} (${msg.payload.size} bytes)`);
+      return msg;
+    }}), 'image'],
+
+    // Process in parallel workers
+    ['image',
+      worker({
+        workers: 2,
+        handler: processImage
+      }),
+      'processed'
+    ],
+
+    // Save results
+    ['processed', 
+      func({ func: (msg) => {
+        const p = msg.payload;
+        console.log(`[Image] Done: ${p.name} → ${p.width}x${p.height}, ${p.size} bytes, watermark: ${p.watermark}`);
+        return msg;
+      }}),
+      'saved'
+    ],
+  ]);
+
+  // Simulate image uploads
+  const images = [
+    { name: 'photo1.jpg', size: 5000000, width: 4000, height: 3000 },
+    { name: 'photo2.jpg', size: 3500000, width: 3000, height: 2000 },
+    { name: 'photo3.jpg', size: 8000000, width: 6000, height: 4000 },
+  ];
+
+  images.forEach((img, i) => {
+    setTimeout(() => {
+      imageProcessor.emit('upload', { payload: img });
+    }, i * 300);
+  });
+
+}, 11000);
+
+
+// ═══════════════════════════════════════════════════════════════
+// Example 8: Error handling in workers
+// ═══════════════════════════════════════════════════════════════
+
+setTimeout(() => {
+  console.log('\n═══ Example 8: Error Handling ═══\n');
+
+  const riskyQueue = taskQueue({
+    workers: 2,
+    handler: (data) => {
+      if (data < 0) {
+        throw new Error(`Negative numbers not allowed: ${data}`);
+      }
+      return Math.sqrt(data);
+    }
+  });
+
+  riskyQueue.on('result', ({ id, result }) => {
+    console.log(`[Risky] Task ${id} succeeded: √${id} = ${result}`);
+  });
+
+  riskyQueue.on('error', ({ id, error }) => {
+    console.log(`[Risky] Task ${id} FAILED: ${error}`);
+  });
+
+  riskyQueue.on('drain', () => {
+    console.log('[Risky] Queue drained (some may have failed)\n');
+  });
+
+  // Submit mix of valid and invalid
+  [4, 9, -1, 16, -25, 36].forEach((n, i) => {
+    riskyQueue.submit({ id: n, data: n });
+  });
+
+}, 13000);
+
+
+// ═══════════════════════════════════════════════════════════════
+// Example 9: Combining with RxJS-style operators
+// ═══════════════════════════════════════════════════════════════
+
+setTimeout(() => {
+  console.log('\n═══ Example 9: Workers + Rx Operators ═══\n');
+
+  const dataProcessor = flow([
+    // Split array into individual items
+    ['data', split(), 'item'],
+    
+    // Process each item in worker
+    ['item',
+      map({ fn: (x) => x * 2 }),  // Quick transform in main thread
+      worker({                     // Heavy work in worker
+        workers: 2,
+        handler: (n) => {
+          let sum = 0;
+          for (let i = 0; i < n * 10000; i++) sum += Math.random();
+          return { n, sum: Math.round(sum) };
+        }
+      }),
+      'processed'
+    ],
+
+    ['processed', debug({ name: 'processed' })],
+  ]);
+
+  console.log('[Rx+Workers] Processing [5, 10, 15, 20, 25]...');
+  dataProcessor.emit('data', { payload: [5, 10, 15, 20, 25] });
+
+}, 15000);
+
+
+// ═══════════════════════════════════════════════════════════════
+// Cleanup and summary
+// ═══════════════════════════════════════════════════════════════
+
+setTimeout(() => {
+  console.log('\n═══ Summary ═══\n');
+  console.log('Worker utilities in lulz:');
+  console.log('');
+  console.log('  taskQueue({ workers, handler })');
+  console.log('    → Standalone EventEmitter-based task queue');
+  console.log('    → .submit(task), .submitAll(tasks), .drain()');
+  console.log('    → Events: result, error, drain, idle');
+  console.log('');
+  console.log('  worker({ workers, handler })');
+  console.log('    → Flow node that processes packets in workers');
+  console.log('    → Non-blocking, preserves packet metadata');
+  console.log('');
+  console.log('  parallelMap({ workers, fn })');
+  console.log('    → Process arrays with parallel workers');
+  console.log('    → Preserves order, emits complete array');
+  console.log('');
+  console.log('  cpuTask(fn)');
+  console.log('    → Quick wrapper: cpuTask(x => x*x)');
+  console.log('    → Shorthand for worker({ handler: fn })');
+  console.log('');
+  console.log('═══ All Examples Complete ═══\n');
+
+  // Terminate queues
+  queue1.terminate();
+  
+  process.exit(0);
+}, 18000);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "lulz",
-  "version": "2.0.2",
+  "version": "2.0.3",
   "description": "A reactive dataflow system that makes coders happy",
   "type": "module",
   "main": "index.js",
@@ -13,6 +13,7 @@
   "scripts": {
     "test": "node test.js",
     "examples": "node examples.js",
+    "examples:queue": "node examples-queue.js",
     "save": "git add .; git commit -m 'Updated Release'; npm version patch; npm publish; git push --follow-tags;"
   },
   "keywords": [