@travetto/worker 3.0.0-rc.2 → 3.0.0-rc.4

package/README.md

@@ -1,5 +1,5 @@
 <!-- This file was generated by @travetto/doc and should not be modified directly -->
-<!-- Please modify https://github.com/travetto/travetto/tree/main/module/worker/doc.ts and execute "npx trv doc" to rebuild -->
+<!-- Please modify https://github.com/travetto/travetto/tree/main/module/worker/DOC.ts and execute "npx trv doc" to rebuild -->
 # Worker
 ## Process management utilities, with a focus on inter-process communication
 
@@ -11,7 +11,7 @@ npm install @travetto/worker
 This module provides the necessary primitives for handling dependent workers.  A worker can be an individual actor or could be a pool of workers. Node provides ipc (inter-process communication) functionality out of the box. This module builds upon that by providing enhanced event management, richer process management, as well as constructs for orchestrating a conversation between two processes.
 
 ## Execution Pools
-With respect to managing multiple executions, [WorkPool](https://github.com/travetto/travetto/tree/main/module/worker/src/pool.ts#L23) is provided to allow for concurrent operation, and processing of jobs concurrently.  To manage the flow of jobs, there are various [WorkSet](https://github.com/travetto/travetto/tree/main/module/worker/src/input/types.ts#L4) implementation that allow for a wide range of use cases.
+With respect to managing multiple executions, [WorkPool](https://github.com/travetto/travetto/tree/main/module/worker/src/pool.ts#L31) is provided to allow for concurrent operation, and processing of jobs concurrently.  To manage the flow of jobs, there are various [WorkSet](https://github.com/travetto/travetto/tree/main/module/worker/src/input/types.ts#L4) implementation that allow for a wide range of use cases.
 
 The only provided [WorkSet](https://github.com/travetto/travetto/tree/main/module/worker/src/input/types.ts#L4) is the [IterableWorkSet](https://github.com/travetto/travetto/tree/main/module/worker/src/input/iterable.ts#L11) which supports all `Iterable` and `Iterator` sources.  Additionally, the module provides [ManualAsyncIterator](https://github.com/travetto/travetto/tree/main/module/worker/src/input/async-iterator.ts#L6) which allows for manual control of iteration, which is useful for event driven work loads.
 
@@ -19,7 +19,7 @@ Below is a pool that will convert images on demand, while queuing as needed.
 
 **Code: Image processing queue, with a fixed batch/pool size**
 ```typescript
-import { ExecUtil, ExecutionState } from '@travetto/boot';
+import { ExecUtil, ExecutionState } from '@travetto/base';
 import { Worker, WorkPool, IterableWorkSet, ManualAsyncIterator } from '@travetto/worker';
 
 class ImageProcessor implements Worker<string> {
@@ -46,7 +46,7 @@ class ImageProcessor implements Worker<string> {
   }
 }
 
-export class ImageCompressor extends WorkPool<string, ImageProcessor> {
+export class ImageCompressor extends WorkPool<string> {
 
   pendingImages = new ManualAsyncIterator<string>();
 
@@ -68,14 +68,14 @@ Once a pool is constructed, it can be shutdown by calling the `.shutdown()` meth
 
 ## IPC Support
 
-Within the `comm` package, there is support for two primary communication elements: [ChildCommChannel](https://github.com/travetto/travetto/tree/main/module/worker/src/comm/child.ts#L6) and [ParentCommChannel](https://github.com/travetto/travetto/tree/main/module/worker/src/comm/parent.ts#L11).  Usually [ParentCommChannel](https://github.com/travetto/travetto/tree/main/module/worker/src/comm/parent.ts#L11) indicates it is the owner of the sub process.  [ChildCommChannel](https://github.com/travetto/travetto/tree/main/module/worker/src/comm/child.ts#L6) indicates that it has been created/spawned/forked by the parent and will communicate back to it's parent.  This generally means that a [ParentCommChannel](https://github.com/travetto/travetto/tree/main/module/worker/src/comm/parent.ts#L11) can be destroyed (i.e. killing the subprocess) where a [ChildCommChannel](https://github.com/travetto/travetto/tree/main/module/worker/src/comm/child.ts#L6) can only exit the process, but the channel cannot be destroyed.
+Within the `comm` package, there is support for two primary communication elements: [ChildCommChannel](https://github.com/travetto/travetto/tree/main/module/worker/src/comm/child.ts#L6) and [ParentCommChannel](https://github.com/travetto/travetto/tree/main/module/worker/src/comm/parent.ts#L10).  Usually [ParentCommChannel](https://github.com/travetto/travetto/tree/main/module/worker/src/comm/parent.ts#L10) indicates it is the owner of the sub process.  [ChildCommChannel](https://github.com/travetto/travetto/tree/main/module/worker/src/comm/child.ts#L6) indicates that it has been created/spawned/forked by the parent and will communicate back to it's parent.  This generally means that a [ParentCommChannel](https://github.com/travetto/travetto/tree/main/module/worker/src/comm/parent.ts#L10) can be destroyed (i.e. killing the subprocess) where a [ChildCommChannel](https://github.com/travetto/travetto/tree/main/module/worker/src/comm/child.ts#L6) can only exit the process, but the channel cannot be destroyed.
 
 ### IPC as a Worker
-A common pattern is to want to model a sub process as a worker, to be a valid candidate in a [WorkPool](https://github.com/travetto/travetto/tree/main/module/worker/src/pool.ts#L23).  The [WorkUtil](https://github.com/travetto/travetto/tree/main/module/worker/src/util.ts#L14) class provides a utility to facilitate this desire.
+A common pattern is to want to model a sub process as a worker, to be a valid candidate in a [WorkPool](https://github.com/travetto/travetto/tree/main/module/worker/src/pool.ts#L31).  The [WorkUtil](https://github.com/travetto/travetto/tree/main/module/worker/src/util.ts#L14) class provides a utility to facilitate this desire.
 
 **Code: Spawned Worker**
 ```typescript
-import { ExecutionState } from '@travetto/boot';
+import { ExecutionState } from '@travetto/base';
 
 import { ParentCommChannel } from './comm/parent';
 import { Worker } from './pool';
@@ -112,17 +112,17 @@ export class WorkUtil {
 }
 ```
 
-When creating your work, via process spawning, you will need to provide the script (and any other features you would like in `SpawnConfig`).   Additionally you must, at a minimum, provide functionality to run whenever an input element is up for grabs in the input source.  This method will be provided the communication channel ([ParentCommChannel](https://github.com/travetto/travetto/tree/main/module/worker/src/comm/parent.ts#L11)) and the input value.  A simple example could look like:
+When creating your work, via process spawning, you will need to provide the script (and any other features you would like in `SpawnConfig`).   Additionally you must, at a minimum, provide functionality to run whenever an input element is up for grabs in the input source.  This method will be provided the communication channel ([ParentCommChannel](https://github.com/travetto/travetto/tree/main/module/worker/src/comm/parent.ts#L10)) and the input value.  A simple example could look like:
 
 **Code: Spawning Pool**
 ```typescript
+import { ExecUtil } from '@travetto/base';
 import { WorkPool, WorkUtil, IterableWorkSet } from '@travetto/worker';
-import { ExecUtil, PathUtil } from '@travetto/boot';
 
 export async function main(): Promise<void> {
   const pool = new WorkPool(() =>
-    WorkUtil.spawnedWorker<{ data: string }, string>(
-      () => ExecUtil.forkMain(PathUtil.resolveUnix(__dirname, 'spawned.ts')),
+    WorkUtil.spawnedWorker<{ data: string }, number>(
+      () => ExecUtil.spawn('trv', ['main', 'support/main.spawned.ts']),
       ch => ch.once('ready'), // Wait for child to indicate it is ready
       async (channel, inp) => {
         const res = channel.once('response'); //  Register response listener
@@ -131,21 +131,20 @@ export async function main(): Promise<void> {
         const { data } = await res; // Get answer
         console.log('Request complete', { input: inp, output: data });
 
-        if (!(inp + inp === data)) {
+        if (!(`${inp + inp}` === data)) {
           // Ensure the answer is double the input
           throw new Error('Did not get the double');
         }
       }
     )
   );
-
-  await pool.process(new IterableWorkSet([1, 2, 3, 4, 5])).then(x => pool.shutdown());
+  await pool.process(new IterableWorkSet([1, 2, 3, 4, 5]));
 }
 ```
 
 **Code: Spawned Worker**
 ```typescript
-import * as timers from 'timers/promises';
+import timers from 'timers/promises';
 import { ChildCommChannel } from '@travetto/worker';
 
 export async function main(): Promise<void> {
@@ -163,11 +162,12 @@ export async function main(): Promise<void> {
 
 **Terminal: Output**
 ```bash
-$ node @travetto/base/bin/main ./doc/spawner.ts 
+$ trv main ./support/main.spawner.ts
 
 Request complete { input: 1, output: 2 }
 Request complete { input: 2, output: 4 }
 Request complete { input: 3, output: 6 }
 Request complete { input: 4, output: 8 }
 Request complete { input: 5, output: 10 }
+
 ```

package/package.json

@@ -1,7 +1,6 @@
 {
   "name": "@travetto/worker",
-  "displayName": "Worker",
-  "version": "3.0.0-rc.2",
+  "version": "3.0.0-rc.4",
   "description": "Process management utilities, with a focus on inter-process communication",
   "keywords": [
     "exec",
@@ -17,18 +16,21 @@
     "name": "Travetto Framework"
   },
   "files": [
-    "index.ts",
+    "__index__.ts",
     "src"
   ],
-  "main": "index.ts",
+  "main": "__index__.ts",
   "repository": {
     "url": "https://github.com/travetto/travetto.git",
     "directory": "module/worker"
   },
   "dependencies": {
-    "@travetto/base": "^3.0.0-rc.2",
+    "@travetto/base": "^3.0.0-rc.4",
     "@types/generic-pool": "^3.1.11",
-    "generic-pool": "^3.8.2"
+    "generic-pool": "^3.9.0"
+  },
+  "travetto": {
+    "displayName": "Worker"
   },
   "publishConfig": {
     "access": "public"

package/src/comm/channel.ts

@@ -1,7 +1,7 @@
 import { ChildProcess } from 'child_process';
 import { EventEmitter } from 'events';
 
-import { ExecUtil } from '@travetto/boot';
+import { ExecUtil } from '@travetto/base';
 
 /**
  * Channel that represents communication between parent/child

package/src/comm/parent.ts

@@ -1,7 +1,6 @@
 import { ChildProcess } from 'child_process';
 
-import { ExecutionState } from '@travetto/boot';
-import { ShutdownManager } from '@travetto/base';
+import { ShutdownManager, ExecutionState } from '@travetto/base';
 
 import { ProcessCommChannel } from './channel';
 
@@ -14,7 +13,7 @@ export class ParentCommChannel<U = unknown> extends ProcessCommChannel<ChildProc
 
   constructor(state: ExecutionState) {
     super(state.process);
-    ShutdownManager.onShutdown({ close: () => this.destroy() });
+    ShutdownManager.onShutdown(this, { close: () => this.destroy() });
     this.#complete = state.result
       .finally(() => { this.proc = undefined; });
   }

package/src/input/async-iterator.ts

@@ -3,17 +3,24 @@ import { Util } from '@travetto/base';
 /**
  * Manual async iterator.  Items are added manually, and consumed asynchronously
  */
-export class ManualAsyncIterator<X> implements AsyncIterator<X> {
+export class ManualAsyncIterator<X> implements AsyncIterator<X>, AsyncIterable<X> {
 
   #queue: X[] = [];
   #done = false;
   #ready = Util.resolvablePromise();
+  #size: number;
 
   /**
    * Initial set of items
    */
   constructor(initial: Iterable<X> = []) {
     this.#queue.push(...initial);
+    this.#size = this.#queue.length;
+  }
+
+  // Allow for iteration
+  [Symbol.asyncIterator](): AsyncIterator<X> {
+    return this;
   }
 
   /**
@@ -38,6 +45,7 @@ export class ManualAsyncIterator<X> implements AsyncIterator<X> {
     } else {
       this.#queue.unshift(...item);
     }
+    this.#size += item.length;
     this.#ready.resolve();
   }
 
@@ -48,4 +56,11 @@ export class ManualAsyncIterator<X> implements AsyncIterator<X> {
     this.#done = true;
     this.#ready.resolve();
   }
+
+  /**
+   * Get size, will change as items are added
+   */
+  get size(): number {
+    return this.#size;
+  }
 }

package/src/input/iterable.ts

@@ -13,6 +13,7 @@ export class IterableWorkSet<X> implements WorkSet<X> {
   #src: Itr<X>;
   #ondeck?: X;
   #done = false;
+  #size?: number;
 
   constructor(src: Iterable<X> | AsyncIterable<X> | (() => Generator<X>) | (() => AsyncGenerator<X>) | Itr<X>) {
     if ('next' in src) {
@@ -26,6 +27,11 @@ export class IterableWorkSet<X> implements WorkSet<X> {
         this.#src = src();
       }
     }
+    if (Array.isArray(src)) {
+      this.#size = src.length;
+    } else if (src instanceof Set) {
+      this.#size = src.size;
+    }
   }
 
   /**
@@ -57,4 +63,11 @@ export class IterableWorkSet<X> implements WorkSet<X> {
     this.#ondeck = undefined;
     return out;
   }
+
+  /**
+   * Get size, if defined
+   */
+  get size(): number | undefined {
+    return this.#size;
+  }
 }

package/src/input/types.ts

@@ -10,4 +10,8 @@ export interface WorkSet<X> {
    * Get next item
    */
   next(): X | Promise<X>;
+  /**
+   * Total size of work set
+   */
+  size?: number;
 }

package/src/pool.ts

@@ -1,9 +1,10 @@
-import * as os from 'os';
-import * as gp from 'generic-pool';
+import os from 'os';
+import gp from 'generic-pool';
 
-import { ShutdownManager, Util } from '@travetto/base';
+import { GlobalEnv, ShutdownManager, TimeUtil } from '@travetto/base';
 
 import { WorkSet } from './input/types';
+import { ManualAsyncIterator } from '../src/input/async-iterator';
 
 /**
  * Worker definition
@@ -17,17 +18,24 @@ export interface Worker<X> {
   release?(): unknown;
 }
 
+type WorkPoolProcessConfig<X> = {
+  shutdownOnComplete?: boolean;
+  onComplete?: (ev: WorkCompletionEvent<X>) => (void | Promise<void>);
+};
+
+type WorkCompletionEvent<X> = { idx: number, total?: number, value: X };
+
 /**
  * Work pool support
  */
-export class WorkPool<X, T extends Worker<X>> {
+export class WorkPool<X> {
 
   static DEFAULT_SIZE = Math.min(os.cpus().length - 1, 4);
 
   /**
    * Generic-pool pool
    */
-  #pool: gp.Pool<T>;
+  #pool: gp.Pool<Worker<X>>;
   /**
    * Number of acquisitions in process
    */
@@ -42,12 +50,22 @@ export class WorkPool<X, T extends Worker<X>> {
    */
   #createErrors = 0;
 
+  /**
+   * Are we tracing
+   */
+  #trace: boolean;
+
+  /**
+   * Cleanup for shutdown
+   */
+  #shutdownCleanup?: () => void;
+
   /**
    *
    * @param getWorker Produces a new worker for the pool
    * @param opts Pool options
    */
-  constructor(getWorker: () => Promise<T> | T, opts?: gp.Options) {
+  constructor(getWorker: () => Promise<Worker<X>> | Worker<X>, opts?: gp.Options) {
     const args = {
       max: WorkPool.DEFAULT_SIZE,
       min: 1,
@@ -59,16 +77,21 @@ export class WorkPool<X, T extends Worker<X>> {
     this.#pool = gp.createPool({
       create: () => this.#createAndTrack(getWorker, args),
       destroy: x => this.destroy(x),
-      validate: async (x: T) => x.active
+      validate: async (x: Worker<X>) => x.active
     }, args);
 
-    ShutdownManager.onShutdown(`worker.pool.${this.constructor.name}`, () => this.shutdown());
+    this.#shutdownCleanup = ShutdownManager.onShutdown(`worker.pool.${this.constructor.name}`, () => {
+      this.#shutdownCleanup = undefined;
+      this.shutdown();
+    });
+
+    this.#trace = !!GlobalEnv.debug?.includes('@travetto/worker');
   }
 
   /**
    * Creates and tracks new worker
    */
-  async #createAndTrack(getWorker: () => Promise<T> | T, opts: gp.Options): Promise<T> {
+  async #createAndTrack(getWorker: () => Promise<Worker<X>> | Worker<X>, opts: gp.Options): Promise<Worker<X>> {
     try {
       this.#pendingAcquires += 1;
       const res = await getWorker();
@@ -83,7 +106,7 @@ export class WorkPool<X, T extends Worker<X>> {
     } catch (err) {
       if (this.#createErrors++ > opts.max!) { // If error count is bigger than pool size, we broke
         console.error('Failed in creating pool', { error: err });
-        process.exit(1);
+        await ShutdownManager.exit(1);
       }
       throw err;
     } finally {
@@ -94,23 +117,25 @@ export class WorkPool<X, T extends Worker<X>> {
   /**
    * Destroy the worker
    */
-  async destroy(worker: T): Promise<void> {
-    console.debug('Destroying', { pid: process.pid, worker: worker.id });
+  async destroy(worker: Worker<X>): Promise<void> {
+    if (this.#trace) {
+      console.debug('Destroying', { pid: process.pid, worker: worker.id });
+    }
     return worker.destroy();
   }
 
   /**
    * Free worker on completion
    */
-  async release(worker: T): Promise<void> {
-    console.debug('Releasing', { pid: process.pid, worker: worker.id });
+  async release(worker: Worker<X>): Promise<void> {
+    if (this.#trace) {
+      console.debug('Releasing', { pid: process.pid, worker: worker.id });
+    }
     try {
       if (worker.active) {
-        if (worker.release) {
-          try {
-            await worker.release();
-          } catch { }
-        }
+        try {
+          await worker.release?.();
+        } catch { }
         await this.#pool.release(worker);
       } else {
         await this.#pool.destroy(worker);
@@ -121,36 +146,63 @@ export class WorkPool<X, T extends Worker<X>> {
   /**
    * Process a given input source
    */
-  async process(src: WorkSet<X>): Promise<void> {
+  async process(src: WorkSet<X>, cfg: WorkPoolProcessConfig<X> = {}): Promise<void> {
     const pending = new Set<Promise<unknown>>();
+    let count = 0;
+
+    if (src.size && cfg.onComplete) {
+      // eslint-disable-next-line @typescript-eslint/consistent-type-assertions
+      cfg.onComplete({ value: undefined as X, idx: 0, total: src.size });
+    }
 
     while (await src.hasNext()) {
       const worker = (await this.#pool.acquire())!;
-      console.debug('Acquired', { pid: process.pid, worker: worker.id });
+      if (this.#trace) {
+        console.debug('Acquired', { pid: process.pid, worker: worker.id });
+      }
       const nextInput = await src.next();
 
       const completion = worker.execute(nextInput)
         .catch(err => this.#errors.push(err)) // Catch error
-        .finally(() => this.release(worker));
+        .finally(() => this.release(worker))
+        .finally(() => cfg.onComplete?.({ value: nextInput, idx: count += 1, total: src.size }));
 
       completion.finally(() => pending.delete(completion));
 
       pending.add(completion);
     }
 
-    await Promise.all(Array.from(pending));
+    try {
+      await Promise.all(Array.from(pending));
 
-    if (this.#errors.length) {
-      throw this.#errors[0];
+      if (this.#errors.length) {
+        throw this.#errors[0];
+      }
+    } finally {
+      if (cfg.shutdownOnComplete !== false) {
+        await this.shutdown();
+      }
     }
   }
 
+  /**
+   * Process a given input source as an async iterable, emitting on completion
+   */
+  iterateProcess(src: WorkSet<X>, shutdownOnComplete?: boolean): AsyncIterable<WorkCompletionEvent<X>> {
+    const itr = new ManualAsyncIterator<WorkCompletionEvent<X>>();
+    const res = this.process(src, { onComplete: ev => itr.add(ev), shutdownOnComplete });
+    res.finally(() => itr.close());
+    return itr;
+  }
+
   /**
    * Shutdown pool
    */
   async shutdown(): Promise<void> {
+    this.#shutdownCleanup?.();
+
     while (this.#pendingAcquires) {
-      await Util.wait(10);
+      await TimeUtil.wait(10);
     }
     await this.#pool.drain();
     await this.#pool.clear();

package/src/support/barrier.ts

@@ -1,4 +1,7 @@
-import { Util, ShutdownManager, TimeSpan } from '@travetto/base';
+import { setTimeout } from 'timers/promises';
+
+import { ShutdownManager, TimeSpan, Util } from '@travetto/base';
+
 import { Timeout } from './timeout';
 
 function canCancel(o: unknown): o is { cancel(): unknown } {
@@ -10,6 +13,23 @@ function canCancel(o: unknown): o is { cancel(): unknown } {
  * Build an execution barrier to handle various limitations
  */
 export class Barrier {
+  /**
+   * Listen for an unhandled event, as a promise
+   */
+  static listenForUnhandled(): Promise<never> & { cancel: () => void } {
+    const uncaught = Util.resolvablePromise<never>();
+    const uncaughtWithCancel: typeof uncaught & { cancel?: () => void } = uncaught;
+    const cancel = ShutdownManager.onUnhandled(err => { setTimeout(1).then(() => uncaught.reject(err)); return true; }, 0);
+    uncaughtWithCancel.cancel = (): void => {
+      cancel(); // Remove the handler
+      // eslint-disable-next-line @typescript-eslint/consistent-type-assertions
+      uncaughtWithCancel.resolve(undefined as never); // Close the promise
+    };
+
+    // eslint-disable-next-line @typescript-eslint/consistent-type-assertions
+    return uncaughtWithCancel as (Promise<never> & { cancel: () => void });
+  }
+
   #support: string[] = [];
   #barriers = new Map<string, Promise<unknown>>([]);
 
@@ -21,7 +41,7 @@ export class Barrier {
       this.add(new Timeout(timeout).wait(), true);
     }
     if (unhandled) {
-      this.add(ShutdownManager.listenForUnhandled());
+      this.add(Barrier.listenForUnhandled());
     }
   }
 

package/src/support/timeout.ts

@@ -1,4 +1,4 @@
-import { TimeSpan, Util } from '@travetto/base';
+import { TimeSpan, TimeUtil, Util } from '@travetto/base';
 import { ExecutionError } from './error';
 
 /**
@@ -12,7 +12,7 @@ export class Timeout extends ExecutionError {
 
   constructor(duration: number | TimeSpan, op: string = 'Operation') {
     super(`${op} timed out after ${duration}${typeof duration === 'number' ? 'ms' : ''}`);
-    this.#duration = Util.timeToMs(duration);
+    this.#duration = TimeUtil.timeToMs(duration);
   }
 
   /**

package/src/util.ts

@@ -1,4 +1,4 @@
-import { ExecutionState } from '@travetto/boot';
+import { ExecutionState } from '@travetto/base';
 
 import { ParentCommChannel } from './comm/parent';
 import { Worker } from './pool';