@hotmeshio/hotmesh 0.0.30 → 0.0.32

package/README.md

@@ -1,13 +1,9 @@
 # HotMesh
 ![alpha release](https://img.shields.io/badge/release-alpha-yellow)
 
-Elevate Redis from an in-memory data cache, and turn your unpredictable functions into unbreakable workflows.
+Elevate Redis from an in-memory data cache, and turn your unpredictable functions into unbreakable workflows. HotMesh is a distributed orchestration engine that governs the execution of your functions, ensuring they always complete successfully.
 
-**HotMesh** is a wrapper for Redis that exposes concepts like ‘activities’, ‘workflows’, and 'jobs'. Behind the scenes, it uses *Redis Data* (Hash, ZSet, List); *Redis Streams* (XReadGroup, XAdd, XLen, etc); and *Redis Publish/Subscribe*.
-
-It's still Redis in the background, but your functions are run as *reentrant processes* and are executed in a distributed environment, with all the benefits of a distributed system, including fault tolerance, scalability, and high availability.
-
-Write functions in your own preferred style, and let Redis govern their execution with its unmatched performance.
+*Write functions in your own preferred style, and let Redis govern their execution.*
 
 ## Install
 [![npm version](https://badge.fury.io/js/%40hotmeshio%2Fhotmesh.svg)](https://badge.fury.io/js/%40hotmeshio%2Fhotmesh)
@@ -15,82 +11,89 @@ Write functions in your own preferred style, and let Redis govern their executio
 ```sh
 npm install @hotmeshio/hotmesh
 ```
-
 ## Design
-1. Start with any ordinary class. Pay attention to unpredictable functions: those that execute slowly, cause problems at scale, or simply fail to return now and then. *Note how the `flaky` function throws an error 50% of the time. This is exactly the type of function that can be fixed using HotMesh.*
-    ```javascript
-    //myworkflow.ts
-
-    export class MyWorkflow {
-
-      async run(name: string): Promise<string> {
-        const hi = await this.flaky(name);
-        const hello = await this.greet(name);
-        return `${hi} ${hello}`;
-      }
 
-      //this function is unpredictable and will fail 50% of the time
-      async flaky(name: string): Promise<string> {
-        if (Math.random() < 0.5) {
-          throw new Error('Ooops!');
-        }
-        return `Hi, ${name}!`;
-      }
+1. Start by defining **activities**. Activities can be written in any style, using any framework, and can even be legacy functions you've already written. The only requirement is that they return a Promise. *Note how the `saludar` example throws an error 50% of the time. It doesn't matter how unpredictable your functions are, HotMesh will retry as necessary until they succeed.*
+    ```javascript
+    //activities.ts
+    export async function greet(name: string): Promise<string> {
+      return `Hello, ${name}!`;
+    }
 
-      async greet(name: string): Promise<string> {
-        return `Hello, ${name}!`;
+    export async function saludar(nombre: string): Promise<string> {
+      if (Math.random() > 0.5) throw new Error('Random error');
+      return `¡Hola, ${nombre}!`;
+    }
+    ```
+2. Define your **workflow** logic. Include conditional branching, loops, etc to control activity execution. It's vanilla code written in your own coding style. The only requirement is to use `proxyActivities`, ensuring your activities are executed with HotMesh's durability guarantee.
+    ```javascript
+    //workflows.ts
+    import { Durable } from '@hotmeshio/hotmesh';
+    import * as activities from './activities';
+
+    const { greet, saludar } = Durable.workflow
+      .proxyActivities<typeof activities>({
+        activities
+      });
+
+    export async function example(name: string, lang: string): Promise<string> {
+      if (lang === 'es') {
+        return await saludar(name);
+      } else {
+        return await greet(name);
       }
     }
     ```
-2. Import and configure `Redis` and `MeshOS` as shown. List those functions that Redis should govern as durable workflows (like `run` and `flaky`). And that's it! *Your functions don't actually change; rather, their governance does.*
+3. Although you could call your workflow directly (it's just a vanilla function), it's only durable when invoked and orchestrated via HotMesh.
     ```javascript
-    //myworkflow.ts
-
+    //client.ts
+    import { Durable } from '@hotmeshio/hotmesh';
     import Redis from 'ioredis'; //OR `import * as Redis from 'redis';`
-    import { MeshOS } from '@hotmeshio/hotmesh';
-
-    export class MyWorkflow extends MeshOS {
-
-      //configure Redis
-      redisClass = Redis;
-      redisOptions = { host: 'localhost', port: 6379 };
-
-      //list functions to run as durable workflows
-      workflowFunctions = ['run'];
-
-      //list functions to retry and cache
-      proxyFunctions = ['flaky'];
-
-      //no need to change anything else!
-
-      async run(name: string): Promise<string> {
-        const hi = await this.flaky(name);
-        const hello = await this.greet(name);
-        return `${hi} ${hello}`;
-      }
+    import { nanoid } from 'nanoid';
 
-      async flaky(name: string): Promise<string> {
-        if (Math.random() < 0.5) {
-          throw new Error('Ooops!');
+    async function run(): Promise<string> {
+      const client = new Durable.Client({
+        connection: {
+          class: Redis,
+          options: { host: 'localhost', port: 6379 }
         }
-        return `Hi, ${name}!`;
-      }
+      });
 
-      async greet(name: string): Promise<string> {
-        return `Hello, ${name}!`;
-      }
+      const handle = await client.workflow.start({
+        args: ['HotMesh', 'es'],
+        taskQueue: 'hello-world',
+        workflowName: 'example',
+        workflowId: nanoid()
+      });
+
+      return await handle.result();
+      //returns '¡Hola, HotMesh!'
     }
     ```
-3. Invoke your class, providing a unique id (it's now an idempotent workflow and needs a GUID). Nothing changes from the outside, *but Redis now governs the end-to-end execution.* It's guaranteed to succeed, even if it breaks a few times along the way. 
+4. Finally, create a **worker** and link your workflow function. Workers listen for tasks on their assigned Redis stream and invoke your workflow function each time they receive an event.
     ```javascript
-    //mycaller.ts
-
-    const workflow = new MyWorkflow({ id: 'my123', await: true });
-    const response = await workflow.run('World');
-    //Hi, World! Hello, World!
+    //worker.ts
+    import { Durable } from '@hotmeshio/hotmesh';
+    import Redis from 'ioredis';
+    import * as workflows from './workflows';
+
+    async function run() {
+      const worker = await Durable.Worker.create({
+        connection: {
+          class: Redis,
+          options: { host: 'localhost', port: 6379 },
+        },
+        taskQueue: 'hello-world',
+        workflow: workflows.example,
+      });
+
+      await worker.run();
+    }
     ```
 
-Redis governance delivers more than just reliability. Externalizing state fundamentally changes the execution profile for your functions, allowing you to design long-running, durable workflows. The `MeshOS` base class (shown in the examples above) provides additional methods for solving the most common state management challenges.
+Redis governance delivers more than just reliability. Externalizing state fundamentally changes the execution profile for your functions, allowing you to design long-running, durable workflows. The `Durable.workflow` base class (shown in the examples above) provides additional methods for solving the most common state management challenges.
+
+Include statements like `await Durable.workflow.sleep('1 month')` to pause your workflow function for a month, or `await Durable.workflow.waitForSignal('signal1', 'signal2')` to pause your function until all signals have arrived.
 
  - `waitForSignal` | Pause your function and wait for external event(s) before continuing. The *waitForSignal* method will collate and cache the signals and only awaken your function once all signals have arrived.
  - `signal` | Send a signal (and optional payload) to any paused function.

package/build/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@hotmeshio/hotmesh",
-    "version": "0.0.30",
+    "version": "0.0.32",
     "description": "Unbreakable Workflows",
     "main": "build/index.js",
     "types": "build/index.d.ts",

package/build/services/durable/client.js

@@ -76,8 +76,8 @@ class ClientService {
         };
         this.workflow = {
             start: async (options) => {
-                const taskQueueName = options.taskQueue;
-                const workflowName = options.workflowName;
+                const taskQueueName = options.entity ?? options.taskQueue;
+                const workflowName = options.entity ?? options.workflowName;
                 const trc = options.workflowTrace;
                 const spn = options.workflowSpan;
                 //topic is concat of taskQueue and workflowName

package/build/services/durable/workflow.js

@@ -31,16 +31,16 @@ class WorkflowService {
         const workflowSpan = store.get('workflowSpan');
         const COUNTER = store.get('counter');
         const execIndex = COUNTER.counter = COUNTER.counter + 1;
-        const prefix = options.prefix ?? '';
-        //this is risky but MUST be allowed. Users MAY set the workflowId,
-        //but if there is a naming collision, the data from the target entity will be used
-        //as there is know way of knowing if the item was generated via a prior run of the workflow
-        const childJobId = options.workflowId ?? `${prefix}-${workflowId}-$${options.workflowName}${workflowDimension}-${execIndex}`;
+        //NOTE: this is the hash prefix; necessary for the search index to locate the entity
+        //if the hash is a helper, a dash begins it, so it isn't indexed
+        const entityOrEmptyString = options.entity ?? '';
+        //If the workflowId is not provided, it is generated from the entity and the workflow name
+        const childJobId = options.workflowId ?? `${entityOrEmptyString}-${workflowId}-$${options.entity ?? options.workflowName}${workflowDimension}-${execIndex}`;
         const parentWorkflowId = `${workflowId}-f`;
         const client = new client_1.ClientService({
             connection: await connection_1.ConnectionService.connect(worker_1.WorkerService.connection),
         });
-        let handle = await client.workflow.getHandle(options.taskQueue, options.workflowName, childJobId, namespace);
+        let handle = await client.workflow.getHandle(options.entity ?? options.taskQueue, options.entity ?? options.workflowName, childJobId, namespace);
         try {
             return await handle.result(true);
         }
@@ -73,10 +73,12 @@ class WorkflowService {
         const workflowSpan = store.get('workflowSpan');
         const COUNTER = store.get('counter');
         const execIndex = COUNTER.counter = COUNTER.counter + 1;
-        const prefix = options.prefix ?? '';
-        const childJobId = options.workflowId ?? `${prefix}-${workflowId}-$${options.workflowName}${workflowDimension}-${execIndex}`;
+        //NOTE: this is the hash prefix; necessary for the search index to locate the entity
+        const entityOrEmptyString = options.entity ?? '';
+        //If the workflowId is not provided, it is generated from the entity and the workflow name
+        const childJobId = options.workflowId ?? `${entityOrEmptyString}-${workflowId}-$${options.entity ?? options.workflowName}${workflowDimension}-${execIndex}`;
         const parentWorkflowId = `${workflowId}-f`;
-        const workflowTopic = `${options.taskQueue}-${options.workflowName}`;
+        const workflowTopic = `${options.entity ?? options.taskQueue}-${options.entity ?? options.workflowName}`;
         try {
             //get the status; if there is no error, return childJobId (what was spawned)
             const hotMeshClient = await worker_1.WorkerService.getHotMesh(workflowTopic, { namespace });
@@ -247,8 +249,8 @@ class WorkflowService {
             const store = asyncLocalStorage_1.asyncLocalStorage.getStore();
             const workflowId = options.workflowId ?? store.get('workflowId');
             let workflowTopic = store.get('workflowTopic');
-            if (options.taskQueue && options.workflowName) {
-                workflowTopic = `${options.taskQueue}-${options.workflowName}`;
+            if (options.entity || (options.taskQueue && options.workflowName)) {
+                workflowTopic = `${options.entity ?? options.taskQueue}-${options.entity ?? options.workflowName}`;
             } //else this is essentially recursion as the function calls itself
             const payload = {
                 arguments: [...options.args],

package/build/services/signaler/stream.js

@@ -220,13 +220,12 @@ class StreamSignaler {
         for (const instance of [...StreamSignaler.signalers]) {
             instance.stopConsuming();
         }
-        await (0, utils_1.sleepFor)(BLOCK_TIME_MS);
     }
     async stopConsuming() {
         this.shouldConsume = false;
         this.logger.info(`stream-consumer-stopping`, this.topic ? { topic: this.topic } : undefined);
         this.cancelThrottle();
-        await (0, utils_1.sleepFor)(BLOCK_TIME_MS);
+        //await sleepFor(BLOCK_TIME_MS);
     }
     cancelThrottle() {
         if (this.currentTimerId !== undefined) {

package/build/types/durable.d.ts

@@ -46,10 +46,10 @@ type WorkflowSearchOptions = {
 };
 type WorkflowOptions = {
     namespace?: string;
-    taskQueue: string;
+    taskQueue?: string;
     args: any[];
     workflowId?: string;
-    prefix?: string;
+    entity?: string;
     workflowName?: string;
     parentWorkflowId?: string;
     workflowTrace?: string;
@@ -61,6 +61,7 @@ type HookOptions = {
     namespace?: string;
     taskQueue?: string;
     args: any[];
+    entity?: string;
     workflowId?: string;
     workflowName?: string;
     search?: WorkflowSearchOptions;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hotmeshio/hotmesh",
-  "version": "0.0.30",
+  "version": "0.0.32",
   "description": "Unbreakable Workflows",
   "main": "build/index.js",
   "types": "build/index.d.ts",

package/services/durable/client.ts

@@ -92,8 +92,8 @@ export class ClientService {
 
   workflow = {
     start: async (options: WorkflowOptions): Promise<WorkflowHandleService> => {
-      const taskQueueName = options.taskQueue;
-      const workflowName = options.workflowName;
+      const taskQueueName = options.entity ?? options.taskQueue;
+      const workflowName = options.entity ?? options.workflowName;
       const trc = options.workflowTrace;
       const spn = options.workflowSpan;
       //topic is concat of taskQueue and workflowName

package/services/durable/workflow.ts

@@ -39,11 +39,11 @@ export class WorkflowService {
     const workflowSpan = store.get('workflowSpan');
     const COUNTER = store.get('counter');
     const execIndex = COUNTER.counter = COUNTER.counter + 1;
-    const prefix = options.prefix ?? '';
-    //this is risky but MUST be allowed. Users MAY set the workflowId,
-    //but if there is a naming collision, the data from the target entity will be used
-    //as there is know way of knowing if the item was generated via a prior run of the workflow
-    const childJobId = options.workflowId ?? `${prefix}-${workflowId}-$${options.workflowName}${workflowDimension}-${execIndex}`;
+    //NOTE: this is the hash prefix; necessary for the search index to locate the entity
+    //if the hash is a helper, a dash begins it, so it isn't indexed
+    const entityOrEmptyString = options.entity ?? '';
+    //If the workflowId is not provided, it is generated from the entity and the workflow name
+    const childJobId = options.workflowId ?? `${entityOrEmptyString}-${workflowId}-$${options.entity ?? options.workflowName}${workflowDimension}-${execIndex}`;
     const parentWorkflowId = `${workflowId}-f`;
 
     const client = new Client({
@@ -51,8 +51,8 @@ export class WorkflowService {
     });
 
     let handle = await client.workflow.getHandle(
-      options.taskQueue,
-      options.workflowName,
+      options.entity ?? options.taskQueue,
+      options.entity ?? options.workflowName,
       childJobId,
       namespace,
     );
@@ -89,10 +89,12 @@ export class WorkflowService {
     const workflowSpan = store.get('workflowSpan');
     const COUNTER = store.get('counter');
     const execIndex = COUNTER.counter = COUNTER.counter + 1;
-    const prefix = options.prefix ?? '';
-    const childJobId = options.workflowId ?? `${prefix}-${workflowId}-$${options.workflowName}${workflowDimension}-${execIndex}`;
+    //NOTE: this is the hash prefix; necessary for the search index to locate the entity
+    const entityOrEmptyString = options.entity ?? '';
+    //If the workflowId is not provided, it is generated from the entity and the workflow name
+    const childJobId = options.workflowId ?? `${entityOrEmptyString}-${workflowId}-$${options.entity ?? options.workflowName}${workflowDimension}-${execIndex}`;
     const parentWorkflowId = `${workflowId}-f`;
-    const workflowTopic = `${options.taskQueue}-${options.workflowName}`;
+    const workflowTopic = `${options.entity ?? options.taskQueue}-${options.entity ?? options.workflowName}`;
 
     try {
       //get the status; if there is no error, return childJobId (what was spawned)
@@ -273,8 +275,8 @@ export class WorkflowService {
       const store = asyncLocalStorage.getStore();
       const workflowId = options.workflowId ?? store.get('workflowId');
       let workflowTopic = store.get('workflowTopic');
-      if (options.taskQueue && options.workflowName) {
-        workflowTopic = `${options.taskQueue}-${options.workflowName}`;
+      if (options.entity || (options.taskQueue && options.workflowName)) {
+        workflowTopic = `${options.entity ?? options.taskQueue}-${options.entity ?? options.workflowName}`;
       } //else this is essentially recursion as the function calls itself
       const payload = {
         arguments: [...options.args],

package/services/signaler/stream.ts

@@ -254,14 +254,13 @@ class StreamSignaler {
     for (const instance of [...StreamSignaler.signalers]) {
       instance.stopConsuming();
     }
-    await sleepFor(BLOCK_TIME_MS);
   }
   
   async stopConsuming() {
     this.shouldConsume = false;
     this.logger.info(`stream-consumer-stopping`, this.topic ? { topic: this.topic } : undefined);
     this.cancelThrottle();
-    await sleepFor(BLOCK_TIME_MS);
+    //await sleepFor(BLOCK_TIME_MS);
   }
 
   cancelThrottle() {

package/types/durable.ts

@@ -54,11 +54,11 @@ type WorkflowSearchOptions = {
 
 type WorkflowOptions = {
   namespace?: string;         //'durable' is the default namespace if not provided; similar to setting `appid` in the YAML
-  taskQueue: string;
+  taskQueue?: string;         //optional if entity is provided
   args: any[];                //input arguments to pass in
   workflowId?: string;        //execution id (the job id)
-  prefix?: string;            //If invoking a hook, the prefix ensures the FT.SEARCH index is properly scoped to those documents with this prefix
-  workflowName?: string;      //the name of the user's workflow function
+  entity?: string;            //If invoking a workflow, passing 'entity' will apply the value as the workflowName, taskQueue, and prefix, ensuring the FT.SEARCH index is properly scoped. This is a convenience method but limits options.
+  workflowName?: string;      //the name of the user's workflow function; optional if 'entity' is provided
   parentWorkflowId?: string;  //system reserved; the id of the parent; if present the flow will not self-clean until the parent that spawned it self-cleans
   workflowTrace?: string;
   workflowSpan?: string;
@@ -68,8 +68,9 @@ type WorkflowOptions = {
 
 type HookOptions = {
   namespace?: string;   //'durable' is the default namespace if not provided; similar to setting `appid` in the YAML
-  taskQueue?: string;
+  taskQueue?: string;   //optional if 'entity' is provided
   args: any[];          //input arguments to pass into the hook
+  entity?: string;      //If invoking a hook, passing 'entity' will apply the value as the workflowName, taskQueue, and prefix, ensuring the FT.SEARCH index is properly scoped. This is a convenience method but limits options.
   workflowId?: string;   //execution id (the job id to hook into)
   workflowName?: string; //the name of the user's hook function
   search?: WorkflowSearchOptions //bind additional search terms immediately before hook reentry