@hotmeshio/hotmesh 0.0.23 → 0.0.24

package/README.md

@@ -3,6 +3,13 @@
 
 Elevate Redis from an in-memory data cache to a game-changing [service mesh](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/faq.md#what-is-hotmesh). Turn your unpredictable functions into unbreakable workflows.
 
+## What is HotMesh?
+HotMesh is a wrapper for Redis that exposes a higher level set of domain constructs like ‘activities’, ‘workflows’, 'jobs', etc. Behind the scenes, it uses *Redis Data* (Hash, ZSet, and List); *Redis Streams* (XReadGroup, XAdd, XLen); and *Redis Publish/Subscribe*.
+
+The ultimate goal is to resurface Redis as a *Durable Service Mesh*, capable of running unbreakable workflows that span your microservices. The technical term for this type of durability is *Reentrant Process Engine*.
+
+It's still Redis in the background, but the information flow is fundamentally different. Your functions no longer call Redis (e.g., to cache a document) and instead are called by Redis.
+
 ## Install
 [![npm version](https://badge.fury.io/js/%40hotmeshio%2Fhotmesh.svg)](https://badge.fury.io/js/%40hotmeshio%2Fhotmesh)
 
@@ -11,92 +18,84 @@ npm install @hotmeshio/hotmesh
 ```
 
 ## Design
-The HotMesh SDK is designed to keep your code front-and-center. Write functions as you normally would, then use HotMesh to make them durable.
+The HotMesh SDK is designed to keep your code front-and-center. Write code as you normally would, then use HotMesh to make it durable.
 
-1. Start by defining **activities**. Activities can be written in any style, using any framework, and can even be legacy functions you've already written. The only requirement is that they return a Promise. *Note how the `saludar` example throws an error 50% of the time. It doesn't matter how unpredictable your functions are, HotMesh will retry as necessary until they succeed.*
+1. Start with any ordinary class. Pay attention to unpredictable functions: those that execute slowly, cause problems at scale, or simply fail to return. *Note how the `saludar` function throws an error 50% of the time. This is exactly the type of function to fix.*
     ```javascript
-    //activities.ts
+    //myworkflow.ts
 
-    export async function greet(name: string): Promise<string> {
-      return `Hello, ${name}!`;
-    }
+    export class MyWorkflow {
 
-    export async function saludar(nombre: string): Promise<string> {
-      if (Math.random() > 0.5) throw new Error('Random error');
-      return `¡Hola, ${nombre}!`;
-    }
-    ```
-2. Define your **workflow** logic. Include conditional branching, loops, etc to control activity execution. It's vanilla code written in your own coding style. The only requirement is to use `proxyActivities`, ensuring your activities are executed with HotMesh's durability guarantee.
-    ```javascript
-    //workflows.ts
-
-    import { Durable } from '@hotmeshio/hotmesh';
-    import * as activities from './activities';
+      //main method
+      async run(name: string): Promise<string> {
+        const salud = await this.saludar(name);
+        const hello = await this.greet(name);
+        return `${hello} ${salud}`;
+      }
 
-    const { greet, saludar } = Durable.workflow
-      .proxyActivities<typeof activities>({
-        activities
-      });
+      //this function only succeeds 50% of the time!
+      async saludar(nombre: string): Promise<string> {
+        if (Math.random() < 0.5) {
+          throw new Error('¡No hablo español!');
+        }
+        return `¡Hola, ${nombre}!`;
+      }
 
-    export async function example(name: string, lang: string): Promise<string> {
-      if (lang === 'es') {
-        return await saludar(name);
-      } else {
-        return await greet(name);
+      //this function always succeeds
+      async greet(name: string): Promise<string> {
+        return `Hello, ${name}!`;
       }
     }
     ```
-
-3. Although you could call your workflow directly (it's just a vanilla function), it's only durable when invoked and orchestrated via HotMesh.
+2. Import `MeshOS` and subclass it as shown. Configure `host`, `port`, etc., and list those functions that should run durably like `run` and `saludar`. Your class is now a durable workflow! *Additional helper methods include `waitForSignal`, `signal`, `hook`, `sleep` (months, years, etc), `executeChild`, `startChild`, `get`, `set`, `incr` (increment), and `mult` (multiply).*
     ```javascript
-    //client.ts
+    //myworkflow.ts
 
-    import { Durable } from '@hotmeshio/hotmesh';
     import Redis from 'ioredis'; //OR `import * as Redis from 'redis';`
-    import { nanoid } from 'nanoid';
+    import { MeshOS } from '@hotmeshio/hotmesh';
 
-    async function run(): Promise<string> {
-      const client = new Durable.Client({
-        connection: {
-          class: Redis,
-          options: { host: 'localhost', port: 6379 }
-        }
-      });
+    export class MyWorkflow extends MeshOS {
 
-      const handle = await client.workflow.start({
-        args: ['HotMesh', 'es'],
-        taskQueue: 'hello-world',
-        workflowName: 'example',
-        workflowId: nanoid()
-      });
+      redisClass = Redis;
+      redisOptions = { host: 'localhost', port: 6379 };
 
-      return await handle.result();
-      //returns '¡Hola, HotMesh!'
-    }
-    ```
+      //list functions to run as durable workflows
+      workflowFunctions = ['run'];
 
-4. Finally, create a **worker** and link your workflow function. Workers listen for tasks on their assigned Redis stream and invoke your workflow function each time they receive an event.
-    ```javascript
-    //worker.ts
-
-    import { Durable } from '@hotmeshio/hotmesh';
-    import Redis from 'ioredis';
-    import * as workflows from './workflows';
-
-    async function run() {
-      const worker = await Durable.Worker.create({
-        connection: {
-          class: Redis,
-          options: { host: 'localhost', port: 6379 },
-        },
-        taskQueue: 'hello-world',
-        workflow: workflows.example,
-      });
-      await worker.run();
+      //list functions to retry and cache
+      proxyFunctions = ['saludar'];
+
+      //main method (Redis will now govern its execution)
+      async run(name: string): Promise<string> {
+        const salud = await this.saludar(name);
+        const hello = await this.greet(name);
+        return `${hello} ${salud}`;
+      }
+
+      //this function will now be retried until it succeeds;
+      async saludar(nombre: string): Promise<string> {
+        if (Math.random() < 0.5) {
+          throw new Error('¡No hablo español!');
+        }
+        return `¡Hola, ${nombre}!`;
+      }
+
+      //this vanilla function is unchanged
+      async greet(name: string): Promise<string> {
+        return `Hello, ${name}!`;
+      }
     }
     ```
+3. Invoke your class, providing a unique GUID (it's now a workflow and it's idempotent). Nothing changes from the outside, *but Redis now governs its end-to-end execution.* It's guaranteed to succeed and return regardless of catastrophic network failure. Redis will simply inflate like a balloon and deflate as things come back online.
+    ```javascript
+    //mycaller.ts
+
+    //...import MyWorkflow, etc...
 
->HotMesh delivers durable workflows without the cost and complexity of a centralized service mesh. Refer to the [samples-typescript](https://github.com/hotmeshio/samples-typescript) Git Repo for a range of examples, including compositional workflows (where one workflow calls another) and remote execution (where calls are brokered across microservices).
+    const workflow = new MyWorkflow('unique123', { await: true }); //await the response
+    const response = await workflow.run('John');
+    //Hello, John! ¡Hola, John!
+    ```
 
 ## Advanced Design
 HotMesh's TypeScript SDK is the easiest way to make your functions durable. But if you need full control over your function lifecycles (including high-volume, high-speed use cases), you can use HotMesh's underlying YAML models to optimize your durable workflows. The following model depicts a sequence of activities orchestrated by HotMesh. Any function you associate with a `topic` in your YAML definition is guaranteed to be durable.

package/build/index.d.ts

@@ -1,4 +1,5 @@
 import { Durable } from './services/durable';
+import { MeshOSService as MeshOS } from './services/durable/meshos';
 import { HotMeshService as HotMesh } from './services/hotmesh';
 import { HotMeshConfig } from './types/hotmesh';
-export { Durable, HotMesh, HotMeshConfig };
+export { Durable, HotMesh, HotMeshConfig, MeshOS };

package/build/index.js

@@ -1,7 +1,9 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.HotMesh = exports.Durable = void 0;
+exports.MeshOS = exports.HotMesh = exports.Durable = void 0;
 const durable_1 = require("./services/durable");
 Object.defineProperty(exports, "Durable", { enumerable: true, get: function () { return durable_1.Durable; } });
+const meshos_1 = require("./services/durable/meshos");
+Object.defineProperty(exports, "MeshOS", { enumerable: true, get: function () { return meshos_1.MeshOSService; } });
 const hotmesh_1 = require("./services/hotmesh");
 Object.defineProperty(exports, "HotMesh", { enumerable: true, get: function () { return hotmesh_1.HotMeshService; } });

package/build/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@hotmeshio/hotmesh",
-    "version": "0.0.23",
+    "version": "0.0.24",
     "description": "Unbreakable Workflows",
     "main": "build/index.js",
     "types": "build/index.d.ts",
@@ -43,7 +43,7 @@
         "test:sub:redis": "NODE_ENV=test jest ./tests/functional/sub/clients/redis.test.ts --detectOpenHandles --forceExit --verbose",
         "test:sub:ioredis": "NODE_ENV=test jest ./tests/functional/sub/clients/ioredis.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable": "NODE_ENV=test jest ./tests/durable/*/index.test.ts --detectOpenHandles --forceExit --verbose",
-        "test:durable:meshdb": "NODE_ENV=test jest ./tests/durable/meshdb/index.test.ts --detectOpenHandles --forceExit --verbose",
+        "test:durable:meshos": "NODE_ENV=test jest ./tests/durable/meshos/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:hello": "NODE_ENV=test jest ./tests/durable/helloworld/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:goodbye": "NODE_ENV=test jest ./tests/durable/goodbye/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:hook": "NODE_ENV=test jest ./tests/durable/hook/index.test.ts --detectOpenHandles --forceExit --verbose",

package/build/services/durable/factory.js

@@ -267,11 +267,11 @@ const getWorkflowYAML = (app, version) => {
                   - ['{@string.concat}']
               cycleWorkflowId:
                 '@pipe':
-                  - ['{$job.metadata.jid}', '-$wfc', '{sig.output.metadata.dad}', '-', '{sigw1.output.data.index}']
+                  - ['-', '{$job.metadata.jid}', '-$wfc', '{sig.output.metadata.dad}', '-', '{sigw1.output.data.index}']
                   - ['{@string.concat}']
               baseWorkflowId:
                 '@pipe':
-                  - ['{$job.metadata.jid}', '-$wfs', '{sig.output.metadata.dad}', '-']
+                  - ['-', '{$job.metadata.jid}', '-$wfs', '{sig.output.metadata.dad}', '-']
                   - ['{@string.concat}']
           output:
             schema:
@@ -315,7 +315,7 @@ const getWorkflowYAML = (app, version) => {
                   - ['{@string.concat}']
               workflowId:
                 '@pipe':
-                  - ['{$job.metadata.jid}', '-$sleep', '{sig.output.metadata.dad}', '-', '{sigw1.output.data.index}']
+                  - ['-', '{$job.metadata.jid}', '-$sleep', '{sig.output.metadata.dad}', '-', '{sigw1.output.data.index}']
                   - ['{@string.concat}']
           output:
             schema:
@@ -385,11 +385,11 @@ const getWorkflowYAML = (app, version) => {
                   - ['{@string.concat}']
               cycleWorkflowId:
                 '@pipe':
-                  - ['{$job.metadata.jid}', '-$wfc-', '{w1.output.data.index}']
+                  - ['-', '{$job.metadata.jid}', '-$wfc-', '{w1.output.data.index}']
                   - ['{@string.concat}']
               baseWorkflowId:
                 '@pipe':
-                  - ['{$job.metadata.jid}', '-$wfs-']
+                  - ['-', '{$job.metadata.jid}', '-$wfs-']
                   - ['{@string.concat}']
           output:
             schema:
@@ -433,7 +433,7 @@ const getWorkflowYAML = (app, version) => {
                   - ['{@string.concat}']
               workflowId:
                 '@pipe':
-                  - ['{$job.metadata.jid}', '-$sleep-', '{w1.output.data.index}']
+                  - ['-', '{$job.metadata.jid}', '-$sleep-', '{w1.output.data.index}']
                   - ['{@string.concat}']
           output:
             schema:

package/build/services/durable/handle.js

@@ -30,10 +30,8 @@ class WorkflowHandleService {
                 throw new Error(JSON.parse(state.metadata.err));
             }
             if (state?.data?.done) {
-                //child flows are never technically 'done' as they have an open hook
-                //that is tied to the parent flow's completion. so, we need to check
-                //the 'done' flag on the child flow's payload (not the 'js' metadata field
-                //which is typically used); the `loadState` parameter triggers this
+                //child flows are never 'done'; they use a hook
+                //that only closes upon parent flow completion.
                 return state.data.response;
             }
         }

package/build/services/durable/index.d.ts

@@ -1,13 +1,13 @@
 import { ClientService } from './client';
 import { ConnectionService } from './connection';
-import { MeshDBService } from './meshdb';
+import { MeshOSService } from './meshos';
 import { WorkerService } from './worker';
 import { WorkflowService } from './workflow';
 import { ContextType } from '../../types/durable';
 export declare const Durable: {
     Client: typeof ClientService;
     Connection: typeof ConnectionService;
-    MeshDB: typeof MeshDBService;
+    MeshOS: typeof MeshOSService;
     Worker: typeof WorkerService;
     workflow: typeof WorkflowService;
 };

package/build/services/durable/index.js

@@ -3,13 +3,13 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.Durable = void 0;
 const client_1 = require("./client");
 const connection_1 = require("./connection");
-const meshdb_1 = require("./meshdb");
+const meshos_1 = require("./meshos");
 const worker_1 = require("./worker");
 const workflow_1 = require("./workflow");
 exports.Durable = {
     Client: client_1.ClientService,
     Connection: connection_1.ConnectionService,
-    MeshDB: meshdb_1.MeshDBService,
+    MeshOS: meshos_1.MeshOSService,
     Worker: worker_1.WorkerService,
     workflow: workflow_1.WorkflowService,
 };

package/build/services/durable/meshos.d.ts

@@ -0,0 +1,108 @@
+import { WorkflowHandleService } from './handle';
+import { FindOptions, MeshOSActivityOptions, MeshOSOptions, WorkflowSearchOptions } from '../../types/durable';
+import { RedisOptions, RedisClass } from '../../types/redis';
+import { StringAnyType } from '../../types';
+import { WorkflowService } from './workflow';
+/**
+ * The base class for running MeshOS workflows.
+ * Extend and register subclass methods by name to
+ * execute as durable workflows, backed by Redis.
+ */
+export declare class MeshOSService {
+    /**
+     * The top-level Redis isolation. All workflow data is
+     * isolated within this namespace. Values should be
+     * lower-case with no spaces (e.g, 'staging', 'prod', 'test',
+     * 'routing-stagig', 'reporting-prod', etc.).
+     *  1) only url-safe values are allowed;
+     *  2) the 'a' symbol is reserved by HotMesh for indexing apps
+     */
+    namespace: string;
+    /**
+     * Data is routed to workers that specify this task queue.
+     * Setting the task queue when the worker is created will
+     * ensure that the worker only receives messages destined
+     * for the queue. Callers can specify the taskQue to when
+     * starting a job to call those workers.
+     */
+    taskQueue: string;
+    /**
+     * These methods run as durable workflows
+     */
+    workflowFunctions: Array<MeshOSOptions | string>;
+    /**
+     * These methods run as hooks (hook into a running workflow)
+     */
+    hookFunctions: Array<MeshOSOptions | string>;
+    /**
+     * These methods run as proxied activities (and are safely memoized)
+     */
+    proxyFunctions: Array<MeshOSActivityOptions | string>;
+    /**
+     * The workflow GUID. Workflows will be persisted to
+     * Redis using the pattern hmsh:<namespace>:j:<id>.
+     */
+    id: string;
+    /**
+     * The Redis connection options. NOTE: Redis and IORedis
+     * use different formats for their connection config.
+     */
+    redisOptions: RedisOptions;
+    /**
+     * The Redis connection class.
+     *
+     * @example
+     * import Redis from 'ioredis';
+     * import * as Redis from 'redis';
+     */
+    redisClass: RedisClass;
+    /**
+     * Optional model declaration (custom workflow state)
+     */
+    model: StringAnyType;
+    /**
+     * Optional configuration for Redis FT search
+     */
+    search: WorkflowSearchOptions;
+    static MeshOS: typeof WorkflowService;
+    static getHotMeshClient(redisClass: RedisClass, redisOptions: RedisOptions, namespace: string, taskQueue: string): Promise<import("../hotmesh").HotMeshService>;
+    /**
+     * mints a workflow ID, using the search prefix.
+     * NOTE: The prefix is necesary when indexing
+     *       HASHes when FT search is enabled.
+     * @returns {string}
+     */
+    static mintGuid(): string;
+    /**
+     * Creates an FT search index
+     */
+    static createIndex(): Promise<void>;
+    /**
+     * Initialize the worker(s) for the entity. This is a static
+     * method that allows for optional task Queue targeting.
+     * NOTE: Allow List may be optionally used to only wrap
+     *       specific methods in this class.
+     * @param {string} taskQueue
+     * @param {string[]} allowList
+     */
+    static startWorkers(taskQueue?: string, allowList?: Array<MeshOSOptions | string>): Promise<void>;
+    /**
+     * executes the redis FT search query
+     * @example '@_quantity:[89 89]'
+     * @param {any[]} args
+     * @returns {string}
+     */
+    static find(options: FindOptions, ...args: string[]): Promise<string[] | [number]>;
+    /**
+     * returns the workflow handle. The handle can then be
+     * used to query for status, state, custom state, etc.
+     * @param {string} id
+     * @returns {Promise<WorkflowHandleService>}
+     */
+    static get(id: string): Promise<WorkflowHandleService>;
+    /**
+     * Optionally include a target taskQueue to exec the
+     * workflow's call on a specific worker queue.
+     */
+    constructor(id?: string, options?: Record<string, any>);
+}