@hotmeshio/hotmesh 0.0.25 → 0.0.26

package/README.md

@@ -1,11 +1,11 @@
 # HotMesh
 ![alpha release](https://img.shields.io/badge/release-alpha-yellow)
 
-Elevate Redis from an in-memory data cache to a game-changing [service mesh](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/faq.md#what-is-hotmesh). Turn your unpredictable functions into unbreakable workflows.
+Elevate Redis from an in-memory data cache, and turn your unpredictable functions into unbreakable workflows.
 
-HotMesh is a wrapper for Redis that exposes a higher level set of domain constructs like ‘activities’, ‘workflows’, 'jobs', etc. Behind the scenes, it uses *Redis Data* (Hash, ZSet, List); *Redis Streams* (XReadGroup, XAdd, XLen, etc); and *Redis Publish/Subscribe*.
+**HotMesh** is a wrapper for Redis that exposes a higher level set of domain constructs like ‘activities’, ‘workflows’, 'jobs', etc. Behind the scenes, it uses *Redis Data* (Hash, ZSet, List); *Redis Streams* (XReadGroup, XAdd, XLen, etc); and *Redis Publish/Subscribe*.
 
-It's still Redis in the background, but the information flow is fundamentally different. Your functions no longer call Redis (e.g., to cache a document) and instead are called by Redis.
+It's still Redis in the background, but the information flow is reversed. Instead of your functions calling Redis (e.g., for caching a document), Redis governs your function execution. If your microservice container goes down or your function simply fails, HotMesh will restore function state at the point of failure and retry until it succeeds.
 
 ## Install
 [![npm version](https://badge.fury.io/js/%40hotmeshio%2Fhotmesh.svg)](https://badge.fury.io/js/%40hotmeshio%2Fhotmesh)
@@ -79,16 +79,30 @@ The HotMesh SDK is designed to keep your code front-and-center. Write code as yo
       }
     }
     ```
-3. Invoke your class, providing a unique id (it's now an idempotent workflow and needs a GUID). Nothing changes from the outside, *but Redis now governs the end-to-end execution.* It's guaranteed to succeed and return regardless of  network or microservice failure. Redis will simply inflate like a balloon and deflate as your services come back online.
+3. Invoke your class, providing a unique id (it's now an idempotent workflow and needs a GUID). Nothing changes from the outside, *but Redis now governs the end-to-end execution.* It's guaranteed to succeed, even if it takes a while. 
     ```javascript
     //mycaller.ts
 
     const workflow = new MyWorkflow({ id: 'my123', await: true });
-    const response = await workflow.run('John');
-    //Hello, John! Hi, John!
+    const response = await workflow.run('World');
+    //Hi, World! Hello, World!
     ```
 
->NOTE: MeshOS provides a full suite of durable workflow methods for extending your functions, including: `waitForSignal`, `signal`, `hook`, `sleep` (months, years, etc), `executeChild`, `startChild`, `get`, `set`, `incr` (increment), and `mult` (multiply).
+Redis governance delivers more than just reliability. Externalizing state fundamentally changes the execution profile for your functions, allowing you to design long-running, durable workflows. Use the following methods to solve the most common state management challenges.
+
+ - `waitForSignal` | Pause and wait for external event(s) before continuing. The *waitForSignal* method will collate and cache the signals and only awaken your function once they've all arrived.
+ - `signal` | Send a signal (and optional payload) to any paused function.
+ - `hook` | Redis governance supercharges your functions, transforming them into 're-entrant processes'. Optionally use the *hook* method to spawn parallel execution threads within any running function.
+ - `sleep` | Pause function execution for a ridiculous amount of time (months, years, etc). There's no risk of information loss, as Redis governs function state. When your function awakens, function state is efficiently (and automatically) restored.
+ - `random` | Generate a deterministic random number that can be used in a reentrant process workflow (replaces `Math.random()`).
+ - `executeChild` | Call another durable function and await the response. *Design sophisticated, multi-process solutions by leveraging this command.*
+ - `startChild` | Call another durable function, but do not await the response.
+ - `set` | Set a value (e.g, `set('name', 'value')`)
+ - `get` | Get a value (e.g, `get('name')`)
+ - `incr` | Increment (or decrement) a number (e.g, `incr('name', -99)`)
+ - `mult` | Multiply (or divide) a number (e.g, `mult('name', 12)`)
+
+Refer to the [hotmeshio/samples-typescript](https://github.com/hotmeshio/samples-typescript) repo for usage examples. 
 
 ## Advanced Design
 HotMesh's TypeScript SDK is the easiest way to make your functions durable. But if you need full control over your function lifecycles (including high-volume, high-speed use cases), you can use HotMesh's underlying YAML models to optimize your durable workflows. The following model depicts a sequence of activities orchestrated by HotMesh. Any function you associate with a `topic` in your YAML definition is guaranteed to be durable.
@@ -225,4 +239,4 @@ HotMesh is a distributed orchestration engine. Refer to the [Distributed Orchest
 Gain insight into HotMesh's monitoring, exception handling, and alarm configurations via the [System Lifecycle Guide](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/system_lifecycle.md).
 
 ## Alpha Release
-So what exacty is an [alpha release](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/alpha.md)?!
+So what exacty is an [alpha release](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/alpha.md)?

package/build/modules/utils.d.ts

@@ -5,6 +5,7 @@ import { RedisClient, RedisMulti } from "../types/redis";
 import { StringAnyType } from "../types/serializer";
 import { StreamCode, StreamStatus } from "../types/stream";
 export declare function sleepFor(ms: number): Promise<unknown>;
+export declare function deterministicRandom(seed: number): number;
 export declare function identifyRedisType(redisInstance: any): 'redis' | 'ioredis' | null;
 export declare const polyfill: {
     resolveActivityType(activityType: string): string;

package/build/modules/utils.js

@@ -1,10 +1,15 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.restoreHierarchy = exports.getValueByPath = exports.getIndexedHash = exports.getSymVal = exports.getSymKey = exports.formatISODate = exports.getTimeSeries = exports.getSubscriptionTopic = exports.findSubscriptionForTrigger = exports.findTopKey = exports.XSleepFor = exports.matchesStatus = exports.matchesStatusCode = exports.identifyRedisTypeFromClass = exports.polyfill = exports.identifyRedisType = exports.sleepFor = void 0;
+exports.restoreHierarchy = exports.getValueByPath = exports.getIndexedHash = exports.getSymVal = exports.getSymKey = exports.formatISODate = exports.getTimeSeries = exports.getSubscriptionTopic = exports.findSubscriptionForTrigger = exports.findTopKey = exports.XSleepFor = exports.matchesStatus = exports.matchesStatusCode = exports.identifyRedisTypeFromClass = exports.polyfill = exports.identifyRedisType = exports.deterministicRandom = exports.sleepFor = void 0;
 async function sleepFor(ms) {
     return new Promise((resolve) => setTimeout(resolve, ms));
 }
 exports.sleepFor = sleepFor;
+function deterministicRandom(seed) {
+    let x = Math.sin(seed) * 10000;
+    return x - Math.floor(x);
+}
+exports.deterministicRandom = deterministicRandom;
 function identifyRedisType(redisInstance) {
     const prototype = Object.getPrototypeOf(redisInstance);
     if ('defineCommand' in prototype || Object.keys(prototype).includes('multi')) {

package/build/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@hotmeshio/hotmesh",
-    "version": "0.0.25",
+    "version": "0.0.26",
     "description": "Unbreakable Workflows",
     "main": "build/index.js",
     "types": "build/index.d.ts",

package/build/services/durable/meshos.d.ts

@@ -1,6 +1,6 @@
 import { WorkflowHandleService } from './handle';
 import { WorkflowService } from './workflow';
-import { FindOptions, MeshOSActivityOptions, MeshOSConfig, MeshOSOptions, MeshOSWorkerOptions, WorkflowSearchOptions } from '../../types/durable';
+import { FindOptions, FindWhereOptions, FindWhereQuery, MeshOSActivityOptions, MeshOSConfig, MeshOSOptions, MeshOSWorkerOptions, WorkflowSearchOptions } from '../../types/durable';
 import { RedisOptions, RedisClass } from '../../types/redis';
 import { StringAnyType } from '../../types/serializer';
 /**
@@ -97,6 +97,14 @@ export declare class MeshOSService {
      * @returns {string}
      */
     static find(options: FindOptions, ...args: string[]): Promise<string[] | [number]>;
+    /**
+     * Provides a JSON abstraction for the Redis FT.search command
+     * (e.g, `count`, `query`, `return`, `limit`)
+     * @param {FindWhereOptions} options
+     * @returns {Promise<string[] | [number]>}
+     */
+    static findWhere(options: FindWhereOptions): Promise<string[] | [number]>;
+    static generateSearchQuery(query: FindWhereQuery[]): string;
     /**
      * returns the workflow handle. The handle can then be
      * used to query for status, state, custom state, etc.

package/build/services/durable/meshos.js

@@ -151,9 +151,8 @@ class MeshOSService {
                 class: my.redisClass,
                 options: my.redisOptions
             } });
-        //workflow name is the function name driving the workflow
         let workflowName;
-        if (options?.workflowName) {
+        if (options.workflowName) {
             workflowName = options?.workflowName;
         }
         else if (my.workflowFunctions?.length) {
@@ -165,7 +164,66 @@ class MeshOSService {
                 workflowName = target.name;
             }
         }
-        return await client.workflow.search(options?.taskQueue ?? my.taskQueue, workflowName, my.namespace, my.search.index, ...args); //[count, [id, fields[], id, fields[], id, fields[], ...]]
+        return await client.workflow.search(options.taskQueue ?? my.taskQueue, workflowName, options.namespace ?? my.namespace, options.index ?? my.search.index, ...args); //[count, [id, fields[]], [id, fields[]], [id, fields[]], ...]]
+    }
+    /**
+     * Provides a JSON abstraction for the Redis FT.search command
+     * (e.g, `count`, `query`, `return`, `limit`)
+     * @param {FindWhereOptions} options
+     * @returns {Promise<string[] | [number]>}
+     */
+    static async findWhere(options) {
+        const args = [this.generateSearchQuery(options.query)];
+        if (options.count) {
+            args.push('LIMIT', '0', '0');
+        }
+        else {
+            //limit which hash fields to return
+            if (options.return?.length) {
+                args.push('RETURN');
+                args.push(options.return.length.toString());
+                options.return.forEach(returnField => {
+                    args.push(`_${returnField}`);
+                });
+            }
+            //paginate
+            if (options.limit) {
+                args.push('LIMIT', options.limit.start.toString(), options.limit.size.toString());
+            }
+        }
+        return await this.find(options.options ?? {}, ...args);
+    }
+    static generateSearchQuery(query) {
+        const my = new this();
+        let queryString = query.map(q => {
+            const { field, is, value, type } = q;
+            const prefixedFieldName = my.search?.schema && field in my.search.schema ? `@_${field}` : `@${field}`;
+            const fieldType = my.search?.schema[field]?.type ?? type ?? 'TEXT';
+            switch (fieldType) {
+                case 'TAG':
+                    return `${prefixedFieldName}:{${value}}`;
+                case 'TEXT':
+                    return `${prefixedFieldName}:"${value}"`;
+                case 'NUMERIC':
+                    let range = '';
+                    if (is.startsWith('=')) {
+                        range = `[${value} ${value}]`;
+                    }
+                    else if (is === '<') {
+                        range = `[-inf ${value}]`;
+                    }
+                    else if (is === '>') {
+                        range = `[${value} +inf]`;
+                    }
+                    else if (is === '[]') {
+                        range = `[${value[0]} ${value[1]}]`;
+                    }
+                    return `${prefixedFieldName}:${range}`;
+                default:
+                    return '';
+            }
+        }).join(' ');
+        return queryString;
     }
     /**
      * returns the workflow handle. The handle can then be

package/build/services/durable/workflow.d.ts

@@ -28,6 +28,13 @@ export declare class WorkflowService {
      * process state and job state)
      */
     static isSideEffectAllowed(hotMeshClient: HotMesh, prefix: string): Promise<boolean>;
+    /**
+     * returns a random number between 0 and 1. This number is deterministic
+     * and will never vary for a given seed. This is useful for randomizing
+     * pathways in a workflow that can be safely replayed.
+     * @returns {number}
+     */
+    static random(): number;
     /**
      * send signal data into any other paused thread (which is paused and
      * awaiting the signal) from within a hook-thread or the main-thread

package/build/services/durable/workflow.js

@@ -14,6 +14,7 @@ const factory_1 = require("./factory");
 const search_1 = require("./search");
 const worker_1 = require("./worker");
 const stream_1 = require("../../types/stream");
+const utils_1 = require("../../modules/utils");
 class WorkflowService {
     /**
      * Spawn a child workflow. await and return the result.
@@ -151,6 +152,18 @@ class WorkflowService {
         const guidValue = Number(await hotMeshClient.engine.store.exec('HINCRBYFLOAT', workflowGuid, sessionId, '1'));
         return guidValue === 1;
     }
+    /**
+     * returns a random number between 0 and 1. This number is deterministic
+     * and will never vary for a given seed. This is useful for randomizing
+     * pathways in a workflow that can be safely replayed.
+     * @returns {number}
+     */
+    static random() {
+        const store = asyncLocalStorage_1.asyncLocalStorage.getStore();
+        const COUNTER = store.get('counter');
+        const seed = COUNTER.counter = COUNTER.counter + 1;
+        return (0, utils_1.deterministicRandom)(seed);
+    }
     /**
      * send signal data into any other paused thread (which is paused and
      * awaiting the signal) from within a hook-thread or the main-thread

package/build/types/durable.d.ts

@@ -82,12 +82,28 @@ type WorkerConfig = {
     options?: WorkerOptions;
     search?: WorkflowSearchOptions;
 };
+type FindWhereQuery = {
+    field: string;
+    is: string;
+    value: string | boolean | number | [number, number];
+    type?: string;
+};
 type FindOptions = {
     workflowName?: string;
     taskQueue?: string;
     namespace?: string;
     index?: string;
 };
+type FindWhereOptions = {
+    options?: FindOptions;
+    count?: boolean;
+    query: FindWhereQuery[];
+    return?: string[];
+    limit?: {
+        start: number;
+        size: number;
+    };
+};
 type MeshOSOptions = {
     name: string;
     options: WorkerOptions;
@@ -126,4 +142,4 @@ type ActivityConfig = {
         maximumInterval: string;
     };
 };
-export { ActivityConfig, ActivityWorkflowDataType, ClientConfig, ContextType, ConnectionConfig, Connection, ProxyType, Registry, SignalOptions, FindOptions, HookOptions, MeshOSActivityOptions, MeshOSWorkerOptions, MeshOSClassConfig, MeshOSConfig, MeshOSOptions, WorkerConfig, WorkflowConfig, WorkerOptions, WorkflowSearchOptions, WorkflowDataType, WorkflowOptions, };
+export { ActivityConfig, ActivityWorkflowDataType, ClientConfig, ContextType, ConnectionConfig, Connection, ProxyType, Registry, SignalOptions, FindOptions, FindWhereOptions, FindWhereQuery, HookOptions, MeshOSActivityOptions, MeshOSWorkerOptions, MeshOSClassConfig, MeshOSConfig, MeshOSOptions, WorkerConfig, WorkflowConfig, WorkerOptions, WorkflowSearchOptions, WorkflowDataType, WorkflowOptions, };

package/build/types/index.d.ts

@@ -3,7 +3,7 @@ export { App, AppVID, AppTransitions, AppSubscriptions } from './app';
 export { AsyncSignal } from './async';
 export { CacheMode } from './cache';
 export { CollationFaultType, CollationStage } from './collator';
-export { ActivityConfig, ActivityWorkflowDataType, ClientConfig, ContextType, ConnectionConfig, Connection, ProxyType, Registry, SignalOptions, FindOptions, HookOptions, MeshOSActivityOptions, MeshOSWorkerOptions, MeshOSClassConfig, MeshOSConfig, MeshOSOptions, WorkflowConfig, WorkerConfig, WorkerOptions, WorkflowSearchOptions, WorkflowDataType, WorkflowOptions, } from './durable';
+export { ActivityConfig, ActivityWorkflowDataType, ClientConfig, ContextType, ConnectionConfig, Connection, ProxyType, Registry, SignalOptions, FindOptions, FindWhereOptions, FindWhereQuery, HookOptions, MeshOSActivityOptions, MeshOSWorkerOptions, MeshOSClassConfig, MeshOSConfig, MeshOSOptions, WorkflowConfig, WorkerConfig, WorkerOptions, WorkflowSearchOptions, WorkflowDataType, WorkflowOptions, } from './durable';
 export { HookCondition, HookConditions, HookGate, HookInterface, HookRule, HookRules, HookSignal } from './hook';
 export { RedisClientType as IORedisClientType, RedisMultiType as IORedisMultiType } from './ioredisclient';
 export { ILogger } from './logger';

package/modules/utils.ts

@@ -8,6 +8,11 @@ export async function sleepFor(ms: number) {
   return new Promise((resolve) => setTimeout(resolve, ms));
 }
 
+export function deterministicRandom(seed: number): number {
+  let x = Math.sin(seed) * 10000;
+  return x - Math.floor(x);
+}
+
 export function identifyRedisType(redisInstance: any): 'redis' | 'ioredis' | null {
   const prototype = Object.getPrototypeOf(redisInstance);
   if ('defineCommand' in prototype || Object.keys(prototype).includes('multi')) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hotmeshio/hotmesh",
-  "version": "0.0.25",
+  "version": "0.0.26",
   "description": "Unbreakable Workflows",
   "main": "build/index.js",
   "types": "build/index.d.ts",

package/services/durable/meshos.ts

@@ -10,6 +10,8 @@ import { WorkflowService } from './workflow';
 import { StreamSignaler } from '../signaler/stream';
 import {
   FindOptions,
+  FindWhereOptions,
+  FindWhereQuery,
   MeshOSActivityOptions,
   MeshOSConfig,
   MeshOSOptions,
@@ -245,9 +247,9 @@ export class MeshOSService {
       class: my.redisClass,
       options: my.redisOptions 
     }});
-    //workflow name is the function name driving the workflow
+
     let workflowName: string;
-    if (options?.workflowName) {
+    if (options.workflowName) {
       workflowName = options?.workflowName
     } else if(my.workflowFunctions?.length) {
       let target = my.workflowFunctions[0];
@@ -258,12 +260,70 @@ export class MeshOSService {
       }
     }
     return await client.workflow.search(
-      options?.taskQueue ?? my.taskQueue,
+      options.taskQueue ?? my.taskQueue,
       workflowName,
-      my.namespace,
-      my.search.index,
+      options.namespace ?? my.namespace,
+      options.index ?? my.search.index,
       ...args,
-    ); //[count, [id, fields[], id, fields[], id, fields[], ...]]
+    ); //[count, [id, fields[]], [id, fields[]], [id, fields[]], ...]]
+  }
+
+  /**
+   * Provides a JSON abstraction for the Redis FT.search command
+   * (e.g, `count`, `query`, `return`, `limit`)
+   * @param {FindWhereOptions} options 
+   * @returns {Promise<string[] | [number]>}
+   */
+  static async findWhere(options: FindWhereOptions): Promise<string[] | [number]> {
+    const args: string[] = [this.generateSearchQuery(options.query)];
+    if (options.count) {
+      args.push('LIMIT', '0', '0');
+    } else {
+      //limit which hash fields to return
+      if (options.return?.length) {
+        args.push('RETURN');
+        args.push(options.return.length.toString());
+        options.return.forEach(returnField => {
+          args.push(`_${returnField}`);
+        });
+      }
+      //paginate
+      if (options.limit) {
+        args.push('LIMIT', options.limit.start.toString(), options.limit.size.toString());
+      }
+    } 
+    return await this.find(options.options ?? {}, ...args);
+  }
+
+  static generateSearchQuery(query: FindWhereQuery[]) {
+    const my = new this();
+    let queryString = query.map(q => {
+      const { field, is, value, type } = q;
+      const prefixedFieldName = my.search?.schema && field in my.search.schema ? `@_${field}` : `@${field}`;
+      const fieldType = my.search?.schema[field]?.type ?? type ?? 'TEXT';
+
+      switch (fieldType) {
+        case 'TAG':
+          return `${prefixedFieldName}:{${value}}`;
+        case 'TEXT':
+          return `${prefixedFieldName}:"${value}"`;
+        case 'NUMERIC':
+          let range = '';
+          if (is.startsWith('=')) {
+            range = `[${value} ${value}]`;
+          } else if (is === '<') {
+            range = `[-inf ${value}]`;
+          } else if (is === '>') {
+            range = `[${value} +inf]`;
+          } else if (is === '[]') {
+            range = `[${value[0]} ${value[1]}]`
+          }
+          return `${prefixedFieldName}:${range}`;
+        default:
+          return '';
+      }
+    }).join(' ');
+    return queryString;
   }
 
   /**

package/services/durable/workflow.ts

@@ -19,6 +19,7 @@ import {
   WorkflowOptions } from "../../types/durable";
 import { JobOutput, JobState } from '../../types/job';
 import { StreamStatus } from '../../types/stream';
+import { deterministicRandom } from '../../modules/utils';
 
 export class WorkflowService {
 
@@ -173,6 +174,19 @@ export class WorkflowService {
     return guidValue === 1;
   }
 
+  /**
+   * returns a random number between 0 and 1. This number is deterministic
+   * and will never vary for a given seed. This is useful for randomizing
+   * pathways in a workflow that can be safely replayed.
+   * @returns {number}
+   */
+  static random(): number {
+    const store = asyncLocalStorage.getStore();
+    const COUNTER = store.get('counter');
+    const seed = COUNTER.counter = COUNTER.counter + 1;
+    return deterministicRandom(seed);
+  }
+
   /**
    * send signal data into any other paused thread (which is paused and
    * awaiting the signal) from within a hook-thread or the main-thread

package/types/durable.ts

@@ -93,6 +93,13 @@ type WorkerConfig = {
   search?: WorkflowSearchOptions;
 }
 
+type FindWhereQuery = {
+  field: string;
+  is: string;
+  value: string | boolean | number | [number, number];
+  type?: string; //default is TEXT
+}
+
 type FindOptions = {
   workflowName?: string; //also the function name
   taskQueue?: string;
@@ -100,6 +107,17 @@ type FindOptions = {
   index?: string;        //the FT search index name
 }
 
+type FindWhereOptions = {
+  options?: FindOptions;
+  count?: boolean;
+  query: FindWhereQuery[];
+  return?: string[];
+  limit?: {
+    start: number,
+    size: number
+  }
+}
+
 type MeshOSOptions = {
   name: string;
   options: WorkerOptions;
@@ -156,6 +174,8 @@ export {
   Registry,
   SignalOptions,
   FindOptions,
+  FindWhereOptions,
+  FindWhereQuery,
   HookOptions,
   MeshOSActivityOptions,
   MeshOSWorkerOptions,

package/types/index.ts

@@ -38,6 +38,8 @@ export {
     Registry,
     SignalOptions,
     FindOptions,
+    FindWhereOptions,
+    FindWhereQuery,
     HookOptions,
     MeshOSActivityOptions,
     MeshOSWorkerOptions,