@hotmeshio/hotmesh 0.1.2 → 0.1.4

package/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,38 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+**Smartphone (please complete the following information):**
+ - Device: [e.g. iPhone6]
+ - OS: [e.g. iOS8.1]
+ - Browser [e.g. stock browser, safari]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.

package/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

package/README.md

@@ -3,7 +3,7 @@
 
 HotMesh transforms Redis into an Orchestration Engine.
 
-*Write functions in your own preferred style, and let Redis govern their execution, reliably and durably.*
+*Write functions in your own style, and let Redis govern their execution, reliably and durably.*
 
 ## Install
 [![npm version](https://badge.fury.io/js/%40hotmeshio%2Fhotmesh.svg)](https://badge.fury.io/js/%40hotmeshio%2Fhotmesh)
@@ -13,7 +13,7 @@ npm install @hotmeshio/hotmesh
 ```
 
 ## Understanding HotMesh
-HotMesh inverts the relationship to Redis: those functions that once used Redis as a cache, are instead *cached and governed* by Redis. Consider the following. It's a typical microservices network, with a tangled mess of services and functions. There's important business logic in there (functions *A*, *B* and *C* are critical!), but they're hard to find and access.
+HotMesh inverts the relationship to Redis: those functions that once used Redis as a cache, are instead *cached and governed* by Redis. Consider the following. It's a typical microservices network, with a tangled mess of services and functions. There's important business logic in there (functions *A*, *B* and *C* are critical), but it's hard to find and access.
 
 <img src="./docs/img/operational_data_layer.png" alt="A Tangled Microservices Network with 3 valuable functions buried within" style="max-width:100%;width:600px;">
 
@@ -185,7 +185,7 @@ Externalizing state fundamentally changes the execution profile for your functio
       const value = await search.mult('name', 12);
       ```
 
-Refer to the [hotmeshio/samples-typescript](https://github.com/hotmeshio/samples-typescript) repo for usage examples. 
+Refer to the [hotmeshio/samples-javascript](https://github.com/hotmeshio/samples-javascript) repo for usage examples.
 
 ### Design | Advanced
 The *Pluck* and *Durable* modules are the easiest way to use HotMesh. But if you need full control over your function lifecycles (including high-volume, high-speed use cases), you can use HotMesh's underlying YAML models to optimize your durable workflows. The following model depicts a sequence of activities orchestrated by HotMesh. Any function you associate with a `topic` in your YAML definition is guaranteed to be durable.
@@ -295,28 +295,31 @@ const hotMesh = await HotMesh.init({
 ```
 
 ### Observability
-Workflows and activities are run according to the rules you define, offering [Graph-Oriented](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/system_lifecycle.md#telemetry) telemetry insights into your legacy function executions.
+Workflows and activities are run according to the rules you define, offering [Graph-Oriented](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/system_lifecycle.md#telemetry) telemetry insights into your legacy function executions.
 
 ## FAQ
-Refer to the [FAQ](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/faq.md) for terminology, definitions, and an exploration of how HotMesh facilitates orchestration use cases.
+Refer to the [FAQ](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/faq.md) for terminology, definitions, and an exploration of how HotMesh facilitates orchestration use cases.
 
 ## Quick Start
-Refer to the [Quick Start](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/quickstart.md) for sample YAML workflows you can copy, paste, and modify to get started.
+Refer to the [Quick Start](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/quickstart.md) for sample YAML workflows you can copy, paste, and modify to get started.
 
 ## Developer Guide
-For more details on the complete development process, including information about schemas, APIs, and deployment, consult the [Developer Guide](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/developer_guide.md).
+For more details on the complete development process, including information about schemas, APIs, and deployment, consult the [Developer Guide](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/developer_guide.md).
 
 ## Model Driven Development
-[Model Driven Development](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/model_driven_development.md) is an established strategy for managing process-oriented tasks. Check out this guide to understand its foundational principles.
+[Model Driven Development](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/model_driven_development.md) is an established strategy for managing process-oriented tasks. Check out this guide to understand its foundational principles.
 
 ## Data Mapping
-Exchanging data between activities is central to HotMesh. For detailed information on supported functions and the functional mapping syntax (@pipes), see the [Data Mapping Overview](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/data_mapping.md).
+Exchanging data between activities is central to HotMesh. For detailed information on supported functions and the functional mapping syntax (@pipes), see the [Data Mapping Overview](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/data_mapping.md).
 
 ## Composition
-While the simplest graphs are linear, detailing a consistent sequence of non-cyclical activities, graphs can be layered to represent intricate business scenarios. Some can even be designed to accommodate long-lasting workflows that span months. For more details, check out the [Composable Workflow Guide](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/composable_workflow.md).
-
-## Distributed Orchestration
-HotMesh is a distributed orchestration engine. Refer to the [Distributed Orchestration Guide](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/distributed_orchestration.md) for a detailed breakdown of the approach.
+While the simplest graphs are linear, detailing a consistent sequence of non-cyclical activities, graphs can be layered to represent intricate business scenarios. Some can even be designed to accommodate long-lasting workflows that span months. For more details, check out the [Composable Workflow Guide](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/composable_workflow.md).
 
 ## System Lifecycle
-Gain insight into HotMesh's monitoring, exception handling, and alarm configurations via the [System Lifecycle Guide](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/system_lifecycle.md).
+Gain insight into HotMesh's monitoring, exception handling, and alarm configurations via the [System Lifecycle Guide](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/system_lifecycle.md).
+
+## Distributed Orchestration | System Overview
+HotMesh is a distributed orchestration engine. Refer to the [Distributed Orchestration Guide](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/distributed_orchestration.md) for a high-level overview of the approach.
+
+## Distributed Orchestration | System Design
+HotMesh is more than Redis and TypeScript. The theory that underlies the architecture is applicable to any number of data storage and streaming backends: [A Message-Oriented Approach to Decentralized Process Orchestration](https://zenodo.org/records/12168558).

package/build/modules/enums.d.ts

@@ -23,6 +23,7 @@ export declare const HMSH_QUORUM_DELAY_MS = 250;
 export declare const HMSH_ACTIVATION_MAX_RETRY = 3;
 export declare const HMSH_OTT_WAIT_TIME: number;
 export declare const HMSH_EXPIRE_JOB_SECONDS: number;
+export declare const MAX_DELAY = 2147483647;
 export declare const HMSH_MAX_RETRIES: number;
 export declare const HMSH_MAX_TIMEOUT_MS: number;
 export declare const HMSH_GRADUATED_INTERVAL_MS: number;

package/build/modules/enums.js

@@ -1,6 +1,6 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.HMSH_GUID_SIZE = exports.HMSH_SCOUT_INTERVAL_SECONDS = exports.HMSH_FIDELITY_SECONDS = exports.HMSH_EXPIRE_DURATION = exports.HMSH_XPENDING_COUNT = exports.HMSH_XCLAIM_COUNT = exports.HMSH_XCLAIM_DELAY_MS = exports.HMSH_BLOCK_TIME_MS = exports.HMSH_DURABLE_EXP_BACKOFF = exports.HMSH_DURABLE_MAX_INTERVAL = exports.HMSH_DURABLE_MAX_ATTEMPTS = exports.HMSH_GRADUATED_INTERVAL_MS = exports.HMSH_MAX_TIMEOUT_MS = exports.HMSH_MAX_RETRIES = exports.HMSH_EXPIRE_JOB_SECONDS = exports.HMSH_OTT_WAIT_TIME = exports.HMSH_ACTIVATION_MAX_RETRY = exports.HMSH_QUORUM_DELAY_MS = exports.HMSH_QUORUM_ROLLCALL_CYCLES = exports.HMSH_STATUS_UNKNOWN = exports.HMSH_CODE_DURABLE_RETRYABLE = exports.HMSH_CODE_DURABLE_FATAL = exports.HMSH_CODE_DURABLE_MAXED = exports.HMSH_CODE_DURABLE_TIMEOUT = exports.HMSH_CODE_DURABLE_WAIT = exports.HMSH_CODE_DURABLE_PROXY = exports.HMSH_CODE_DURABLE_CHILD = exports.HMSH_CODE_DURABLE_ALL = exports.HMSH_CODE_DURABLE_SLEEP = exports.HMSH_CODE_UNACKED = exports.HMSH_CODE_TIMEOUT = exports.HMSH_CODE_UNKNOWN = exports.HMSH_CODE_INTERRUPT = exports.HMSH_CODE_NOTFOUND = exports.HMSH_CODE_PENDING = exports.HMSH_CODE_SUCCESS = exports.HMSH_IS_CLUSTER = exports.HMSH_LOGLEVEL = void 0;
+exports.HMSH_GUID_SIZE = exports.HMSH_SCOUT_INTERVAL_SECONDS = exports.HMSH_FIDELITY_SECONDS = exports.HMSH_EXPIRE_DURATION = exports.HMSH_XPENDING_COUNT = exports.HMSH_XCLAIM_COUNT = exports.HMSH_XCLAIM_DELAY_MS = exports.HMSH_BLOCK_TIME_MS = exports.HMSH_DURABLE_EXP_BACKOFF = exports.HMSH_DURABLE_MAX_INTERVAL = exports.HMSH_DURABLE_MAX_ATTEMPTS = exports.HMSH_GRADUATED_INTERVAL_MS = exports.HMSH_MAX_TIMEOUT_MS = exports.HMSH_MAX_RETRIES = exports.MAX_DELAY = exports.HMSH_EXPIRE_JOB_SECONDS = exports.HMSH_OTT_WAIT_TIME = exports.HMSH_ACTIVATION_MAX_RETRY = exports.HMSH_QUORUM_DELAY_MS = exports.HMSH_QUORUM_ROLLCALL_CYCLES = exports.HMSH_STATUS_UNKNOWN = exports.HMSH_CODE_DURABLE_RETRYABLE = exports.HMSH_CODE_DURABLE_FATAL = exports.HMSH_CODE_DURABLE_MAXED = exports.HMSH_CODE_DURABLE_TIMEOUT = exports.HMSH_CODE_DURABLE_WAIT = exports.HMSH_CODE_DURABLE_PROXY = exports.HMSH_CODE_DURABLE_CHILD = exports.HMSH_CODE_DURABLE_ALL = exports.HMSH_CODE_DURABLE_SLEEP = exports.HMSH_CODE_UNACKED = exports.HMSH_CODE_TIMEOUT = exports.HMSH_CODE_UNKNOWN = exports.HMSH_CODE_INTERRUPT = exports.HMSH_CODE_NOTFOUND = exports.HMSH_CODE_PENDING = exports.HMSH_CODE_SUCCESS = exports.HMSH_IS_CLUSTER = exports.HMSH_LOGLEVEL = void 0;
 // HOTMESH SYSTEM
 exports.HMSH_LOGLEVEL = process.env.HMSH_LOGLEVEL || 'info';
 exports.HMSH_IS_CLUSTER = process.env.HMSH_IS_CLUSTER === 'true';
@@ -32,6 +32,7 @@ exports.HMSH_ACTIVATION_MAX_RETRY = 3;
 exports.HMSH_OTT_WAIT_TIME = parseInt(process.env.HMSH_OTT_WAIT_TIME, 10) || 1000;
 exports.HMSH_EXPIRE_JOB_SECONDS = parseInt(process.env.HMSH_EXPIRE_JOB_SECONDS, 10) || 1;
 // STREAM ROUTER
+exports.MAX_DELAY = 2147483647; // Maximum allowed delay in milliseconds for setTimeout
 exports.HMSH_MAX_RETRIES = parseInt(process.env.HMSH_MAX_RETRIES, 10) || 3;
 exports.HMSH_MAX_TIMEOUT_MS = parseInt(process.env.HMSH_MAX_TIMEOUT_MS, 10) || 60000;
 exports.HMSH_GRADUATED_INTERVAL_MS = parseInt(process.env.HMSH_GRADUATED_INTERVAL_MS, 10) || 5000;

package/build/modules/key.d.ts

@@ -4,14 +4,13 @@ import { KeyStoreParams, KeyType } from '../types/hotmesh';
  *
  * hmsh ->                                            {hash}    hotmesh config {version: "0.0.1", namespace: "hmsh"}
  * hmsh:a:<appid> ->                                  {hash}    app profile { "id": "appid", "version": "2", "versions/1": "GMT", "versions/2": "GMT"}
- * hmsh:<appid>:e:<engineId> ->                       {string}  setnx to ensure only one engine of given id
+ * hmsh:<appid>:r: ->                                 {hash}    throttle rates {':': '23', 'topic.thing': '555'} => {':i': 'all', 'topic.thing': '555seconds'}
  * hmsh:<appid>:w: ->                                 {zset}    work items/tasks an engine must do like garbage collect or hook a set of matching records (hookAll)
  * hmsh:<appid>:t: ->                                 {zset}    an ordered set of list (work lists) ids
  * hmsh:<appid>:t:<timeValue?> ->                     {list}    a worklist of `jobId+activityId` items that should be awakened
  * hmsh:<appid>:q: ->                                 {hash}    quorum-wide messages
  * hmsh:<appid>:q:<ngnid> ->                          {hash}    engine-targeted messages (targeted quorum-oriented message)
  * hmsh:<appid>:j:<jobid> ->                          {hash}    job data
- * hmsh:<appid>:j:<jobid>:<activityid>  ->            {hash}    job activity data (a1)
  * hmsh:<appid>:s:<jobkey>:<dateTime> ->              {hash}    job stats (general)
  * hmsh:<appid>:s:<jobkey>:<dateTime>:mdn:<field/path>:<fieldvalue> ->      {zset}    job stats (median)
  * hmsh:<appid>:s:<jobkey>:<dateTime>:index:<field/path>:<fieldvalue> ->    {list}    job stats (index of jobid[])

package/build/modules/key.js

@@ -8,14 +8,13 @@ Object.defineProperty(exports, "KeyType", { enumerable: true, get: function () {
  *
  * hmsh ->                                            {hash}    hotmesh config {version: "0.0.1", namespace: "hmsh"}
  * hmsh:a:<appid> ->                                  {hash}    app profile { "id": "appid", "version": "2", "versions/1": "GMT", "versions/2": "GMT"}
- * hmsh:<appid>:e:<engineId> ->                       {string}  setnx to ensure only one engine of given id
+ * hmsh:<appid>:r: ->                                 {hash}    throttle rates {':': '23', 'topic.thing': '555'} => {':i': 'all', 'topic.thing': '555seconds'}
  * hmsh:<appid>:w: ->                                 {zset}    work items/tasks an engine must do like garbage collect or hook a set of matching records (hookAll)
  * hmsh:<appid>:t: ->                                 {zset}    an ordered set of list (work lists) ids
  * hmsh:<appid>:t:<timeValue?> ->                     {list}    a worklist of `jobId+activityId` items that should be awakened
  * hmsh:<appid>:q: ->                                 {hash}    quorum-wide messages
  * hmsh:<appid>:q:<ngnid> ->                          {hash}    engine-targeted messages (targeted quorum-oriented message)
  * hmsh:<appid>:j:<jobid> ->                          {hash}    job data
- * hmsh:<appid>:j:<jobid>:<activityid>  ->            {hash}    job activity data (a1)
  * hmsh:<appid>:s:<jobkey>:<dateTime> ->              {hash}    job stats (general)
  * hmsh:<appid>:s:<jobkey>:<dateTime>:mdn:<field/path>:<fieldvalue> ->      {zset}    job stats (median)
  * hmsh:<appid>:s:<jobkey>:<dateTime>:index:<field/path>:<fieldvalue> ->    {list}    job stats (index of jobid[])
@@ -56,8 +55,8 @@ class KeyService {
         switch (keyType) {
             case hotmesh_1.KeyType.HOTMESH:
                 return namespace;
-            case hotmesh_1.KeyType.ENGINE_ID:
-                return `${namespace}:${params.appId}:e:${params.engineId}`;
+            case hotmesh_1.KeyType.THROTTLE_RATE:
+                return `${namespace}:${params.appId}:r:`;
             case hotmesh_1.KeyType.WORK_ITEMS:
                 return `${namespace}:${params.appId}:w:${params.scoutType || ''}`;
             case hotmesh_1.KeyType.TIME_RANGE:

package/build/modules/utils.js

@@ -5,7 +5,6 @@ var __importDefault = (this && this.__importDefault) || function (mod) {
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.restoreHierarchy = exports.getValueByPath = exports.getIndexedHash = exports.getSymVal = exports.getSymKey = exports.formatISODate = exports.getTimeSeries = exports.getSubscriptionTopic = exports.findSubscriptionForTrigger = exports.findTopKey = exports.XSleepFor = exports.matchesStatus = exports.matchesStatusCode = exports.identifyRedisTypeFromClass = exports.polyfill = exports.identifyRedisType = exports.deterministicRandom = exports.guid = exports.deepCopy = exports.sleepImmediate = exports.sleepFor = exports.getSystemHealth = void 0;
 const os_1 = __importDefault(require("os"));
-const systeminformation_1 = __importDefault(require("systeminformation"));
 const nanoid_1 = require("nanoid");
 const enums_1 = require("./enums");
 async function safeExecute(operation, defaultValue) {
@@ -21,23 +20,23 @@ async function getSystemHealth() {
     const totalMemory = os_1.default.totalmem();
     const freeMemory = os_1.default.freemem();
     const usedMemory = totalMemory - freeMemory;
-    const cpus = os_1.default.cpus();
+    //const cpus = os.cpus();
     // CPU load calculation remains unchanged
-    const cpuLoad = cpus.map((cpu, i) => {
-        const total = Object.values(cpu.times).reduce((acc, tv) => acc + tv, 0);
-        const idle = cpu.times.idle;
-        const usage = ((total - idle) / total) * 100;
-        return { [`CPU ${i} Usage`]: `${usage.toFixed(2)}%` };
-    });
+    // const cpuLoad = cpus.map((cpu, i) => {
+    //   const total = Object.values(cpu.times).reduce((acc, tv) => acc + tv, 0);
+    //   const idle = cpu.times.idle;
+    //   const usage = ((total - idle) / total) * 100;
+    //   return { [`CPU ${i} Usage`]: `${usage.toFixed(2)}%` };
+    // });
     // Wrap each systeminformation call with safeExecute
-    const networkStats = await safeExecute(systeminformation_1.default.networkStats(), []);
+    //const networkStats = await safeExecute(si.networkStats(), []);
     // Construct the system health object with error handling in mind
     const systemHealth = {
         TotalMemoryGB: `${(totalMemory / 1024 / 1024 / 1024).toFixed(2)} GB`,
         FreeMemoryGB: `${(freeMemory / 1024 / 1024 / 1024).toFixed(2)} GB`,
         UsedMemoryGB: `${(usedMemory / 1024 / 1024 / 1024).toFixed(2)} GB`,
-        CPULoad: cpuLoad,
-        NetworkStats: networkStats,
+        CPULoad: [],
+        NetworkStats: [],
     };
     return systemHealth;
 }

package/build/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@hotmeshio/hotmesh",
-    "version": "0.1.2",
+    "version": "0.1.4",
     "description": "Unbreakable Workflows",
     "main": "./build/index.js",
     "types": "./build/index.d.ts",
@@ -31,7 +31,7 @@
         "test:durable:collision": "NODE_ENV=test jest ./tests/durable/collision/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:fatal": "NODE_ENV=test jest ./tests/durable/fatal/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:goodbye": "NODE_ENV=test jest ./tests/durable/goodbye/index.test.ts --detectOpenHandles --forceExit --verbose",
-        "test:durable:hello": "NODE_ENV=test jest ./tests/durable/helloworld/index.test.ts --detectOpenHandles --forceExit --verbose",
+        "test:durable:hello": "HMSH_IS_CLUSTER=true NODE_ENV=test jest ./tests/durable/helloworld/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:hook": "NODE_ENV=test jest ./tests/durable/hook/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:interrupt": "NODE_ENV=test jest ./tests/durable/interrupt/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:loopactivity": "NODE_ENV=test jest ./tests/durable/loopactivity/index.test.ts --detectOpenHandles --forceExit --verbose",
@@ -41,6 +41,7 @@
         "test:durable:signal": "NODE_ENV=test jest ./tests/durable/signal/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:unknown": "NODE_ENV=test jest ./tests/durable/unknown/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:emit": "NODE_ENV=test jest ./tests/functional/emit/index.test.ts --detectOpenHandles --forceExit --verbose",
+        "test:expired": "NODE_ENV=test jest ./tests/functional/expired/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:functional": "NODE_ENV=test jest ./tests/functional/*/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:hmsh": "NODE_ENV=test jest ./tests/functional/index.test.ts --detectOpenHandles --verbose",
         "test:hook": "NODE_ENV=test jest ./tests/functional/hook/index.test.ts --detectOpenHandles --forceExit --verbose",

package/build/services/activities/activity.js

@@ -112,7 +112,7 @@ class Activity {
                 stack: error.stack,
                 name: error.name,
             });
-            telemetry && telemetry.setActivityError(error.message);
+            telemetry?.setActivityError(error.message);
             throw error;
         }
         finally {

package/build/services/activities/await.js

@@ -58,7 +58,7 @@ class Await extends activity_1.Activity {
             else {
                 this.logger.error('await-process-error', { ...error });
             }
-            telemetry.setActivityError(error.message);
+            telemetry?.setActivityError(error.message);
             throw error;
         }
         finally {

package/build/services/activities/cycle.js

@@ -58,7 +58,7 @@ class Cycle extends activity_1.Activity {
             else {
                 this.logger.error('cycle-process-error', { ...error });
             }
-            telemetry.setActivityError(error.message);
+            telemetry?.setActivityError(error.message);
             throw error;
         }
         finally {

package/build/services/activities/hook.js

@@ -53,7 +53,7 @@ class Hook extends activity_1.Activity {
             else {
                 this.logger.error('hook-process-error', { ...error });
             }
-            telemetry.setActivityError(error.message);
+            telemetry?.setActivityError(error.message);
             throw error;
         }
         finally {

package/build/services/activities/interrupt.js

@@ -45,7 +45,7 @@ class Interrupt extends activity_1.Activity {
             else {
                 this.logger.error('interrupt-process-error', { ...error });
             }
-            telemetry.setActivityError(error.message);
+            telemetry?.setActivityError(error.message);
             throw error;
         }
         finally {

package/build/services/activities/signal.js

@@ -66,7 +66,7 @@ class Signal extends activity_1.Activity {
             else {
                 this.logger.error('signal-process-error', { ...error });
             }
-            telemetry.setActivityError(error.message);
+            telemetry?.setActivityError(error.message);
             throw error;
         }
         finally {

package/build/services/activities/trigger.d.ts

@@ -7,6 +7,7 @@ declare class Trigger extends Activity {
     config: TriggerActivity;
     constructor(config: ActivityType, data: ActivityData, metadata: ActivityMetadata, hook: ActivityData | null, engine: EngineService, context?: JobState);
     process(options?: ExtensionType): Promise<string>;
+    setExpired(seconds: number, multi: RedisMulti): Promise<void>;
     safeKey(key: string): string;
     bindSearchData(options?: ExtensionType): void;
     bindMarkerData(options?: ExtensionType): void;
@@ -20,7 +21,7 @@ declare class Trigger extends Activity {
     getJobStatus(): number;
     resolveJobId(context: Partial<JobState>): string;
     resolveJobKey(context: Partial<JobState>): string;
-    setStateNX(): Promise<void>;
+    setStateNX(status?: number): Promise<void>;
     /**
      * Registers this job as a dependent of the parent job; when the
      * parent job is interrupted, this job will be interrupted

package/build/services/activities/trigger.js

@@ -25,15 +25,20 @@ class Trigger extends activity_1.Activity {
             telemetry.startJobSpan();
             telemetry.startActivitySpan(this.leg);
             this.mapJobData();
-            await this.setStateNX();
+            await this.setStateNX(options?.expired ? -1 : undefined);
             this.adjacencyList = await this.filterAdjacent();
-            await this.setStatus(this.adjacencyList.length);
+            await this.setStatus(options?.expired ? -1 : this.adjacencyList.length);
             this.bindSearchData(options);
             this.bindMarkerData(options);
             const multi = this.store.getMulti();
             await this.setState(multi);
             await this.setStats(multi);
-            await this.registerJobDependency(multi);
+            if (options?.expired) {
+                await this.setExpired(options?.expired, multi);
+            }
+            else {
+                await this.registerJobDependency(multi);
+            }
             await multi.exec();
             //if the parent (spawner) chose not to await,
             // emit the job_id as the data payload { job_id }
@@ -42,9 +47,13 @@ class Trigger extends activity_1.Activity {
             const jobStatus = Number(this.context.metadata.js);
             telemetry.setJobAttributes({ 'app.job.jss': jobStatus });
             const attrs = { 'app.job.jss': jobStatus };
-            const messageIds = await this.transition(this.adjacencyList, jobStatus);
-            if (messageIds.length) {
-                attrs['app.activity.mids'] = messageIds.join(',');
+            //only transition if not inited in an expired state
+            //todo: enable resume from expired state
+            if (jobStatus !== -1) {
+                const messageIds = await this.transition(this.adjacencyList, jobStatus);
+                if (messageIds.length) {
+                    attrs['app.activity.mids'] = messageIds.join(',');
+                }
             }
             telemetry.setActivityAttributes(attrs);
             return this.context.metadata.jid;
@@ -56,7 +65,7 @@ class Trigger extends activity_1.Activity {
             else {
                 this.logger.error('trigger-process-error', { ...error });
             }
-            telemetry.setActivityError(error.message);
+            telemetry?.setActivityError(error.message);
             throw error;
         }
         finally {
@@ -69,6 +78,9 @@ class Trigger extends activity_1.Activity {
             });
         }
     }
+    async setExpired(seconds, multi) {
+        await this.store.expireJob(this.context.metadata.jid, seconds, multi);
+    }
     safeKey(key) {
         return `_${key}`;
     }
@@ -190,9 +202,9 @@ class Trigger extends activity_1.Activity {
         const jobKey = this.config.stats?.key;
         return jobKey ? pipe_1.Pipe.resolve(jobKey, context) : '';
     }
-    async setStateNX() {
+    async setStateNX(status) {
         const jobId = this.context.metadata.jid;
-        if (!await this.store.setStateNX(jobId, this.engine.appId)) {
+        if (!await this.store.setStateNX(jobId, this.engine.appId, status)) {
             throw new errors_1.DuplicateJobError(jobId);
         }
     }

package/build/services/activities/worker.js

@@ -57,7 +57,7 @@ class Worker extends activity_1.Activity {
             else {
                 this.logger.error('worker-process-error', { ...error });
             }
-            telemetry.setActivityError(error.message);
+            telemetry?.setActivityError(error.message);
             throw error;
         }
         finally {

package/build/services/compiler/deployer.js

@@ -479,7 +479,7 @@ class Deployer {
             await this.store.xgroup('CREATE', stream, group, '$', 'MKSTREAM');
         }
         catch (err) {
-            this.store.logger.info('consumer-group-exists', { stream, group });
+            this.store.logger.info('router-stream-group-exists', { stream, group });
         }
     }
 }

package/build/services/durable/client.js

@@ -73,7 +73,11 @@ class ClientService {
                         1000,
                 };
                 const context = { metadata: { trc, spn }, data: {} };
-                const jobId = await hotMeshClient.pub(`${options.namespace ?? factory_1.APP_ID}.execute`, payload, context, { search: options?.search?.data, marker: options?.marker });
+                const jobId = await hotMeshClient.pub(`${options.namespace ?? factory_1.APP_ID}.execute`, payload, context, {
+                    search: options?.search?.data,
+                    marker: options?.marker,
+                    expired: options?.expired,
+                });
                 return new handle_1.WorkflowHandleService(hotMeshClient, workflowTopic, jobId);
             },
             /**

package/build/services/durable/worker.js

@@ -86,7 +86,7 @@ class WorkerService {
         }
     }
     async run() {
-        this.workflowRunner.engine.logger.info('WorkerService is running');
+        this.workflowRunner.engine.logger.info('durable-worker-running');
     }
     async initActivityWorker(config, activityTopic) {
         const redisConfig = {

package/build/services/engine/index.d.ts

@@ -47,7 +47,7 @@ declare class EngineService {
     initStoreChannel(store: RedisClient): Promise<void>;
     initSubChannel(sub: RedisClient): Promise<void>;
     initStreamChannel(stream: RedisClient): Promise<void>;
-    initRouter(config: HotMeshConfig): Router;
+    initRouter(config: HotMeshConfig): Promise<Router>;
     getVID(vid?: AppVID): Promise<AppVID>;
     setCacheMode(cacheMode: CacheMode, untilVersion: string): void;
     routeToSubscribers(topic: string, message: JobOutput): Promise<void>;

package/build/services/engine/index.js

@@ -40,7 +40,7 @@ class EngineService {
             await instance.initStoreChannel(config.engine.store);
             await instance.initSubChannel(config.engine.sub);
             await instance.initStreamChannel(config.engine.stream);
-            instance.router = instance.initRouter(config);
+            instance.router = await instance.initRouter(config);
             instance.router.consumeMessages(instance.stream.mintKey(key_1.KeyType.STREAMS, { appId: instance.appId }), 'ENGINE', instance.guid, instance.processStreamMessage.bind(instance));
             instance.taskService = new task_1.TaskService(instance.store, logger);
             instance.exporter = new exporter_1.ExporterService(instance.appId, instance.store, logger);
@@ -82,7 +82,8 @@ class EngineService {
         }
         await this.stream.init(this.namespace, this.appId, this.logger);
     }
-    initRouter(config) {
+    async initRouter(config) {
+        const throttle = await this.store.getThrottleRate(':');
         return new router_1.Router({
             namespace: this.namespace,
             appId: this.appId,
@@ -90,6 +91,7 @@ class EngineService {
             role: stream_1.StreamRole.ENGINE,
             reclaimDelay: config.engine.reclaimDelay,
             reclaimCount: config.engine.reclaimCount,
+            throttle,
         }, this.stream, this.store, this.logger);
     }
     async getVID(vid) {
@@ -114,7 +116,7 @@ class EngineService {
         }
     }
     setCacheMode(cacheMode, untilVersion) {
-        this.logger.info(`engine-rule-cache-updated`, {
+        this.logger.info(`engine-cache-updated`, {
             mode: cacheMode,
             until: untilVersion,
         });
@@ -135,7 +137,12 @@ class EngineService {
         this.taskService.processTimeHooks(this.hookTime.bind(this));
     }
     async throttle(delayInMillis) {
-        this.router.setThrottle(delayInMillis);
+        try {
+            this.router.setThrottle(delayInMillis);
+        }
+        catch (e) {
+            this.logger.error('engine-throttle-error', { error: e });
+        }
     }
     // ************* METADATA/MODEL METHODS *************
     async initActivity(topic, data = {}, context) {
@@ -220,7 +227,7 @@ class EngineService {
     }
     // ****************** STREAM RE-ENTRY POINT *****************
     async processStreamMessage(streamData) {
-        this.logger.debug('engine-process-stream-message', {
+        this.logger.debug('engine-process', {
             jid: streamData.metadata.jid,
             gid: streamData.metadata.gid,
             dad: streamData.metadata.dad,
@@ -279,7 +286,7 @@ class EngineService {
             const activityHandler = (await this.initActivity(`.${streamData.metadata.aid}`, streamData.data, context));
             await activityHandler.processEvent(streamData.status, streamData.code, 'output');
         }
-        this.logger.debug('engine-process-stream-message-end', {
+        this.logger.debug('engine-process-end', {
             jid: streamData.metadata.jid,
             gid: streamData.metadata.gid,
             aid: streamData.metadata.aid,

package/build/services/hotmesh/index.js

@@ -11,6 +11,7 @@ const logger_1 = require("../logger");
 const quorum_1 = require("../quorum");
 const router_1 = require("../router");
 const worker_1 = require("../worker");
+const enums_1 = require("../../modules/enums");
 class HotMeshService {
     constructor() {
         this.engine = null;
@@ -94,9 +95,19 @@ class HotMeshService {
         return await this.quorum?.rollCall(delay);
     }
     async throttle(options) {
+        let throttle;
+        if (options.throttle === -1) {
+            throttle = enums_1.MAX_DELAY;
+        }
+        else {
+            throttle = options.throttle;
+        }
+        if (!Number.isInteger(throttle) || throttle < 0 || throttle > enums_1.MAX_DELAY) {
+            throw new Error(`Throttle must be a non-negative integer and not exceed ${enums_1.MAX_DELAY} ms; send -1 to throttle indefinitely`);
+        }
         const throttleMessage = {
             type: 'throttle',
-            throttle: options.throttle,
+            throttle: throttle,
         };
         if (options.guid) {
             throttleMessage.guid = options.guid;
@@ -104,6 +115,7 @@ class HotMeshService {
         else if (options.topic) {
             throttleMessage.topic = options.topic;
         }
+        await this.engine.store.setThrottleRate(throttleMessage);
         return await this.quorum?.pub(throttleMessage);
     }
     // ************* COMPILER METHODS *************

package/build/services/router/index.d.ts

@@ -42,6 +42,8 @@ declare class Router {
     customSleep(): Promise<void>;
     consumeMessages(stream: string, group: string, consumer: string, callback: (streamData: StreamData) => Promise<StreamDataResponse | void>): Promise<void>;
     isStreamMessage(result: any): boolean;
+    isPaused(): boolean;
+    isStopped(group: string, consumer: string, stream: string): boolean;
     consumeOne(stream: string, group: string, id: string, message: string[], callback: (streamData: StreamData) => Promise<StreamDataResponse | void>): Promise<void>;
     execStreamLeg(input: StreamData, stream: string, id: string, callback: (streamData: StreamData) => Promise<StreamDataResponse | void>): Promise<StreamData>;
     ackAndDelete(stream: string, group: string, id: string): Promise<void>;