@hotmeshio/hotmesh 0.1.3 → 0.1.4

package/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,38 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+**Smartphone (please complete the following information):**
+ - Device: [e.g. iPhone6]
+ - OS: [e.g. iOS8.1]
+ - Browser [e.g. stock browser, safari]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.

package/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

package/README.md

@@ -185,7 +185,7 @@ Externalizing state fundamentally changes the execution profile for your functio
       const value = await search.mult('name', 12);
       ```
 
-Refer to the [hotmeshio/samples-typescript](https://github.com/hotmeshio/samples-typescript) repo for usage examples. 
+Refer to the [hotmeshio/samples-javascript](https://github.com/hotmeshio/samples-javascript) repo for usage examples.
 
 ### Design | Advanced
 The *Pluck* and *Durable* modules are the easiest way to use HotMesh. But if you need full control over your function lifecycles (including high-volume, high-speed use cases), you can use HotMesh's underlying YAML models to optimize your durable workflows. The following model depicts a sequence of activities orchestrated by HotMesh. Any function you associate with a `topic` in your YAML definition is guaranteed to be durable.
@@ -295,28 +295,31 @@ const hotMesh = await HotMesh.init({
 ```
 
 ### Observability
-Workflows and activities are run according to the rules you define, offering [Graph-Oriented](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/system_lifecycle.md#telemetry) telemetry insights into your legacy function executions.
+Workflows and activities are run according to the rules you define, offering [Graph-Oriented](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/system_lifecycle.md#telemetry) telemetry insights into your legacy function executions.
 
 ## FAQ
-Refer to the [FAQ](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/faq.md) for terminology, definitions, and an exploration of how HotMesh facilitates orchestration use cases.
+Refer to the [FAQ](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/faq.md) for terminology, definitions, and an exploration of how HotMesh facilitates orchestration use cases.
 
 ## Quick Start
-Refer to the [Quick Start](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/quickstart.md) for sample YAML workflows you can copy, paste, and modify to get started.
+Refer to the [Quick Start](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/quickstart.md) for sample YAML workflows you can copy, paste, and modify to get started.
 
 ## Developer Guide
-For more details on the complete development process, including information about schemas, APIs, and deployment, consult the [Developer Guide](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/developer_guide.md).
+For more details on the complete development process, including information about schemas, APIs, and deployment, consult the [Developer Guide](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/developer_guide.md).
 
 ## Model Driven Development
-[Model Driven Development](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/model_driven_development.md) is an established strategy for managing process-oriented tasks. Check out this guide to understand its foundational principles.
+[Model Driven Development](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/model_driven_development.md) is an established strategy for managing process-oriented tasks. Check out this guide to understand its foundational principles.
 
 ## Data Mapping
-Exchanging data between activities is central to HotMesh. For detailed information on supported functions and the functional mapping syntax (@pipes), see the [Data Mapping Overview](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/data_mapping.md).
+Exchanging data between activities is central to HotMesh. For detailed information on supported functions and the functional mapping syntax (@pipes), see the [Data Mapping Overview](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/data_mapping.md).
 
 ## Composition
-While the simplest graphs are linear, detailing a consistent sequence of non-cyclical activities, graphs can be layered to represent intricate business scenarios. Some can even be designed to accommodate long-lasting workflows that span months. For more details, check out the [Composable Workflow Guide](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/composable_workflow.md).
-
-## Distributed Orchestration
-HotMesh is a distributed orchestration engine. Refer to the [Distributed Orchestration Guide](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/distributed_orchestration.md) for a detailed breakdown of the approach.
+While the simplest graphs are linear, detailing a consistent sequence of non-cyclical activities, graphs can be layered to represent intricate business scenarios. Some can even be designed to accommodate long-lasting workflows that span months. For more details, check out the [Composable Workflow Guide](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/composable_workflow.md).
 
 ## System Lifecycle
-Gain insight into HotMesh's monitoring, exception handling, and alarm configurations via the [System Lifecycle Guide](https://github.com/hotmeshio/sdk-typescript/tree/main/docs/system_lifecycle.md).
+Gain insight into HotMesh's monitoring, exception handling, and alarm configurations via the [System Lifecycle Guide](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/system_lifecycle.md).
+
+## Distributed Orchestration | System Overview
+HotMesh is a distributed orchestration engine. Refer to the [Distributed Orchestration Guide](https://github.com/hotmeshio/sdk-typescript/blob/main/docs/distributed_orchestration.md) for a high-level overview of the approach.
+
+## Distributed Orchestration | System Design
+HotMesh is more than Redis and TypeScript. The theory that underlies the architecture is applicable to any number of data storage and streaming backends: [A Message-Oriented Approach to Decentralized Process Orchestration](https://zenodo.org/records/12168558).

package/build/modules/enums.d.ts

@@ -23,6 +23,7 @@ export declare const HMSH_QUORUM_DELAY_MS = 250;
 export declare const HMSH_ACTIVATION_MAX_RETRY = 3;
 export declare const HMSH_OTT_WAIT_TIME: number;
 export declare const HMSH_EXPIRE_JOB_SECONDS: number;
+export declare const MAX_DELAY = 2147483647;
 export declare const HMSH_MAX_RETRIES: number;
 export declare const HMSH_MAX_TIMEOUT_MS: number;
 export declare const HMSH_GRADUATED_INTERVAL_MS: number;

package/build/modules/enums.js

@@ -1,6 +1,6 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.HMSH_GUID_SIZE = exports.HMSH_SCOUT_INTERVAL_SECONDS = exports.HMSH_FIDELITY_SECONDS = exports.HMSH_EXPIRE_DURATION = exports.HMSH_XPENDING_COUNT = exports.HMSH_XCLAIM_COUNT = exports.HMSH_XCLAIM_DELAY_MS = exports.HMSH_BLOCK_TIME_MS = exports.HMSH_DURABLE_EXP_BACKOFF = exports.HMSH_DURABLE_MAX_INTERVAL = exports.HMSH_DURABLE_MAX_ATTEMPTS = exports.HMSH_GRADUATED_INTERVAL_MS = exports.HMSH_MAX_TIMEOUT_MS = exports.HMSH_MAX_RETRIES = exports.HMSH_EXPIRE_JOB_SECONDS = exports.HMSH_OTT_WAIT_TIME = exports.HMSH_ACTIVATION_MAX_RETRY = exports.HMSH_QUORUM_DELAY_MS = exports.HMSH_QUORUM_ROLLCALL_CYCLES = exports.HMSH_STATUS_UNKNOWN = exports.HMSH_CODE_DURABLE_RETRYABLE = exports.HMSH_CODE_DURABLE_FATAL = exports.HMSH_CODE_DURABLE_MAXED = exports.HMSH_CODE_DURABLE_TIMEOUT = exports.HMSH_CODE_DURABLE_WAIT = exports.HMSH_CODE_DURABLE_PROXY = exports.HMSH_CODE_DURABLE_CHILD = exports.HMSH_CODE_DURABLE_ALL = exports.HMSH_CODE_DURABLE_SLEEP = exports.HMSH_CODE_UNACKED = exports.HMSH_CODE_TIMEOUT = exports.HMSH_CODE_UNKNOWN = exports.HMSH_CODE_INTERRUPT = exports.HMSH_CODE_NOTFOUND = exports.HMSH_CODE_PENDING = exports.HMSH_CODE_SUCCESS = exports.HMSH_IS_CLUSTER = exports.HMSH_LOGLEVEL = void 0;
+exports.HMSH_GUID_SIZE = exports.HMSH_SCOUT_INTERVAL_SECONDS = exports.HMSH_FIDELITY_SECONDS = exports.HMSH_EXPIRE_DURATION = exports.HMSH_XPENDING_COUNT = exports.HMSH_XCLAIM_COUNT = exports.HMSH_XCLAIM_DELAY_MS = exports.HMSH_BLOCK_TIME_MS = exports.HMSH_DURABLE_EXP_BACKOFF = exports.HMSH_DURABLE_MAX_INTERVAL = exports.HMSH_DURABLE_MAX_ATTEMPTS = exports.HMSH_GRADUATED_INTERVAL_MS = exports.HMSH_MAX_TIMEOUT_MS = exports.HMSH_MAX_RETRIES = exports.MAX_DELAY = exports.HMSH_EXPIRE_JOB_SECONDS = exports.HMSH_OTT_WAIT_TIME = exports.HMSH_ACTIVATION_MAX_RETRY = exports.HMSH_QUORUM_DELAY_MS = exports.HMSH_QUORUM_ROLLCALL_CYCLES = exports.HMSH_STATUS_UNKNOWN = exports.HMSH_CODE_DURABLE_RETRYABLE = exports.HMSH_CODE_DURABLE_FATAL = exports.HMSH_CODE_DURABLE_MAXED = exports.HMSH_CODE_DURABLE_TIMEOUT = exports.HMSH_CODE_DURABLE_WAIT = exports.HMSH_CODE_DURABLE_PROXY = exports.HMSH_CODE_DURABLE_CHILD = exports.HMSH_CODE_DURABLE_ALL = exports.HMSH_CODE_DURABLE_SLEEP = exports.HMSH_CODE_UNACKED = exports.HMSH_CODE_TIMEOUT = exports.HMSH_CODE_UNKNOWN = exports.HMSH_CODE_INTERRUPT = exports.HMSH_CODE_NOTFOUND = exports.HMSH_CODE_PENDING = exports.HMSH_CODE_SUCCESS = exports.HMSH_IS_CLUSTER = exports.HMSH_LOGLEVEL = void 0;
 // HOTMESH SYSTEM
 exports.HMSH_LOGLEVEL = process.env.HMSH_LOGLEVEL || 'info';
 exports.HMSH_IS_CLUSTER = process.env.HMSH_IS_CLUSTER === 'true';
@@ -32,6 +32,7 @@ exports.HMSH_ACTIVATION_MAX_RETRY = 3;
 exports.HMSH_OTT_WAIT_TIME = parseInt(process.env.HMSH_OTT_WAIT_TIME, 10) || 1000;
 exports.HMSH_EXPIRE_JOB_SECONDS = parseInt(process.env.HMSH_EXPIRE_JOB_SECONDS, 10) || 1;
 // STREAM ROUTER
+exports.MAX_DELAY = 2147483647; // Maximum allowed delay in milliseconds for setTimeout
 exports.HMSH_MAX_RETRIES = parseInt(process.env.HMSH_MAX_RETRIES, 10) || 3;
 exports.HMSH_MAX_TIMEOUT_MS = parseInt(process.env.HMSH_MAX_TIMEOUT_MS, 10) || 60000;
 exports.HMSH_GRADUATED_INTERVAL_MS = parseInt(process.env.HMSH_GRADUATED_INTERVAL_MS, 10) || 5000;

package/build/modules/key.d.ts

@@ -4,14 +4,13 @@ import { KeyStoreParams, KeyType } from '../types/hotmesh';
  *
  * hmsh ->                                            {hash}    hotmesh config {version: "0.0.1", namespace: "hmsh"}
  * hmsh:a:<appid> ->                                  {hash}    app profile { "id": "appid", "version": "2", "versions/1": "GMT", "versions/2": "GMT"}
- * hmsh:<appid>:e:<engineId> ->                       {string}  setnx to ensure only one engine of given id
+ * hmsh:<appid>:r: ->                                 {hash}    throttle rates {':': '23', 'topic.thing': '555'} => {':i': 'all', 'topic.thing': '555seconds'}
  * hmsh:<appid>:w: ->                                 {zset}    work items/tasks an engine must do like garbage collect or hook a set of matching records (hookAll)
  * hmsh:<appid>:t: ->                                 {zset}    an ordered set of list (work lists) ids
  * hmsh:<appid>:t:<timeValue?> ->                     {list}    a worklist of `jobId+activityId` items that should be awakened
  * hmsh:<appid>:q: ->                                 {hash}    quorum-wide messages
  * hmsh:<appid>:q:<ngnid> ->                          {hash}    engine-targeted messages (targeted quorum-oriented message)
  * hmsh:<appid>:j:<jobid> ->                          {hash}    job data
- * hmsh:<appid>:j:<jobid>:<activityid>  ->            {hash}    job activity data (a1)
  * hmsh:<appid>:s:<jobkey>:<dateTime> ->              {hash}    job stats (general)
  * hmsh:<appid>:s:<jobkey>:<dateTime>:mdn:<field/path>:<fieldvalue> ->      {zset}    job stats (median)
  * hmsh:<appid>:s:<jobkey>:<dateTime>:index:<field/path>:<fieldvalue> ->    {list}    job stats (index of jobid[])

package/build/modules/key.js

@@ -8,14 +8,13 @@ Object.defineProperty(exports, "KeyType", { enumerable: true, get: function () {
  *
  * hmsh ->                                            {hash}    hotmesh config {version: "0.0.1", namespace: "hmsh"}
  * hmsh:a:<appid> ->                                  {hash}    app profile { "id": "appid", "version": "2", "versions/1": "GMT", "versions/2": "GMT"}
- * hmsh:<appid>:e:<engineId> ->                       {string}  setnx to ensure only one engine of given id
+ * hmsh:<appid>:r: ->                                 {hash}    throttle rates {':': '23', 'topic.thing': '555'} => {':i': 'all', 'topic.thing': '555seconds'}
  * hmsh:<appid>:w: ->                                 {zset}    work items/tasks an engine must do like garbage collect or hook a set of matching records (hookAll)
  * hmsh:<appid>:t: ->                                 {zset}    an ordered set of list (work lists) ids
  * hmsh:<appid>:t:<timeValue?> ->                     {list}    a worklist of `jobId+activityId` items that should be awakened
  * hmsh:<appid>:q: ->                                 {hash}    quorum-wide messages
  * hmsh:<appid>:q:<ngnid> ->                          {hash}    engine-targeted messages (targeted quorum-oriented message)
  * hmsh:<appid>:j:<jobid> ->                          {hash}    job data
- * hmsh:<appid>:j:<jobid>:<activityid>  ->            {hash}    job activity data (a1)
  * hmsh:<appid>:s:<jobkey>:<dateTime> ->              {hash}    job stats (general)
  * hmsh:<appid>:s:<jobkey>:<dateTime>:mdn:<field/path>:<fieldvalue> ->      {zset}    job stats (median)
  * hmsh:<appid>:s:<jobkey>:<dateTime>:index:<field/path>:<fieldvalue> ->    {list}    job stats (index of jobid[])
@@ -56,8 +55,8 @@ class KeyService {
         switch (keyType) {
             case hotmesh_1.KeyType.HOTMESH:
                 return namespace;
-            case hotmesh_1.KeyType.ENGINE_ID:
-                return `${namespace}:${params.appId}:e:${params.engineId}`;
+            case hotmesh_1.KeyType.THROTTLE_RATE:
+                return `${namespace}:${params.appId}:r:`;
             case hotmesh_1.KeyType.WORK_ITEMS:
                 return `${namespace}:${params.appId}:w:${params.scoutType || ''}`;
             case hotmesh_1.KeyType.TIME_RANGE:

package/build/modules/utils.js

@@ -5,7 +5,6 @@ var __importDefault = (this && this.__importDefault) || function (mod) {
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.restoreHierarchy = exports.getValueByPath = exports.getIndexedHash = exports.getSymVal = exports.getSymKey = exports.formatISODate = exports.getTimeSeries = exports.getSubscriptionTopic = exports.findSubscriptionForTrigger = exports.findTopKey = exports.XSleepFor = exports.matchesStatus = exports.matchesStatusCode = exports.identifyRedisTypeFromClass = exports.polyfill = exports.identifyRedisType = exports.deterministicRandom = exports.guid = exports.deepCopy = exports.sleepImmediate = exports.sleepFor = exports.getSystemHealth = void 0;
 const os_1 = __importDefault(require("os"));
-const systeminformation_1 = __importDefault(require("systeminformation"));
 const nanoid_1 = require("nanoid");
 const enums_1 = require("./enums");
 async function safeExecute(operation, defaultValue) {
@@ -21,23 +20,23 @@ async function getSystemHealth() {
     const totalMemory = os_1.default.totalmem();
     const freeMemory = os_1.default.freemem();
     const usedMemory = totalMemory - freeMemory;
-    const cpus = os_1.default.cpus();
+    //const cpus = os.cpus();
     // CPU load calculation remains unchanged
-    const cpuLoad = cpus.map((cpu, i) => {
-        const total = Object.values(cpu.times).reduce((acc, tv) => acc + tv, 0);
-        const idle = cpu.times.idle;
-        const usage = ((total - idle) / total) * 100;
-        return { [`CPU ${i} Usage`]: `${usage.toFixed(2)}%` };
-    });
+    // const cpuLoad = cpus.map((cpu, i) => {
+    //   const total = Object.values(cpu.times).reduce((acc, tv) => acc + tv, 0);
+    //   const idle = cpu.times.idle;
+    //   const usage = ((total - idle) / total) * 100;
+    //   return { [`CPU ${i} Usage`]: `${usage.toFixed(2)}%` };
+    // });
     // Wrap each systeminformation call with safeExecute
-    const networkStats = await safeExecute(systeminformation_1.default.networkStats(), []);
+    //const networkStats = await safeExecute(si.networkStats(), []);
     // Construct the system health object with error handling in mind
     const systemHealth = {
         TotalMemoryGB: `${(totalMemory / 1024 / 1024 / 1024).toFixed(2)} GB`,
         FreeMemoryGB: `${(freeMemory / 1024 / 1024 / 1024).toFixed(2)} GB`,
         UsedMemoryGB: `${(usedMemory / 1024 / 1024 / 1024).toFixed(2)} GB`,
-        CPULoad: cpuLoad,
-        NetworkStats: networkStats,
+        CPULoad: [],
+        NetworkStats: [],
     };
     return systemHealth;
 }

package/build/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@hotmeshio/hotmesh",
-    "version": "0.1.3",
+    "version": "0.1.4",
     "description": "Unbreakable Workflows",
     "main": "./build/index.js",
     "types": "./build/index.d.ts",
@@ -31,7 +31,7 @@
         "test:durable:collision": "NODE_ENV=test jest ./tests/durable/collision/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:fatal": "NODE_ENV=test jest ./tests/durable/fatal/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:goodbye": "NODE_ENV=test jest ./tests/durable/goodbye/index.test.ts --detectOpenHandles --forceExit --verbose",
-        "test:durable:hello": "NODE_ENV=test jest ./tests/durable/helloworld/index.test.ts --detectOpenHandles --forceExit --verbose",
+        "test:durable:hello": "HMSH_IS_CLUSTER=true NODE_ENV=test jest ./tests/durable/helloworld/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:hook": "NODE_ENV=test jest ./tests/durable/hook/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:interrupt": "NODE_ENV=test jest ./tests/durable/interrupt/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:loopactivity": "NODE_ENV=test jest ./tests/durable/loopactivity/index.test.ts --detectOpenHandles --forceExit --verbose",
@@ -41,6 +41,7 @@
         "test:durable:signal": "NODE_ENV=test jest ./tests/durable/signal/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:durable:unknown": "NODE_ENV=test jest ./tests/durable/unknown/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:emit": "NODE_ENV=test jest ./tests/functional/emit/index.test.ts --detectOpenHandles --forceExit --verbose",
+        "test:expired": "NODE_ENV=test jest ./tests/functional/expired/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:functional": "NODE_ENV=test jest ./tests/functional/*/index.test.ts --detectOpenHandles --forceExit --verbose",
         "test:hmsh": "NODE_ENV=test jest ./tests/functional/index.test.ts --detectOpenHandles --verbose",
         "test:hook": "NODE_ENV=test jest ./tests/functional/hook/index.test.ts --detectOpenHandles --forceExit --verbose",

package/build/services/activities/trigger.d.ts

@@ -7,6 +7,7 @@ declare class Trigger extends Activity {
     config: TriggerActivity;
     constructor(config: ActivityType, data: ActivityData, metadata: ActivityMetadata, hook: ActivityData | null, engine: EngineService, context?: JobState);
     process(options?: ExtensionType): Promise<string>;
+    setExpired(seconds: number, multi: RedisMulti): Promise<void>;
     safeKey(key: string): string;
     bindSearchData(options?: ExtensionType): void;
     bindMarkerData(options?: ExtensionType): void;
@@ -20,7 +21,7 @@ declare class Trigger extends Activity {
     getJobStatus(): number;
     resolveJobId(context: Partial<JobState>): string;
     resolveJobKey(context: Partial<JobState>): string;
-    setStateNX(): Promise<void>;
+    setStateNX(status?: number): Promise<void>;
     /**
      * Registers this job as a dependent of the parent job; when the
      * parent job is interrupted, this job will be interrupted

package/build/services/activities/trigger.js

@@ -25,15 +25,20 @@ class Trigger extends activity_1.Activity {
             telemetry.startJobSpan();
             telemetry.startActivitySpan(this.leg);
             this.mapJobData();
-            await this.setStateNX();
+            await this.setStateNX(options?.expired ? -1 : undefined);
             this.adjacencyList = await this.filterAdjacent();
-            await this.setStatus(this.adjacencyList.length);
+            await this.setStatus(options?.expired ? -1 : this.adjacencyList.length);
             this.bindSearchData(options);
             this.bindMarkerData(options);
             const multi = this.store.getMulti();
             await this.setState(multi);
             await this.setStats(multi);
-            await this.registerJobDependency(multi);
+            if (options?.expired) {
+                await this.setExpired(options?.expired, multi);
+            }
+            else {
+                await this.registerJobDependency(multi);
+            }
             await multi.exec();
             //if the parent (spawner) chose not to await,
             // emit the job_id as the data payload { job_id }
@@ -42,9 +47,13 @@ class Trigger extends activity_1.Activity {
             const jobStatus = Number(this.context.metadata.js);
             telemetry.setJobAttributes({ 'app.job.jss': jobStatus });
             const attrs = { 'app.job.jss': jobStatus };
-            const messageIds = await this.transition(this.adjacencyList, jobStatus);
-            if (messageIds.length) {
-                attrs['app.activity.mids'] = messageIds.join(',');
+            //only transition if not inited in an expired state
+            //todo: enable resume from expired state
+            if (jobStatus !== -1) {
+                const messageIds = await this.transition(this.adjacencyList, jobStatus);
+                if (messageIds.length) {
+                    attrs['app.activity.mids'] = messageIds.join(',');
+                }
             }
             telemetry.setActivityAttributes(attrs);
             return this.context.metadata.jid;
@@ -69,6 +78,9 @@ class Trigger extends activity_1.Activity {
             });
         }
     }
+    async setExpired(seconds, multi) {
+        await this.store.expireJob(this.context.metadata.jid, seconds, multi);
+    }
     safeKey(key) {
         return `_${key}`;
     }
@@ -190,9 +202,9 @@ class Trigger extends activity_1.Activity {
         const jobKey = this.config.stats?.key;
         return jobKey ? pipe_1.Pipe.resolve(jobKey, context) : '';
     }
-    async setStateNX() {
+    async setStateNX(status) {
         const jobId = this.context.metadata.jid;
-        if (!await this.store.setStateNX(jobId, this.engine.appId)) {
+        if (!await this.store.setStateNX(jobId, this.engine.appId, status)) {
             throw new errors_1.DuplicateJobError(jobId);
         }
     }

package/build/services/compiler/deployer.js

@@ -479,7 +479,7 @@ class Deployer {
             await this.store.xgroup('CREATE', stream, group, '$', 'MKSTREAM');
         }
         catch (err) {
-            this.store.logger.info('consumer-group-exists', { stream, group });
+            this.store.logger.info('router-stream-group-exists', { stream, group });
         }
     }
 }

package/build/services/durable/client.js

@@ -73,7 +73,11 @@ class ClientService {
                         1000,
                 };
                 const context = { metadata: { trc, spn }, data: {} };
-                const jobId = await hotMeshClient.pub(`${options.namespace ?? factory_1.APP_ID}.execute`, payload, context, { search: options?.search?.data, marker: options?.marker });
+                const jobId = await hotMeshClient.pub(`${options.namespace ?? factory_1.APP_ID}.execute`, payload, context, {
+                    search: options?.search?.data,
+                    marker: options?.marker,
+                    expired: options?.expired,
+                });
                 return new handle_1.WorkflowHandleService(hotMeshClient, workflowTopic, jobId);
             },
             /**

package/build/services/durable/worker.js

@@ -86,7 +86,7 @@ class WorkerService {
         }
     }
     async run() {
-        this.workflowRunner.engine.logger.info('WorkerService is running');
+        this.workflowRunner.engine.logger.info('durable-worker-running');
     }
     async initActivityWorker(config, activityTopic) {
         const redisConfig = {

package/build/services/engine/index.d.ts

@@ -47,7 +47,7 @@ declare class EngineService {
     initStoreChannel(store: RedisClient): Promise<void>;
     initSubChannel(sub: RedisClient): Promise<void>;
     initStreamChannel(stream: RedisClient): Promise<void>;
-    initRouter(config: HotMeshConfig): Router;
+    initRouter(config: HotMeshConfig): Promise<Router>;
     getVID(vid?: AppVID): Promise<AppVID>;
     setCacheMode(cacheMode: CacheMode, untilVersion: string): void;
     routeToSubscribers(topic: string, message: JobOutput): Promise<void>;

package/build/services/engine/index.js

@@ -40,7 +40,7 @@ class EngineService {
             await instance.initStoreChannel(config.engine.store);
             await instance.initSubChannel(config.engine.sub);
             await instance.initStreamChannel(config.engine.stream);
-            instance.router = instance.initRouter(config);
+            instance.router = await instance.initRouter(config);
             instance.router.consumeMessages(instance.stream.mintKey(key_1.KeyType.STREAMS, { appId: instance.appId }), 'ENGINE', instance.guid, instance.processStreamMessage.bind(instance));
             instance.taskService = new task_1.TaskService(instance.store, logger);
             instance.exporter = new exporter_1.ExporterService(instance.appId, instance.store, logger);
@@ -82,7 +82,8 @@ class EngineService {
         }
         await this.stream.init(this.namespace, this.appId, this.logger);
     }
-    initRouter(config) {
+    async initRouter(config) {
+        const throttle = await this.store.getThrottleRate(':');
         return new router_1.Router({
             namespace: this.namespace,
             appId: this.appId,
@@ -90,6 +91,7 @@ class EngineService {
             role: stream_1.StreamRole.ENGINE,
             reclaimDelay: config.engine.reclaimDelay,
             reclaimCount: config.engine.reclaimCount,
+            throttle,
         }, this.stream, this.store, this.logger);
     }
     async getVID(vid) {
@@ -114,7 +116,7 @@ class EngineService {
         }
     }
     setCacheMode(cacheMode, untilVersion) {
-        this.logger.info(`engine-rule-cache-updated`, {
+        this.logger.info(`engine-cache-updated`, {
             mode: cacheMode,
             until: untilVersion,
         });
@@ -135,7 +137,12 @@ class EngineService {
         this.taskService.processTimeHooks(this.hookTime.bind(this));
     }
     async throttle(delayInMillis) {
-        this.router.setThrottle(delayInMillis);
+        try {
+            this.router.setThrottle(delayInMillis);
+        }
+        catch (e) {
+            this.logger.error('engine-throttle-error', { error: e });
+        }
     }
     // ************* METADATA/MODEL METHODS *************
     async initActivity(topic, data = {}, context) {
@@ -220,7 +227,7 @@ class EngineService {
     }
     // ****************** STREAM RE-ENTRY POINT *****************
     async processStreamMessage(streamData) {
-        this.logger.debug('engine-process-stream-message', {
+        this.logger.debug('engine-process', {
             jid: streamData.metadata.jid,
             gid: streamData.metadata.gid,
             dad: streamData.metadata.dad,
@@ -279,7 +286,7 @@ class EngineService {
             const activityHandler = (await this.initActivity(`.${streamData.metadata.aid}`, streamData.data, context));
             await activityHandler.processEvent(streamData.status, streamData.code, 'output');
         }
-        this.logger.debug('engine-process-stream-message-end', {
+        this.logger.debug('engine-process-end', {
             jid: streamData.metadata.jid,
             gid: streamData.metadata.gid,
             aid: streamData.metadata.aid,

package/build/services/hotmesh/index.js

@@ -11,6 +11,7 @@ const logger_1 = require("../logger");
 const quorum_1 = require("../quorum");
 const router_1 = require("../router");
 const worker_1 = require("../worker");
+const enums_1 = require("../../modules/enums");
 class HotMeshService {
     constructor() {
         this.engine = null;
@@ -94,9 +95,19 @@ class HotMeshService {
         return await this.quorum?.rollCall(delay);
     }
     async throttle(options) {
+        let throttle;
+        if (options.throttle === -1) {
+            throttle = enums_1.MAX_DELAY;
+        }
+        else {
+            throttle = options.throttle;
+        }
+        if (!Number.isInteger(throttle) || throttle < 0 || throttle > enums_1.MAX_DELAY) {
+            throw new Error(`Throttle must be a non-negative integer and not exceed ${enums_1.MAX_DELAY} ms; send -1 to throttle indefinitely`);
+        }
         const throttleMessage = {
             type: 'throttle',
-            throttle: options.throttle,
+            throttle: throttle,
         };
         if (options.guid) {
             throttleMessage.guid = options.guid;
@@ -104,6 +115,7 @@ class HotMeshService {
         else if (options.topic) {
             throttleMessage.topic = options.topic;
         }
+        await this.engine.store.setThrottleRate(throttleMessage);
         return await this.quorum?.pub(throttleMessage);
     }
     // ************* COMPILER METHODS *************

package/build/services/router/index.d.ts

@@ -42,6 +42,8 @@ declare class Router {
     customSleep(): Promise<void>;
     consumeMessages(stream: string, group: string, consumer: string, callback: (streamData: StreamData) => Promise<StreamDataResponse | void>): Promise<void>;
     isStreamMessage(result: any): boolean;
+    isPaused(): boolean;
+    isStopped(group: string, consumer: string, stream: string): boolean;
     consumeOne(stream: string, group: string, id: string, message: string[], callback: (streamData: StreamData) => Promise<StreamDataResponse | void>): Promise<void>;
     execStreamLeg(input: StreamData, stream: string, id: string, callback: (streamData: StreamData) => Promise<StreamDataResponse | void>): Promise<StreamData>;
     ackAndDelete(stream: string, group: string, id: string): Promise<void>;

package/build/services/router/index.js

@@ -22,6 +22,7 @@ class Router {
         this.topic = config.topic;
         this.stream = stream;
         this.store = store;
+        this.throttle = config.throttle;
         this.reclaimDelay = config.reclaimDelay || enums_1.HMSH_XCLAIM_DELAY_MS;
         this.reclaimCount = config.reclaimCount || enums_1.HMSH_XCLAIM_COUNT;
         this.logger = logger;
@@ -38,7 +39,7 @@ class Router {
             await this.store.xgroup('CREATE', stream, group, '$', 'MKSTREAM');
         }
         catch (err) {
-            this.logger.debug('consumer-group-exists', { stream, group });
+            this.logger.debug('router-stream-group-exists', { stream, group });
         }
     }
     async publishMessage(topic, streamData, multi) {
@@ -81,25 +82,31 @@ class Router {
         });
     }
     async consumeMessages(stream, group, consumer, callback) {
-        this.logger.info(`stream-consumer-starting`, { group, consumer, stream });
+        this.logger.info(`router-stream-starting`, { group, consumer, stream });
         Router.instances.add(this);
         this.shouldConsume = true;
         await this.createGroup(stream, group);
         let lastCheckedPendingMessagesAt = Date.now();
         async function consume() {
             await this.customSleep();
-            if (!this.shouldConsume) {
-                this.logger.info(`stream-consumer-stopping`, {
-                    group,
-                    consumer,
-                    stream,
-                });
+            if (this.isStopped(group, consumer, stream)) {
+                return;
+            }
+            else if (this.isPaused()) {
+                setImmediate(consume.bind(this));
                 return;
             }
             try {
                 //randomizer that asymptotes at 150% of `HMSH_BLOCK_TIME_MS`
                 const streamDuration = enums_1.HMSH_BLOCK_TIME_MS + Math.round(enums_1.HMSH_BLOCK_TIME_MS * Math.random());
                 const result = await this.stream.xreadgroup('GROUP', group, consumer, 'BLOCK', streamDuration, 'STREAMS', stream, '>');
+                if (this.isStopped(group, consumer, stream)) {
+                    return;
+                }
+                else if (this.isPaused()) {
+                    setImmediate(consume.bind(this));
+                    return;
+                }
                 if (this.isStreamMessage(result)) {
                     const [[, messages]] = result;
                     for (const [id, message] of messages) {
@@ -119,7 +126,7 @@ class Router {
             }
             catch (err) {
                 if (this.shouldConsume && process.env.NODE_ENV !== 'test') {
-                    this.logger.error(`stream-consume-message-error`, {
+                    this.logger.error(`router-stream-error`, {
                         err,
                         stream,
                         group,
@@ -136,8 +143,21 @@ class Router {
     isStreamMessage(result) {
         return Array.isArray(result) && Array.isArray(result[0]);
     }
+    isPaused() {
+        return this.throttle === enums_1.MAX_DELAY;
+    }
+    isStopped(group, consumer, stream) {
+        if (!this.shouldConsume) {
+            this.logger.info(`router-stream-stopped`, {
+                group,
+                consumer,
+                stream,
+            });
+        }
+        return !this.shouldConsume;
+    }
     async consumeOne(stream, group, id, message, callback) {
-        this.logger.debug(`stream-consume-one`, { group, stream, id });
+        this.logger.debug(`stream-read-one`, { group, stream, id });
         const [err, input] = this.parseStreamData(message[1]);
         let output;
         let telemetry;
@@ -151,14 +171,14 @@ class Router {
             this.errorCount = 0;
         }
         catch (err) {
-            this.logger.error(`stream-consume-one-error`, { group, stream, id, err });
+            this.logger.error(`stream-read-one-error`, { group, stream, id, err });
             telemetry.setStreamError(err.message);
         }
         const messageId = await this.publishResponse(input, output);
         telemetry.setStreamAttributes({ 'app.worker.mid': messageId });
         await this.ackAndDelete(stream, group, id);
         telemetry.endStreamSpan();
-        this.logger.debug(`stream-consume-one-end`, { group, stream, id });
+        this.logger.debug(`stream-read-one-end`, { group, stream, id });
     }
     async execStreamLeg(input, stream, id, callback) {
         let output;
@@ -293,7 +313,7 @@ class Router {
     }
     async stopConsuming() {
         this.shouldConsume = false;
-        this.logger.info(`stream-consumer-stopping`, this.topic ? { topic: this.topic } : undefined);
+        this.logger.info(`router-stream-stopping`, this.topic ? { topic: this.topic } : undefined);
         this.cancelThrottle();
     }
     cancelThrottle() {
@@ -303,8 +323,10 @@ class Router {
         this.resetThrottleState();
     }
     setThrottle(delayInMillis) {
-        if (!Number.isInteger(delayInMillis) || delayInMillis < 0) {
-            throw new Error('Throttle must be a non-negative integer');
+        if (!Number.isInteger(delayInMillis) ||
+            delayInMillis < 0 ||
+            delayInMillis > enums_1.MAX_DELAY) {
+            throw new Error(`Throttle must be a non-negative integer and not exceed ${enums_1.MAX_DELAY} ms; send -1 to throttle indefinitely`);
         }
         const wasDecreased = delayInMillis < this.throttle;
         this.throttle = delayInMillis;