@yroshcha/node-red-contrib-redis-full 1.0.2 → 1.0.6

package/CHANGELOG.md

@@ -2,6 +2,37 @@
 
 All notable changes to this project are documented in this file.
 
+## 1.0.6 — 2026-08-11
+
+### Fixed
+
+- `redis xreadgroup` now restores its consumer group with `MKSTREAM` when a running consumer receives `NOGROUP`, including a deleted or restarted DLQ stream.
+
+## 1.0.5 — 2026-08-11
+
+### Fixed
+
+- `redis xreadgroup` now raises the dedicated connection command timeout to cover its configured `BLOCK` duration, preventing a valid long poll from being reported as a timeout.
+- Stream consumer timing and backoff settings are bounded at runtime even when a malformed flow configuration bypasses editor validation.
+- `redis sub` now rejects an empty channel list instead of remaining indefinitely in a starting state.
+- `redis xadd` now reports an empty fields object before sending an invalid command to Redis.
+
+### Changed
+
+- `redis scan` now has a 10,000-item per-message safety limit by default. It exposes `msg.scanComplete` and `msg.scanTruncated` so a flow cannot mistake a bounded result for a complete scan.
+
+## 1.0.4 — 2026-08-11
+
+### Fixed
+
+- `redis sub` now retries its initial subscription up to five times with backoff and jitter instead of remaining permanently unsubscribed after one transient timeout.
+
+## 1.0.3 — 2026-08-11
+
+### Fixed
+
+- `redis xreadgroup` now retries dedicated-connection startup and consumer-group initialisation up to five times with backoff and jitter instead of stopping after one transient timeout.
+
 ## 1.0.2 — 2026-08-10
 
 Documentation-only patch release.

package/README.md

@@ -45,7 +45,7 @@ For a local archive, use `npm install /path/to/yroshcha-node-red-contrib-redis-f
 
 ## Release status
 
-**`1.0.2` is the current stable release.** See [CHANGELOG.md](CHANGELOG.md) for release notes and compatibility information.
+**`1.0.6` is the current stable release.** See [CHANGELOG.md](CHANGELOG.md) for release notes and compatibility information.
 
 ## Production profile
 
@@ -61,7 +61,10 @@ Safe consumer defaults and important settings:
 - **Batch interval (ms)** sets the minimum delay between emitted batches regardless of queue depth. For one batch of up to 50 entries per second, use `COUNT = 50` and `Batch interval = 1000`.
 - **Max deliveries = 5.** After the limit is exceeded, an entry is atomically moved to `<stream>:dlq` and acknowledged. In Redis Cluster, source and DLQ keys must share a hash tag, for example `orders:{eu}` and `orders:{eu}:dlq`.
 - **Start paused (wait for resume)** makes a consumer create/verify its group without calling `XREADGROUP`. It takes no work until a control-plane `resume` request arrives; use it for controlled pod startup after readiness checks.
+- Startup initialisation retries a failed dedicated connection, `XGROUP CREATE`, and the initial `SUBSCRIBE` up to **five times** with exponential backoff and jitter. A transient Redis/TLS/DNS delay does not permanently stop a new consumer or subscriber.
+- If a consumer group disappears after startup (for example after a Redis restart, `XGROUP DESTROY`, or deleting an empty DLQ stream), `redis xreadgroup` recreates it with `MKSTREAM` and resumes.
 - `redis xautoclaim` limits each manual recovery run through `Run limit` (default 1000), avoiding oversized Node-RED payloads.
+- `redis scan` limits one emitted Node-RED message to **10,000** items by default. It sets `msg.scanTruncated = true` when that safety limit is reached; raise `Result limit` / `msg.maxResults` only after sizing the Node-RED heap for the result.
 - `MAXLEN` in `redis xadd` remains blocked until **Allow unsafe MAXLEN trim** is explicitly confirmed. Trimming can remove a payload still represented in a PEL.
 
 ## Production readiness boundaries

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@yroshcha/node-red-contrib-redis-full",
-  "version": "1.0.2",
+  "version": "1.0.6",
   "description": "Production-oriented Redis palette for Node-RED: generic commands, Pub/Sub, scans, transactions, and Redis Streams consumer groups.",
   "keywords": ["node-red", "redis", "redis-streams", "pubsub", "xadd", "xreadgroup", "ioredis"],
   "license": "MIT",

package/redis.html

@@ -273,7 +273,8 @@
       name: { value: '' },
       server: { type: 'yroshcha-redis-config', required: true },
       scanType: { value: 'SCAN' },
-      count: { value: 100, validate: RED.validators.number() }
+      count: { value: 100, validate: RED.validators.number() },
+      maxResults: { value: 10000, validate: RED.validators.number() }
     },
     inputs: 1,
     outputs: 1,
@@ -303,6 +304,10 @@
     <label for="node-input-count">COUNT hint</label>
     <input type="text" id="node-input-count">
   </div>
+  <div class="form-row">
+    <label for="node-input-maxResults" title="Maximum items returned in one Node-RED message">Result limit</label>
+    <input type="text" id="node-input-maxResults">
+  </div>
   <div class="form-row">
     <label for="node-input-name"><i class="fa fa-tag"></i> Name</label>
     <input type="text" id="node-input-name">
@@ -310,8 +315,8 @@
 </script>
 
 <script type="text/html" data-help-name="yroshcha-redis-scan">
-  <p>Full cursor-based non-blocking scan. HSCAN/SSCAN/ZSCAN require <code>msg.key</code>.</p>
-  <p>Output: <code>msg.payload</code> is an array; HSCAN/ZSCAN return <code>{member, value}</code>.</p>
+  <p>Cursor-based non-blocking scan. HSCAN/SSCAN/ZSCAN require <code>msg.key</code>. To protect the Node-RED heap, one invocation returns at most <b>Result limit</b> items (default 10000); override it with <code>msg.maxResults</code>.</p>
+  <p>Output: <code>msg.payload</code> is an array; HSCAN/ZSCAN return <code>{member, value}</code>. <code>msg.scanComplete</code> is false and <code>msg.scanTruncated</code> is true when the result limit was reached.</p>
 </script>
 
 <script type="text/javascript">

package/redis.js

@@ -173,6 +173,10 @@ module.exports = function (RED) {
     return obj;
   }
 
+  function isNoGroupError(err) {
+    return String(err && err.message ? err.message : err).includes('NOGROUP');
+  }
+
   // ---------------------------------------------------------------------
   // yroshcha-redis-command — БУДЬ-ЯКА команда Redis через ioredis .call().
   // "block" форсує окреме з'єднання (для BLPOP/BRPOP/WAIT тощо).
@@ -244,8 +248,14 @@ module.exports = function (RED) {
     node.mode = config.mode === 'psubscribe' ? 'psubscribe' : 'subscribe';
     node.channels = (config.channels || '').split(',').map((s) => s.trim()).filter(Boolean);
 
+    if (!node.channels.length) {
+      node.error('At least one Redis channel or pattern is required');
+      return;
+    }
+
     const client = node.server.getDedicatedClient();
     let subscribed = [];
+    let stopped = false;
 
     const eventName = node.mode === 'psubscribe' ? 'pmessage' : 'message';
     client.on(eventName, (a, b, c) => {
@@ -271,10 +281,40 @@ module.exports = function (RED) {
       node.status({ fill: 'green', shape: 'dot', text: `listening (${subscribed.length})` });
     }
 
-    doSubscribe(node.channels).catch((err) => {
-      node.status({ fill: 'red', shape: 'ring', text: 'subscribe failed' });
-      node.error(`Redis subscribe failed: ${err.message}`);
-    });
+    async function startSubscription() {
+      // SUBSCRIBE is idempotent for an already subscribed channel. Retrying it
+      // is therefore safe when Redis accepted the command but its reply was
+      // lost to a timeout during pod startup.
+      const maxStartupRetries = 5;
+      let retries = 0;
+      let delayMs = 500;
+
+      while (!stopped) {
+        try {
+          await doSubscribe(node.channels);
+          return;
+        } catch (err) {
+          if (retries >= maxStartupRetries) {
+            node.status({ fill: 'red', shape: 'ring', text: 'subscribe failed' });
+            node.error(`Redis subscribe failed after ${maxStartupRetries} retries: ${err.message}`);
+            return;
+          }
+
+          const jitter = Math.random() * delayMs * 0.3;
+          const delay = Math.min(delayMs, node.server.retryMaxDelay) + jitter;
+          retries += 1;
+          node.status({ fill: 'red', shape: 'ring', text: `subscribe retry ${retries}/${maxStartupRetries}` });
+          node.warn(
+            `Redis subscribe startup attempt ${retries}/${maxStartupRetries} failed: ${err.message}; ` +
+            `retrying in ${Math.round(delay)}ms`
+          );
+          await new Promise((resolve) => setTimeout(resolve, delay));
+          delayMs = Math.min(delayMs * 2, node.server.retryMaxDelay);
+        }
+      }
+    }
+
+    startSubscription();
 
     node.on('input', async function (msg, send, done) {
       try {
@@ -287,6 +327,7 @@ module.exports = function (RED) {
     });
 
     node.on('close', async function (done) {
+      stopped = true;
       try { await client.quit(); } catch (e) { /* ignore */ }
       node.status({});
       done();
@@ -346,14 +387,18 @@ module.exports = function (RED) {
     if (!node.server) return;
 
     node.scanType = config.scanType || 'SCAN';
-    node.defaultCount = parseInt(config.count, 10) || 100;
+    node.defaultCount = Math.max(1, parseInt(config.count, 10) || 100);
+    // A cursor scan is non-blocking for Redis, but collecting an unbounded
+    // result into one Node-RED message can still exhaust the Node.js heap.
+    node.defaultMaxResults = Math.max(1, parseInt(config.maxResults, 10) || 10000);
 
     node.on('input', async function (msg, send, done) {
       send = send || function () { node.send.apply(node, arguments); };
       try {
         const scanType = (msg.scanType || node.scanType).toUpperCase();
         const match = msg.match;
-        const count = msg.count !== undefined ? msg.count : node.defaultCount;
+        const count = Math.max(1, parseInt(msg.count, 10) || node.defaultCount);
+        const maxResults = Math.max(1, parseInt(msg.maxResults, 10) || node.defaultMaxResults);
         const client = node.server.getClient();
 
         let key = null;
@@ -381,14 +426,22 @@ module.exports = function (RED) {
           if (scanType === 'HSCAN' || scanType === 'ZSCAN') {
             for (let i = 0; i < elements.length; i += 2) {
               collected.push({ member: elements[i], value: elements[i + 1] });
+              if (collected.length >= maxResults) break;
             }
           } else {
-            collected.push(...elements);
+            collected.push(...elements.slice(0, maxResults - collected.length));
           }
-        } while (cursor !== '0');
+        } while (cursor !== '0' && collected.length < maxResults);
 
         msg.payload = collected;
-        node.status({ fill: 'green', shape: 'dot', text: `${collected.length} items` });
+        msg.scanComplete = cursor === '0';
+        msg.scanTruncated = !msg.scanComplete;
+        if (msg.scanTruncated) {
+          node.status({ fill: 'yellow', shape: 'ring', text: `${collected.length} items (limit)` });
+          node.warn(`Redis ${scanType} stopped at the configured ${maxResults}-item limit`);
+        } else {
+          node.status({ fill: 'green', shape: 'dot', text: `${collected.length} items` });
+        }
         send(msg);
         done();
       } catch (err) {
@@ -432,6 +485,8 @@ module.exports = function (RED) {
           fields.push(k, typeof v === 'object' ? JSON.stringify(v) : String(v));
         }
 
+        if (!fields.length) throw new Error('msg.payload must contain at least one field for XADD');
+
         const args = [streamKey];
         const maxlen = msg.maxlen !== undefined ? msg.maxlen : node.maxlen;
         const unsafeTrim = msg.allowUnsafeTrim === true || node.unsafeTrim;
@@ -477,8 +532,8 @@ module.exports = function (RED) {
       || `${process.env.HOSTNAME || RED.settings.get('flowfile') || 'nr'}-${node.id}`;
     // 100 amortises Redis/network round-trips while Max pending still limits
     // the total work allowed into Node-RED.
-    node.count = parseInt(config.count, 10) || 100;
-    node.blockMs = parseInt(config.blockMs, 10) || 5000;
+    node.count = Math.max(1, parseInt(config.count, 10) || 100);
+    node.blockMs = Math.max(1, parseInt(config.blockMs, 10) || 5000);
     node.readIntervalMs = Math.max(0, parseInt(config.readIntervalMs, 10) || 0);
     node.rateLimitPerSecond = Math.max(0, parseFloat(config.rateLimitPerSecond) || 0);
     node.batchWindowMs = Math.max(0, parseInt(config.batchWindowMs, 10) || 0);
@@ -508,9 +563,9 @@ module.exports = function (RED) {
     // Backoff/jitter при помилках циклу — щоб при масовому падінні Redis
     // (рестарт кластера, failover) весь HPA-пул не долбив reconnect
     // синхронно одним і тим же інтервалом ("thundering herd").
-    node.initialBackoffMs = parseInt(config.initialBackoffMs, 10) || 500;
-    node.maxBackoffMs = parseInt(config.maxBackoffMs, 10) || 30000;
-    node.backoffMultiplier = parseFloat(config.backoffMultiplier) || 2;
+    node.initialBackoffMs = Math.max(50, parseInt(config.initialBackoffMs, 10) || 500);
+    node.maxBackoffMs = Math.max(node.initialBackoffMs, parseInt(config.maxBackoffMs, 10) || 30000);
+    node.backoffMultiplier = Math.max(1, parseFloat(config.backoffMultiplier) || 2);
 
     // PEL-alert: періодична неблокуюча перевірка розміру pending list через
     // XPENDING на СПІЛЬНОМУ з'єднанні конфігу (не на blockingClient, щоб не
@@ -552,6 +607,52 @@ module.exports = function (RED) {
       }
     }
 
+    async function initializeBlockingClient() {
+      // A new pod can reach Node-RED before its Redis route, TLS handshake, or
+      // ElastiCache endpoint is ready. Do not permanently lose the consumer
+      // on one transient startup timeout. Five retries means six total tries.
+      const maxStartupRetries = 5;
+      let retries = 0;
+
+      while (!stopped) {
+        const candidate = node.server.getDedicatedClient({
+          // commandTimeout applies to each Redis command. It must be longer
+          // than BLOCK, otherwise a valid idle XREADGROUP is reported as a
+          // timeout when an operator chooses BLOCK above the config timeout.
+          commandTimeout: Math.max(node.server.commandTimeout, node.blockMs + 1000),
+          maxRetriesPerRequest: 1,
+          autoResendUnfulfilledCommands: false
+        });
+        candidate.on('error', (err) => {
+          node.status({ fill: 'red', shape: 'ring', text: 'redis error' });
+          node.error(`Redis stream-in error: ${err.message}`);
+        });
+
+        try {
+          await ensureGroup(candidate);
+          blockingClient = candidate;
+          return true;
+        } catch (err) {
+          try { candidate.disconnect(false); } catch (e) { /* ignore */ }
+          if (retries >= maxStartupRetries) {
+            throw new Error(`Startup failed after ${maxStartupRetries} retries: ${err.message}`);
+          }
+
+          const jitter = Math.random() * currentBackoffMs * 0.3;
+          const delay = Math.min(currentBackoffMs, node.maxBackoffMs) + jitter;
+          retries += 1;
+          node.status({ fill: 'red', shape: 'ring', text: `startup retry ${retries}/${maxStartupRetries}` });
+          node.warn(
+            `Redis stream-in startup attempt ${retries}/${maxStartupRetries} failed: ${err.message}; ` +
+            `retrying in ${Math.round(delay)}ms`
+          );
+          await new Promise((resolve) => setTimeout(resolve, delay));
+          currentBackoffMs = Math.min(currentBackoffMs * node.backoffMultiplier, node.maxBackoffMs);
+        }
+      }
+      return false;
+    }
+
     async function waitForReadInterval() {
       if (!node.readIntervalMs || !lastReadAt) return true;
       const waitMs = node.readIntervalMs - (Date.now() - lastReadAt);
@@ -802,18 +903,7 @@ module.exports = function (RED) {
     streamConsumers.set(node.id, node);
 
     async function loop() {
-      blockingClient = node.server.getDedicatedClient({
-        // XREADGROUP BLOCK must not stay zombie after a silent network split.
-        blockingTimeout: node.blockMs + node.server.commandTimeout + 1000,
-        maxRetriesPerRequest: 1,
-        autoResendUnfulfilledCommands: false
-      });
-      blockingClient.on('error', (err) => {
-        node.status({ fill: 'red', shape: 'ring', text: 'redis error' });
-        node.error(`Redis stream-in error: ${err.message}`);
-      });
-
-      await ensureGroup(blockingClient);
+      if (!await initializeBlockingClient()) return;
       node.status({
         fill: paused ? 'yellow' : 'green',
         shape: paused ? 'ring' : 'dot',
@@ -876,6 +966,22 @@ module.exports = function (RED) {
         } catch (err) {
           if (stopped) break;
 
+          // A Redis restart, a manual XGROUP DESTROY, or deletion of an empty
+          // DLQ stream can remove the group after this node has started.
+          // Recreate it before backing off; ensureGroup handles a concurrent
+          // creator through BUSYGROUP and is safe to call repeatedly.
+          if (isNoGroupError(err)) {
+            try {
+              await ensureGroup(blockingClient);
+              currentBackoffMs = node.initialBackoffMs;
+              node.status({ fill: 'yellow', shape: 'ring', text: 'consumer group restored' });
+              node.warn(`Redis stream consumer group restored: ${node.group}@${node.streamKey}`);
+              continue;
+            } catch (restoreErr) {
+              err = restoreErr;
+            }
+          }
+
           const jitter = Math.random() * currentBackoffMs * 0.3; // до +30%
           const delay = Math.min(currentBackoffMs, node.maxBackoffMs) + jitter;