rascal 13.0.5 → 13.1.2

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Change Log
 
+## 13.1.2
+- Fixed various issues when queue names contained period characters. Reported in https://github.com/guidesmiths/rascal/issues/166
+
+## 13.1.1
+- Moved setMaxListeners to createConnection task to suppress misleading 'Possible EventEmitter memory leak detected' warning. See https://github.com/guidesmiths/rascal/issues/164 for more details.
+
+## 13.1.0
+- Fixed bug where Rascal could wait indefinitely for channels to be destroyed if shutdown was called following a heartbeat timeout. See https://github.com/guidesmiths/rascal/issues/158 for more details.
+
+## 13.0.6
+- Fixed bug where Rascal attempted to remove a listener from a nulled connection and crashed.
+
 ## 13.0.5
 - Set channel pool acquireTimeoutMillis in default configuration - thanks @matej-prokop
 - Add snyk package health badge

package/README.md

@@ -39,13 +39,16 @@ Rascal seeks to either solve these problems, make them easier to deal with or br
 * TDD support
 
 ## Concepts
-Rascal extends the existing [RabbitMQ Concepts](https://www.rabbitmq.com/tutorials/amqp-concepts.html) of Brokers, Vhosts, Exchanges Queues, Channels and Connections with with two new ones
+Rascal extends the existing [RabbitMQ Concepts](https://www.rabbitmq.com/tutorials/amqp-concepts.html) of Brokers, Vhosts, Exchanges, Queues, Channels and Connections with with two new ones
 
 1. Publications
 1. Subscriptions
 
 A **publication** is a named configuration for publishing a message, including the destination queue or exchange, routing configuration, encryption profile and reliability guarantees, message options, etc. A **subscription** is a named configuration for consuming messages, including the source queue, encryption profile, content encoding, delivery options (e.g. acknowledgement handling and prefetch), etc. These must be [configured](#configuration) and supplied when creating the Rascal broker. After the broker has been created the subscriptions and publications can be retrivied from the broker and used to publish and consume messages.
 
+### Special Note
+RabbitMQ 3.8.0 introduced [quorum queues](https://www.rabbitmq.com/quorum-queues.html). Although quorum queues may not be suitable in all situations, they provide [poison message handling](https://www.rabbitmq.com/quorum-queues.html#poison-message-handling) without the need for an external [redelivery counter](https://github.com/guidesmiths/rascal#dealing-with-redeliveries) and offer better data safety in the event of a network partition. You can read more about them [here](https://www.cloudamqp.com/blog/reasons-you-should-switch-to-quorum-queues.html) and [here](https://blog.rabbitmq.com/posts/2020/06/quorum-queues-local-delivery).
+
 ## Examples
 
 ### Async/Await
@@ -1135,7 +1138,7 @@ Rascal can be configured to automatically decrypt inbound messages.
 Any message that was published using the "well-known-v1" encryption profile will be automatically decrypted by the subscriber.
 
 #### Dealing With Redeliveries
-If your app crashes before acknowledging a message, the message will be rolled back. It is common for node applications to automatically restart, however if the crash was caused by something in the message content, it will crash and restart indefinitely, thrashing the host. Unfortunately RabbitMQ doesn't allow you to limit the number of redeliveries per message or provide a redelivery count. For this reason subscribers can be configured with a redelivery counter and will update the ```message.properties.headers.rascal.redeliveries``` header with the number of hits. If the number of redeliveries exceeds the subscribers limit, the subscriber will emit a "redeliveries_exceeded" event, and can be handled by your application. e.g.
+If your app crashes before acknowledging a message, the message will be rolled back. It is common for node applications to automatically restart, however if the crash was caused by something in the message content, it will crash and restart indefinitely, thrashing the host. Prior to version 3.8.0, RabbitMQ didn't allow you to limit the number of redeliveries per message or provide a redelivery count. This is now possible using [quorum queues](https://www.rabbitmq.com/quorum-queues.html#poison-message-handling), but for those on older versions, or in situations where quorum queues are not appropriate, subscribers can be configured with a redelivery counter and will update the ```message.properties.headers.rascal.redeliveries``` header with the number of hits. If the number of redeliveries exceeds the subscribers limit, the subscriber will emit a "redeliveries_exceeded" event, and can be handled by your application. e.g.
 
 ```json
 "subscriptions": {

package/examples/simple/config.js

@@ -7,7 +7,9 @@ module.exports = {
         }
       },
       connection: {
-        heartbeat: 1,
+        options: {
+          heartbeat: 10
+        },
         socketOptions: {
           timeout: 1000
         }

package/lib/amqp/SubscriberError.js

@@ -64,7 +64,7 @@ module.exports = function SubscriptionRecovery(broker, vhost) {
         debug('Republishing message: %s', message.properties.messageId);
 
         const originalQueue = _.get(message, 'properties.headers.rascal.originalQueue');
-        const republished = _.get(message, format('properties.headers.rascal.recovery.%s.republished', originalQueue), 0);
+        const republished = _.get(message, ['properties', 'headers', 'rascal', 'recovery', originalQueue, 'republished'], 0);
 
         if (strategyConfig.attempts && strategyConfig.attempts <= republished) {
           debug('Skipping recovery - message: %s has already been republished %d times.', message.properties.messageId, republished);
@@ -72,14 +72,14 @@ module.exports = function SubscriptionRecovery(broker, vhost) {
         }
 
         const publishOptions = _.cloneDeep(message.properties);
-        _.set(publishOptions, format('headers.rascal.recovery.%s.republished', originalQueue), republished + 1);
+        _.set(publishOptions, ['headers', 'rascal', 'recovery', originalQueue, 'republished'], republished + 1);
         _.set(publishOptions, 'headers.rascal.originalExchange', message.fields.exchange);
         _.set(publishOptions, 'headers.rascal.originalRoutingKey', message.fields.routingKey);
         _.set(publishOptions, 'headers.rascal.error.message', _.truncate(err.message, { length: 1024 }));
         _.set(publishOptions, 'headers.rascal.error.code', err.code);
         _.set(publishOptions, 'headers.rascal.restoreRoutingHeaders', _.has(strategyConfig, 'restoreRoutingHeaders') ? strategyConfig.restoreRoutingHeaders : true);
 
-        if (strategyConfig.immediateNack) _.set(publishOptions, format('headers.rascal.recovery.%s.immediateNack', originalQueue), true);
+        if (strategyConfig.immediateNack) _.set(publishOptions, ['headers', 'rascal', 'recovery', originalQueue, 'immediateNack'], true);
 
         vhost.getConfirmChannel((err, publisherChannel) => {
 
@@ -105,7 +105,7 @@ module.exports = function SubscriptionRecovery(broker, vhost) {
         debug('Forwarding message: %s', message.properties.messageId);
 
         const originalQueue = _.get(message, 'properties.headers.rascal.originalQueue');
-        const forwarded = _.get(message, format('properties.headers.rascal.recovery.%s.forwarded', originalQueue), 0);
+        const forwarded = _.get(message, ['properties', 'headers', 'rascal', 'recovery', originalQueue, 'forwarded'], 0);
 
         if (strategyConfig.attempts && strategyConfig.attempts <= forwarded) {
           debug('Skipping recovery - message: %s has already been forwarded %d times.', message.properties.messageId, forwarded);
@@ -117,7 +117,7 @@ module.exports = function SubscriptionRecovery(broker, vhost) {
 
         const forwardOverrides = _.cloneDeep(strategyConfig.options) || {};
         _.set(forwardOverrides, 'restoreRoutingHeaders', _.has(strategyConfig, 'restoreRoutingHeaders') ? strategyConfig.restoreRoutingHeaders : true);
-        _.set(forwardOverrides, format('options.headers.rascal.recovery.%s.forwarded', originalQueue), forwarded + 1);
+        _.set(forwardOverrides, ['options', 'headers', 'rascal', 'recovery', originalQueue, 'forwarded'], forwarded + 1);
         _.set(forwardOverrides, 'options.headers.rascal.error.message', _.truncate(err.message, { length: 1024 }));
         _.set(forwardOverrides, 'options.headers.rascal.error.code', err.code);
 

package/lib/amqp/Subscription.js

@@ -198,8 +198,7 @@ function Subscription(broker, vhost, config, counter) {
   }
 
   function immediateNack(message) {
-    if (_.get(message, format('properties.headers.rascal.recovery.%s.immediateNack', message.properties.headers.rascal.originalQueue))) return true;
-    if (_.get(message, format('properties.headers.rascal.recovery.%s.immediateNack', message.properties.headers.rascal.originalQueue))) return true;
+    if (_.get(message, ['properties', 'headers', 'rascal', 'recovery', message.properties.headers.rascal.originalQueue, 'immediateNack'])) return true;
     return false;
   }
 

package/lib/amqp/Vhost.js

@@ -34,6 +34,7 @@ function Vhost(config) {
   let timer = backoff({});
   let paused = true;
   let shuttingDown = false;
+  let reconnectTimeout;
 
   this.name = config.name;
   this.connectionIndex = 0;
@@ -41,6 +42,11 @@ function Vhost(config) {
   pauseChannelAllocation();
 
   this.init = function(next) {
+    if (shuttingDown) {
+      debug('Aborting initialisation. Vhost %s is shutting down.', self.name);
+      return next();
+    }
+
     debug('Initialising vhost: %s', self.name);
     pauseChannelAllocation();
 
@@ -68,7 +74,7 @@ function Vhost(config) {
   };
 
   this.forewarn = function(next) {
-    debug('Forwarning vhost: %s about impending shutdown', self.name);
+    debug('Forewarning vhost: %s about impending shutdown', self.name);
     pauseChannelAllocation();
     shuttingDown = true;
     channelCreator.resume();
@@ -77,6 +83,7 @@ function Vhost(config) {
 
   this.shutdown = function(next) {
     debug('Shutting down vhost: %s', self.name);
+    clearTimeout(reconnectTimeout);
     pauseChannelAllocation();
     drainChannelPools((err) => {
       if (err) return next(err);
@@ -144,7 +151,7 @@ function Vhost(config) {
   };
 
   this.borrowChannel = function(next) {
-    if (!regularChannelPool) return next(new Error(format('VHost: %s must be initialised before you can borrow a channel', self.name)));
+    if (!regularChannelPool) return next(new Error(format('Vhost: %s must be initialised before you can borrow a channel', self.name)));
     regularChannelPool.borrow(next);
   };
 
@@ -159,7 +166,7 @@ function Vhost(config) {
   };
 
   this.borrowConfirmChannel = function(next) {
-    if (!confirmChannelPool) return next(new Error(format('VHost: %s must be initialised before you can borrow a confirm channel', self.name)));
+    if (!confirmChannelPool) return next(new Error(format('Vhost: %s must be initialised before you can borrow a confirm channel', self.name)));
     confirmChannelPool.borrow(next);
   };
 
@@ -192,19 +199,14 @@ function Vhost(config) {
         return new Promise((resolve, reject) => {
           debug('Creating pooled %s channel for vhost: %s', mode, config.name);
           createChannelWhenInitialised(options.confirm, (err, channel) => {
-            if (err) {
-              debug('Error creating pooled %s channel for vhost: %s. %s', mode, config.name, err.message);
-              return deferRejection(reject, err);
-            }
-            if (!channel) {
-              debug('Error creating pooled %s channel for vhost: %s. Vhost is shutting down', mode, config.name);
-              return deferRejection(reject, new Error('Vhost is shutting down'));
-            }
+            if (err) return deferRejection(reject, err);
+            if (!channel) return deferRejection(reject, new Error('Vhost is shutting down'));
             const destroyChannel = _.once(() => {
+              debug('Destroying %s channel: %s for vhost: %s due to error or close event', mode, channel._rascal_id, config.name);
               channel._rascal_closed = true;
               if (pool.isBorrowedResource(channel)) {
                 pool.destroy(channel).catch((err) => {
-                  debug('Error destroying pooled %s channel: %s for vhost: %s. %s', mode, config.name, err.message);
+                  debug('Error destroying %s channel: %s for vhost: %s. %s', mode, channel._rascal_id, config.name, err.message);
                 });
               }
             });
@@ -215,12 +217,24 @@ function Vhost(config) {
         });
       },
       destroy(channel) {
-        return new Promise((resolve) => {
-          debug('Destroying channel: %s', channel._rascal_id);
+        return new Promise((resolve, reject) => {
+          debug('Destroying %s channel: %s for vhost: %s', mode, channel._rascal_id, config.name);
           if (channel._rascal_closed) return resolve();
-          channel.close(() => {
+          channel.removeAllListeners();
+          channel.on('error', reject);
+          const closeChannelCb = (err) => {
+            if (err) return reject(err);
             resolve();
-          });
+          };
+          // When a connection drops it may take a while for the heartbeat protocol or TCP keep alives to notice
+          // Consequently a publication using confirm channels may timeout and attempt to close the channel
+          // before Rascal notices that the connection died. In this circumstance the channel close command
+          // will never receive a response from the broker, and the callback will never yield.
+          const once = _.once(closeChannelCb);
+          setTimeoutUnref(() => {
+            once(new Error(format('Timeout after %dms closing %s channel: %s for vhost: %s', options.pool.destroyTimeoutMillis, mode, channel._rascal_id, config.name)));
+          }, 1000);
+          channel.close(once);
         });
       },
       validate(channel) {
@@ -302,6 +316,12 @@ function Vhost(config) {
 
     debug('Creating %s channel pool %o', mode, options.pool);
     pool = genericPool.createPool(factory, options.pool);
+    pool.on('factoryCreateError', (err) => {
+      debug('Create error emitted by %s channel pool: %s', mode, err.message);
+    });
+    pool.on('factoryDestroyError', (err) => {
+      debug('Destroy error emitted by %s channel pool: %s', mode, err.message);
+    });
 
     poolQueue = async.queue((__, next) => {
       pool.acquire().then((channel) => {
@@ -325,19 +345,24 @@ function Vhost(config) {
   function createChannelWhenInitialised(confirm, next) {
     if (connection) return createChannel(confirm, next);
     debug('Vhost: %s is not initialised. Deferring channel creation', self.name);
-    self.once('vhost_initialised', () => {
-      debug('Vhost: %s was initialised. Resuming channel creation', self.name);
-      createChannel(confirm, next);
+    setTimeoutUnref(() => {
+      self.removeListener('vhost_initialised', onVhostInitialised);
+      next(new Error('Timedout acquiring channel'), 5000);
     });
+    function onVhostInitialised() {
+      debug('Vhost: %s was initialised. Resuming channel creation', self.name);
+      createChannel(confirm, next);      
+    }
+    self.once('vhost_initialised', onVhostInitialised);
   }
 
   function createChannel(confirm, next) {
 
     if (shuttingDown) {
-      debug('Ignoring create channel request. VHost is shutting down.');
+      debug('Ignoring create channel request. Vhost: %s is shutting down.', self.name);
       return next();
     }
-    if (!connection) return next(new Error(format('VHost: %s must be initialised before you can create a channel', self.name)));
+    if (!connection) return next(new Error(format('Vhost: %s must be initialised before you can create a channel', self.name)));
 
     // Same problem as https://github.com/guidesmiths/rascal/issues/17
     const once = _.once(next);
@@ -358,8 +383,8 @@ function Vhost(config) {
 
     function callback(err, channel) {
       invocations++;
-      connection.removeListener('close', closeHandler);
-      connection.removeListener('error', errorHandler);
+      connection && connection.removeListener('close', closeHandler);
+      connection && connection.removeListener('error', errorHandler);
       if (err) {
         debug('Error creating channel: %s from %s: %s', channelId, connectionConfig.loggableUrl, err.message);
         return once(err);
@@ -367,7 +392,6 @@ function Vhost(config) {
 
       channel._rascal_id = channelId;
       channel.connection._rascal_id = connection._rascal_id;
-      channel.connection.setMaxListeners(0);
       debug('Created %s channel: %s from connection: %s', getChannelMode(confirm), channel._rascal_id, connection._rascal_id);
 
       // See https://github.com/squaremo/amqp.node/issues/388
@@ -442,7 +466,7 @@ function Vhost(config) {
       if (!err) return;
       const delay = timer.next();
       debug('Will attempt reconnection in in %dms', delay);
-      setTimeoutUnref(handleConnectionError.bind(null, borked, config, err), delay);
+      reconnectTimeout = setTimeoutUnref(handleConnectionError.bind(null, borked, config, err), delay);
     });
   }
 }

package/lib/amqp/tasks/createConnection.js

@@ -65,6 +65,8 @@ function connect(connectionConfig, cb) {
       return connection.close();
     }
 
+    connection.setMaxListeners(0);
+
     once(null, connection);
   });
 }

package/lib/config/baseline.js

@@ -11,6 +11,7 @@ module.exports = {
           rejectionDelayMillis: 1000,
           testOnBorrow: true,
           acquireTimeoutMillis: 15000,
+          destroyTimeoutMillis: 1000,
         },
         confirmPool: {
           autostart: false,
@@ -21,6 +22,7 @@ module.exports = {
           rejectionDelayMillis: 1000,
           testOnBorrow: true,
           acquireTimeoutMillis: 15000,
+          destroyTimeoutMillis: 1000,          
         },
       },
       connectionStrategy: 'random',

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "rascal",
-  "version": "13.0.5",
+  "version": "13.1.2",
   "description": "A config driven wrapper for amqplib supporting multi-host connections, automatic error recovery, redelivery flood protection, transparent encryption / decryption,  channel pooling and publication timeouts",
   "main": "index.js",
   "dependencies": {