rascal 17.0.2 → 18.0.1

package/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Change Log
 
+## 18.0.1
+
+- Removed console.log when the channel pool destroyed a channel
+- Add streams example
+
+## 18.0.0
+
+- Fixes https://github.com/onebeyond/rascal/issues/227 by requiring special characters to be URL encoded.
+- Consolidated broker and management url configuration logic
+
 ## 17.0.2
 
 - Update guidesmiths references to onebeyond.

package/README.md

@@ -47,9 +47,9 @@ Rascal extends the existing [RabbitMQ Concepts](https://www.rabbitmq.com/tutoria
 
 A **publication** is a named configuration for publishing a message, including the destination queue or exchange, routing configuration, encryption profile and reliability guarantees, message options, etc. A **subscription** is a named configuration for consuming messages, including the source queue, encryption profile, content encoding, delivery options (e.g. acknowledgement handling and prefetch), etc. These must be [configured](#configuration) and supplied when creating the Rascal broker. After the broker has been created the subscriptions and publications can be retrieved from the broker and used to publish and consume messages.
 
-### Breaking Changes in Rascal@14
+### Breaking Changes
 
-Rascal@14 waits for inflight messages to be acknowledged before closing subscriber channels. Prior to this version Rascal just waited an arbitrary amount of time. If your application does not acknowledge a message for some reason (quite likely in tests) calling `subscription.cancel`, `broker.unsubscribeAll`, `broker.bounce`, `broker.shutdown` or `broker.nuke` will wait indefinitely. You can specify a `closeTimeout` in your subscription config, however if this is exceeded the `subscription.cancel` and `broker.unsubscribeAll` methods will yield an error via callback or rejection, while the `broker.bounce`, `broker.shutdown` and `broker.nuke` methods will emit an error event, but attempt to continue. In both cases the error will have a code of `ETIMEDOUT`.
+Please refer to the [Change Log](https://github.com/onebeyond/rascal/blob/master/CHANGELOG.md)
 
 ### Special Note
 
@@ -318,6 +318,7 @@ The simplest way to specify a connection is with a url
   }
 }
 ```
+As of Rascal v18.0.0 you must URL encode special characters appearing in the username, password and vhost, e.g. `amqp://guest:secr%23t@broker.example.com:5672/v1?heartbeat=10`
 
 Alternatively you can specify the individual connection details
 
@@ -345,6 +346,8 @@ Alternatively you can specify the individual connection details
 }
 ```
 
+Special characters do not need to be encoded when specified in this form.
+
 Any attributes you add to the "options" sub document will be converted to query parameters. Any attributes you add in the "socketOptions" sub document will be passed directly to amqplib's connect method (which hands them off to `net` or `tls`. Providing you merge your configuration with the default configuration `rascal.withDefaultConfig(config)` you need only specify the attributes you want to override
 
 ```json
@@ -459,7 +462,7 @@ The AMQP protocol doesn't support assertion or checking of vhosts, so Rascal use
 }
 ```
 
-Rascal uses [superagent](https://github.com/visionmedia/superagent) under the hood. URL configuration is supported.
+Rascal uses [superagent](https://github.com/visionmedia/superagent) under the hood. URL configuration is also supported.
 
 ```json
 {
@@ -750,6 +753,62 @@ To define a queue with extensions such as `x-queue-type` add arguments to the op
 
 Refer to the [amqplib](https://www.squaremobius.net/amqp.node/channel_api.html) documentation for further queue options.
 
+#### streams
+
+Rascal supports [RabbitMQ Streams](https://www.rabbitmq.com/docs/streams) via x-queue-type argument, i.e.
+
+```json
+{
+  "queues": {
+    "q1": {
+      "options": {
+        "arguments": {
+          "x-queue-type": "stream"
+        }
+      }
+    }
+  }
+}
+```
+
+The [Stream Plugin](https://www.rabbitmq.com/docs/stream) and associated binary protocol extension are not supported.
+
+Streams are **not** a replacement for regular messaging - instead they are best suited for when you can tolerate occasional message loss and need for higher throughput, such as sampling web based analytics.
+
+When working with streams you need to think carefully about [data retention](https://www.rabbitmq.com/docs/streams#retention). Unless you specify retention configuration, messages will never be deleted and eventually you will run out of space. Conversely, if you automatically delete messages based on queue size or age, they may be lost without ever being read.
+
+You also need to think about how you will [track the consumer offset](https://www.rabbitmq.com/blog/2021/09/13/rabbitmq-streams-offset-tracking). Typically you will need to store this in a database after successfully processing the message and use it to tell the broker where to resume from after your application restarts. For example...
+
+```js
+  const initialOffset = (await loadOffset('/my-queue')) || 'first';
+
+  const overrides = {
+    options: {
+      arguments: {
+        'x-stream-offset': initialOffset
+      }
+    }
+  };
+
+  const subscription = await broker.subscribe('/my-queue', overrides);
+
+  subscription.on('message', async (message, content, ackOrNack) => {
+    const currentOffset = message.properties.headers['x-stream-offset'];
+    try {
+      await handleMessage(content);
+      await updateOffset('/my-queue', currentOffset);
+    } catch (err) {
+      await handleError('/my-queue', currentOffset, err);
+    } finally {
+      ackOrNack(); // Streams do not support nack so do not pass the error argument
+    }
+  });
+```
+
+However, if your application is offline for too long, and messages are still being published to the stream, it may not be able to resume from where you left off, since those messages may have been deleted. Furthremore, if your application consumes  messages concurrently, you need to think about how you will recover should one fail. If you naively override the previouly saved offset, you may be replacing a higher/later offset with an lower/older one, causing in your application to restart from the wrong point. Finally, you also need to decide what to do if the message cannot be processed. You cannot simply replay the message since you are working with a stream, rather than a classic queue. You could cancel the subscription and resume from the current offset, but this will lead to duplicates if you have been consuming messages concurrently. Alternatively you could republish the failures to a dead letter queue and process them separately.
+
+For the above reasons, we only recommend considering streams when you genuinely need the extra throughput.
+
 #### bindings
 
 You can bind exchanges to exchanges, or exchanges to queues.

package/examples/streams/package-lock.json

@@ -0,0 +1,13 @@
+{
+  "name": "streams",
+  "version": "1.0.0",
+  "lockfileVersion": 1,
+  "requires": true,
+  "dependencies": {
+    "random-readable": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/random-readable/-/random-readable-1.0.1.tgz",
+      "integrity": "sha512-Y++VltLA4yRsvFDAPbODh9hMw7cfkng+c/S+44ob6xGt0itLr8s6VhANl7kY7igEv3igPgzdc+T8EhBjQWjd9g=="
+    }
+  }
+}

package/examples/streams/package.json

@@ -0,0 +1,14 @@
+{
+  "name": "streams",
+  "version": "1.0.0",
+  "description": "",
+  "main": "index.js",
+  "scripts": {
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "author": "",
+  "license": "ISC",
+  "dependencies": {
+    "random-readable": "^1.0.1"
+  }
+}

package/examples/streams/publisher-config.json

@@ -0,0 +1,37 @@
+{
+  "$schema": "../../lib/config/schema.json",
+  "vhosts": {
+    "/": {
+      "publicationChannelPools": {
+        "regularPool": {
+          "max": 10,
+          "min": 10,
+          "evictionRunIntervalMillis": 1000,
+          "idleTimeoutMillis": 5000,
+          "autostart": true
+        }
+      },
+      "connection": {
+        "socketOptions": {
+          "timeout": 1000
+        }
+      },
+      "queues": {
+        "demo_stream": {
+          "options": {
+            "arguments": {
+              "x-queue-type": "stream",
+              "x-max-length-bytes": 10485760
+            }
+          }
+        }
+      },
+      "publications": {
+        "demo_pub": {
+          "queue": "demo_stream",
+          "confirm": false
+        }
+      }
+    }
+  }
+}

package/examples/streams/publisher.js

@@ -0,0 +1,38 @@
+const Rascal = require('../..');
+const config = require('./publisher-config');
+const random = require('random-readable');
+const max = parseInt(process.argv[2], 10) || Infinity;
+
+Rascal.Broker.create(Rascal.withDefaultConfig(config), (err, broker) => {
+  if (err) throw err;
+
+  broker.on('error', console.error);
+
+  let count = 0;
+
+  const stream = random
+    .createRandomStream()
+    .on('error', console.error)
+    .on('data', (data) => {
+      broker.publish('demo_pub', data, (err, publication) => {
+        if (err) throw err;
+        else if (count >= max) stream.destroy();
+        else count++;
+        publication.on('error', console.error);
+      });
+    })
+    .on('close', () => {
+      console.log(`Published ${count} messages`)
+      broker.shutdown();
+    });    ;
+
+  broker.on('busy', (details) => {
+    console.log(Date.now(), `Pausing vhost: ${details.vhost} (mode: ${details.mode}, queue: ${details.queue}, size: ${details.size}, borrowed: ${details.borrowed}, available: ${details.available})`);
+    stream.pause();
+  });
+
+  broker.on('ready', (details) => {
+    console.log(Date.now(), `Resuming vhost: ${details.vhost} (mode: ${details.mode}, queue: ${details.queue}, size: ${details.size}, borrowed: ${details.borrowed}, available: ${details.available})`);
+    stream.resume();
+  });
+});

package/examples/streams/subscriber-config.json

@@ -0,0 +1,28 @@
+{
+  "$schema": "../../lib/config/schema.json",
+  "vhosts": {
+    "/": {
+      "connection": {
+        "socketOptions": {
+          "timeout": 1000
+        }
+      },
+      "queues": {
+        "demo_stream": {
+          "options": {
+            "arguments": {
+              "x-queue-type": "stream",
+              "x-max-length-bytes": 10485760
+            }
+          }
+        }
+      },
+      "subscriptions": {
+        "demo_sub": {
+          "queue": "demo_stream",
+          "prefetch": 250
+        }
+      }
+    }
+  }
+}

package/examples/streams/subscriber.js

@@ -0,0 +1,25 @@
+const Rascal = require('../..');
+const config = require('./subscriber-config');
+const offset = parseInt(process.argv[2], 10) || 'first';
+
+Rascal.Broker.create(Rascal.withDefaultConfig(config), (err, broker) => {
+  if (err) throw err;
+
+  broker.on('error', console.error);
+
+  const overrides = {
+    options: {
+      arguments: {
+        'x-stream-offset': offset
+      }
+    }
+  };
+
+  broker.subscribe('demo_sub', overrides, (err, subscription) => {
+    if (err) throw err;
+    subscription.on('message', (message, content, ackOrNack) => {
+      console.log(`Received message: ${message.properties.headers['x-stream-offset']}`)
+      ackOrNack();
+    });
+  });
+});

package/lib/amqp/Vhost.js

@@ -201,8 +201,7 @@ function Vhost(vhostConfig, components) {
           createChannelWhenInitialised(options.confirm, (err, channel) => {
             if (err) return deferRejection(reject, err);
             if (!channel) return deferRejection(reject, new Error('Vhost is shutting down'));
-            const destroyChannel = _.once((err) => {
-              console.log('Destroying channel', err);
+            const destroyChannel = _.once(() => {
               debug('Destroying %s channel: %s for vhost: %s due to error or close event', mode, channel._rascal_id, vhostConfig.name);
               channel._rascal_closed = true;
               if (pool.isBorrowedResource(channel)) {

package/lib/config/configure.js

@@ -91,7 +91,7 @@ module.exports = _.curry((rascalConfig, next) => {
     } = new URL(connectionString);
     const options = Array.from(searchParams).reduce((attributes, entry) => ({ ...attributes, [entry[0]]: entry[1] }), {});
     return {
-      protocol, hostname, port, user, password, vhost, options,
+      protocol, hostname: decodeURIComponent(hostname), port, user: decodeURIComponent(user), password: decodeURIComponent(password), vhost: decodeURIComponent(vhost), options,
     };
   }
 
@@ -105,10 +105,14 @@ module.exports = _.curry((rascalConfig, next) => {
   }
 
   function configureManagementConnection(vhostConfig, vhostName, connection) {
-    _.defaultsDeep(connection.management, { hostname: connection.hostname });
-    const auth = connection.management.auth || getAuth(connection.management.user, connection.management.password) || getAuth(connection.user, connection.password);
-    connection.management.url = connection.management.url || url.format({ ...connection.management, auth });
-    connection.management.loggableUrl = connection.management.url.replace(/:[^:]*?@/, ':***@');
+    connection.management = _.isString(connection.management) ? { url: connection.management } : connection.management;
+    const attributesFromUrl = parseConnectionUrl(connection.management.url);
+    const attributesFromConfig = getConnectionAttributes(connection.management);
+    const defaults = { user: connection.user, password: connection.password, hostname: connection.hostname };
+
+    const connectionAttributes = _.defaultsDeep({ options: null }, attributesFromUrl, attributesFromConfig, defaults);
+    setConnectionAttributes(connection.management, connectionAttributes);
+    setConnectionUrls(connection.management);
   }
 
   function setConnectionAttributes(connection, attributes, defaults) {
@@ -120,7 +124,6 @@ module.exports = _.curry((rascalConfig, next) => {
     const auth = getAuth(connection.user, connection.password);
     const pathname = connection.vhost === '/' ? '' : connection.vhost;
     const query = connection.options;
-
     connection.url = url.format({
       slashes: true, ...connection, auth, pathname, query,
     });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "rascal",
-  "version": "17.0.2",
+  "version": "18.0.1",
   "description": "A config driven wrapper for amqplib supporting multi-host connections, automatic error recovery, redelivery flood protection, transparent encryption / decryption,  channel pooling and publication timeouts",
   "main": "index.js",
   "dependencies": {