keuss 1.6.10 → 1.6.13

package/docusaurus/docs/quickstart.md

@@ -1,195 +0,0 @@
----
-id: quickstart
-title: Quickstart
-sidebar_label: Quickstart
----
-
-## Package Install
-
-`keuss` is installed in the regular way for any npm package:
-
-```bash
-npm install keuss
-```
-
-## Basic usage (with regular MongoDB backend)
-
-Here's a minimal example of how keuss works. [async](https://www.npmjs.com/package/async) is used to implement asynchronous flows in a much readable manner
-
-```javascript
-const async = require ('async');
-const MQ =    require ('keuss/backends/mongo');
-
-MQ ({
-  url: 'mongodb://localhost/keuss_test'
-}, (err, factory) => {
-  if (err) return console.error(err);
-
-  // factory ready, create one queue
-  const q = factory.queue ('test_queue', {});
-
-  async.series([
-    cb => q.push (
-      {elem: 1, headline: 'something something', tags: {a: 1, b: 2}}, // this is the payload
-      {
-        hdrs: {h1: 'aaa', h2: 12, h3: false}  // let's add some headers too
-      },
-      cb
-    ),
-    cb => q.pop ('consumer-1', cb)
-  ], (err, res) => {
-    if (err) {
-      console.error (err);
-    }
-    else {
-      console.log (res[1]);
-      // this should print something like:
-      // {
-      //   _id: <some id>,
-      //   mature: <some date>,
-      //   payload: { elem: 1, headline: 'something something', tags: { a: 1, b: 2 } },
-      //   tries: 0,
-      //   hdrs: {h1: 'aaa', h2: 12, h3: false}
-      // }
-    }
-
-    factory.close ();
-  });
-});
-```
-
-This small test creates a queue named `test_queue` backed by mongodb in the mongoDB collection at `mongodb://localhost/keuss_test`. Then, a single element is first inserted in the queue, then read from it and printed
-
-## Backend interchangeability
-
-This example works with any available definition of `MQ`; you just need to specify the chosen backend. For example, to use the `redis-list` backend:
-
-```js
-const MQ = require ('keuss/backends/redis-list');
-```
-
-## reserve-commit-rollback
-
-```javascript
-const async = require ('async');
-const MQ =    require ('keuss/backends/mongo');
-
-MQ ({
-  url: 'mongodb://localhost/keuss_test'
-}, (err, factory) => {
-  if (err) return console.error(err);
-
-  // factory ready, create one queue
-  const q = factory.queue ('test_queue', {});
-
-  async.waterfall ([
-    cb => q.push ({elem: 1, headline: 'something something', tags: {a: 1, b: 2}}, cb),  // (1)
-    (item_id, cb) => q.pop ('consumer-1', {reserve: true}, cb),                         // (2)
-    (item, cb) => {
-      console.log ('%s: got %o', new Date().toString (), item);                         // (3)
-      const next_t = new Date().getTime () + 1500;
-      q.ko (item, next_t, cb);                                                          // (4)
-    },
-    (ko_res, cb) => q.pop ('consumer-1', {reserve: true}, cb),                          // (5)
-    (item, cb) => {
-      console.log ('%s: got %o', new Date().toString (), item);                         // (6)
-      q.ok (item, cb);                                                                  // (7)
-    },
-  ], (err, res) => {
-    if (err) console.error (err);
-    factory.close ();
-  });
-});
-```
-
-1. An element is inserted.
-2. An element is reserved. It reserves the element previously inserted, and returns it.
-3. This should print the element reserved.
-4. The element reserved is rejected, indicating that it should not be made available until `now + 1500` millisecs.
-5. A second attempt at a reserve, this should return an element after 1500 millisecs.
-6. The same element should be printed here, except for the `tries` that should be `1` instead of `0`.
-7. The element is committed and thus removed from the queue.
-
-## Backend interchangeability
-
-This example works with any definition of `MQ` that supports reserve/commit (that is, any except `redis-list` and `bucket-mongo`); you just need to specify the chosen backend. For example, to use the `bucket-mongo-safe` backend:
-
-```js
-const MQ = require ('keuss/backends/bucket-mongo-safe');
-```
-
-## Full producer and consumer loops
-
-This is a more convoluted example: a set of producers inserting messages, and another set of consumers consumig them, all in parallel. The queue stats (elements pushed, elements popped) are shown every second.
-
-Try and change the uncommented `const MQ = require('keuss/backends/...');`  to see the performance differences between backends.
-
-Also, notice that, when, running with any mongodb-based backend, stats figures are cumulative across different executions: if you run it several times, you'll see the stats' figures also include data from previous executions.
-
-```js
-const async = require ('async');
-
-// choice of backend
-const MQ =    require ('keuss/backends/bucket-mongo-safe');
-//const MQ =    require ('keuss/backends/redis-oq');
-//const MQ =    require ('keuss/backends/mongo');
-//const MQ =    require ('keuss/backends/ps-mongo');
-
-MQ ({
-  url: 'mongodb://localhost/keuss_test'
-}, (err, factory) => {
-  if (err) return console.error(err);
-
-  const consumers = 3;
-  const producers = 3;
-  const msgs = 100000;
-
-  // factory ready, create one queue
-  const q = factory.queue ('test_queue', {});
-
-  // show stats every sec
-  const timer = setInterval (() => {
-    q.stats ((err, res) => console.log ('  --> stats now: %o', res));
-  }, 1000);
-
-  async.parallel ([
-    // producers' loop
-    cb => async.timesLimit (msgs, producers, (n, next) => {
-      q.push ({elem: n, headline: 'something something', tags: {a: 1, b: 2}}, next);
-    }, err => {
-      console.log ('producer loop ended');
-      cb (err);
-    }),
-    // consumers' loop
-    cb => async.timesLimit (msgs, consumers, (n, next) => {
-      q.pop ('theconsumer', {reserve: true}, (err, item) => {
-        if (err) return cb (err);
-        q.ok (item, next);
-      });
-    }, err => {
-      console.log ('consumer loop ended');
-      cb (err);
-    })
-  ], err => {
-    if (err) return console.error (err);
-
-    clearInterval (timer);
-
-    // all loops completed, cleanup & show stats
-    async.series ([
-      cb => q.drain (cb),
-      cb => q.stats (cb),
-      cb => setTimeout (cb, 1000),
-      cb => q.stats (cb),
-    ], (err, res) => {
-      if (err) console.error (err);
-      else {
-        console.log ('stats right after drain: %o', res[1]);
-        console.log ('stats once dust settled: %o', res[3]);
-      }
-
-      factory.close ();
-    });
-  });
-});
-```

package/docusaurus/docs/style-guide.md

@@ -1,202 +0,0 @@
----
-id: doc1
-title: Style Guide
-sidebar_label: Style Guide
----
-
-You can write content using [GitHub-flavored Markdown syntax](https://github.github.com/gfm/).
-
-## Markdown Syntax
-
-To serve as an example page when styling markdown based Docusaurus sites.
-
-## Headers
-
-# H1 - Create the best documentation
-
-## H2 - Create the best documentation
-
-### H3 - Create the best documentation
-
-#### H4 - Create the best documentation
-
-##### H5 - Create the best documentation
-
-###### H6 - Create the best documentation
-
----
-
-## Emphasis
-
-Emphasis, aka italics, with *asterisks* or _underscores_.
-
-Strong emphasis, aka bold, with **asterisks** or __underscores__.
-
-Combined emphasis with **asterisks and _underscores_**.
-
-Strikethrough uses two tildes. ~~Scratch this.~~
-
----
-
-## Lists
-
-1. First ordered list item
-1. Another item
-   - Unordered sub-list.
-1. Actual numbers don't matter, just that it's a number
-   1. Ordered sub-list
-1. And another item.
-
-* Unordered list can use asterisks
-
-- Or minuses
-
-+ Or pluses
-
----
-
-## Links
-
-[I'm an inline-style link](https://www.google.com/)
-
-[I'm an inline-style link with title](https://www.google.com/ "Google's Homepage")
-
-[I'm a reference-style link][arbitrary case-insensitive reference text]
-
-[You can use numbers for reference-style link definitions][1]
-
-Or leave it empty and use the [link text itself].
-
-URLs and URLs in angle brackets will automatically get turned into links. http://www.example.com/ or <http://www.example.com/> and sometimes example.com (but not on GitHub, for example).
-
-Some text to show that the reference links can follow later.
-
-[arbitrary case-insensitive reference text]: https://www.mozilla.org/
-[1]: http://slashdot.org/
-[link text itself]: http://www.reddit.com/
-
----
-
-## Images
-
-Here's our logo (hover to see the title text):
-
-Inline-style: ![alt text](https://github.com/adam-p/markdown-here/raw/master/src/common/images/icon48.png 'Logo Title Text 1')
-
-Reference-style: ![alt text][logo]
-
-[logo]: https://github.com/adam-p/markdown-here/raw/master/src/common/images/icon48.png 'Logo Title Text 2'
-
-Images from any folder can be used by providing path to file. Path should be relative to markdown file.
-
-![img](../static/img/logo.svg)
-
----
-
-## Code
-
-```javascript
-var s = 'JavaScript syntax highlighting';
-alert(s);
-```
-
-```python
-s = "Python syntax highlighting"
-print(s)
-```
-
-```
-No language indicated, so no syntax highlighting.
-But let's throw in a <b>tag</b>.
-```
-
-```js {2}
-function highlightMe() {
-  console.log('This line can be highlighted!');
-}
-```
-
----
-
-## Tables
-
-Colons can be used to align columns.
-
-| Tables        |      Are      |   Cool |
-| ------------- | :-----------: | -----: |
-| col 3 is      | right-aligned | \$1600 |
-| col 2 is      |   centered    |   \$12 |
-| zebra stripes |   are neat    |    \$1 |
-
-There must be at least 3 dashes separating each header cell. The outer pipes (|) are optional, and you don't need to make the raw Markdown line up prettily. You can also use inline Markdown.
-
-| Markdown | Less      | Pretty     |
-| -------- | --------- | ---------- |
-| _Still_  | `renders` | **nicely** |
-| 1        | 2         | 3          |
-
----
-
-## Blockquotes
-
-> Blockquotes are very handy in email to emulate reply text. This line is part of the same quote.
-
-Quote break.
-
-> This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can _put_ **Markdown** into a blockquote.
-
----
-
-## Inline HTML
-
-<dl>
-  <dt>Definition list</dt>
-  <dd>Is something people use sometimes.</dd>
-
-  <dt>Markdown in HTML</dt>
-  <dd>Does *not* work **very** well. Use HTML <em>tags</em>.</dd>
-</dl>
-
----
-
-## Line Breaks
-
-Here's a line for us to start with.
-
-This line is separated from the one above by two newlines, so it will be a _separate paragraph_.
-
-This line is also a separate paragraph, but... This line is only separated by a single newline, so it's a separate line in the _same paragraph_.
-
----
-
-## Admonitions
-
-:::note
-
-This is a note
-
-:::
-
-:::tip
-
-This is a tip
-
-:::
-
-:::important
-
-This is important
-
-:::
-
-:::caution
-
-This is a caution
-
-:::
-
-:::warning
-
-This is a warning
-
-:::

package/docusaurus/docs/usage/buckets.md

@@ -1,43 +0,0 @@
----
-id: buckets
-title: Bucket-based backends
-sidebar_label: Bucket-based backends
----
-
-Up to version 1.4.X all backends worked in the same way, one element at a time: pushing and popping elements fired one or more operations per element on the underlying storage. This means the bottleneck would end up being the storage's I/O; redis and mongo both allow quite high I/O rates, enough to work at thousands of operations per second. Still, the limit was there.
-
-Starting with v1.5.2 keuss includes 2 backends that do not share this limitation: they work by packing many elements inside a single 'storage unit'. Sure enough, this adds some complexity and extra risks, but the throughput improvement is staggering: on mongodb it goes from 3-4 Ktps to 35-40Ktps, and the bottleneck shifted from mongod to the client's cpu, busy serializing and deserializing payloads.
-
-Two bucked-based backends were added, both based on mongodb: [bucket-mongo](#bucket-mongo) and [bucket-mongo-safe](#bucket-mongo-safe). Both are usable, but there is little gain on using fhe first over the second: `bucket-mongo` was used as a prototyping area, and although perfectly usable, it turned out `bucket-mongo-safe` is better in almost every aspect: it provides better guarantees and more features, at about the same performance.
-
-### bucket-mongo-safe
-
-In addition to the general options, the factory accepts the following extra options:
-
-* `bucket_max_size`: maximum number of elements in a bucket, defaults to 1024
-* `bucket_max_wait`: milliseconds to wait before flushing a push bucket: pushes are buffered in a push bucket, which are flushed when they're full (reach `bucket_max_size` elements). If this amount of millisecs go by and the push bucket is not yet full, it is flushed as is. Defaults to 500.
-* `reserve_delay`: number of seconds a bucket keeps its 'reserved' status when read from mongodb. Defaults to 30.
-* `state_flush_period`: changes in state on each active/read bucket are flushed to mongodb every those milliseconds. Defaults to 500.
-* `reject_delta_base`, `reject_delta_factor`: if no call to `ko` provide a `next_t`, the backend will set one using a simple grade-1 polynom, in the form of `reject_delta_factor * tries + reject_delta_base`, in millisecs. They default to `10000` and `((reserve_delay * 1000) || 30000)` respectively
-* `reject_timeout_grace`: number of seconds to wait since a bucket is reserved/read until it is considered timed out; after this, what is left of the bucket is rejected/retried. Defaults to (`reserve_delay` * `0.8`)
-* `state_flush_period`: flush intermediate state changes in each active read bucked every this amount of millisecs
-
-Bucket-mongo-safe works by packing many payloads in a single mongodb object:
-
-* At `push()` time, objects are buffered in memory and pushed (inserted) only when bucket_max_size has been reached or when a bucket has been getting filled for longer than bucket_max_wait millisecs.
-* At `pop/reserve` time full objects are read into mem, and then individual payloads returned from there. Both commits and pops are just marked in memory and then flushed every state_flush_period millisecs, or when the bucked is exhausted.
-* Buckets remain unmodified since they are created in terms of the payloads they contain: a `pop()` or `ko/ok` would only mark payloads inside buckets as read/not-anymore-available, but buckets are never splitted nor merged.
-
-Thus, it is important to call `drain()` on queues of this backend: this call ensures all pending write buckets are interted in mongodb, and also ensures all in-memory buckets left are completely read (served through pop/reserve).
-
-Also, there is little difference in performance and I/O between `pop` and `reserve/commit`; performance is no longer a reason to prefer one over the other.
-
-:::note
-Scheduling on `bucket-mongo-safe` is perfectly possible, but with a twist: the effective `mature_t` of a message will be the oldest in the whole bucket it resides in. This applies to both insert and rollback/ko. In practice this is usually not a big deal, since anyway the `mature_t` is a 'not before' time, and that's all Keuss (or any other queuing middleware) would guarantee.
-:::
-
-### bucket-mongo
-
-This is a simpler version of buckets-on-mongodb, and for all purposes `bucket-mongo-safe` should be preferred; it does not provide reserve, nor schedule. It is however a tad faster and lighter on I/O.
-
-It is provided only for historical and educational purposes.

package/docusaurus/docs/usage/no-signaller.md

@@ -1,16 +0,0 @@
----
-id: no-signaller
-title: Using no signaller
-sidebar_label: Using no signaller
----
-
-Even when using signallers, `pop` operations on queue never block or wait forever; waiting `pop` operations are anyway terminated after 15000 millisecs
-or whatever specified in the `pollInterval` parameter) and silently re-initiated. That is, a `pop()` on an empty queue will appear blocked forever to
-the caller, but behind the scenes it'll work pretty much as if it were doing a poll every 15 secs
-
-If a signaller is used (or if a signaller other than `local` is used, if `push()` and `pop()` happen on different machines) the `pop()` will be awaken almost
-immediately after the `push()`; if no signaller is used (or `local`is used, but the action happens in separated machines) `pop()` will behave exactly as if
-it were doing a poll() internally;
-
-Another way to put it is, `pop()` operations would have a maximum latency of `pollInterval` millisecs, but also provides a safe backup in the event of
-signalling loss.

package/docusaurus/docs/usage/pipelines/about.md

@@ -1,128 +0,0 @@
----
-id: about
-title: About
-sidebar_label: About
----
-
-`Pipelines` is a Keuss extension for building [ETL](https://en.wikipedia.org/wiki/Extract,_transform,_load) processing graphs with ease while guaranteeing atomicity in the processing: whatever happens at the processing of an element, the element is guaranteed to be in either the source or in the destination queue; never in both, never in none.
-
-Keuss pipelines are build upon Keuss Queues with *pipeline* capacity, which means Pipelines inherit all their advantages in terms of HA, durability and performance. So far, Keuss offers only one Queue backend with pipeline capacity, `pl-mongo`
-
-Queues are linked together with processing units named *Processors*, which glue together a source queue with zero or more destination queues. Each processor encapsulates a loop that could be described -in its simplest form- as follows:
-
-```javascript
-forever do
-  src_queue.reserve () -> element    # reserve an element from entry queue
-  process (element) -> err, res      # process the element
-
-  if (err) then
-    if (err.drop) do                 # error tells processor to drop/ignore the element
-      src_queue.commit (element)
-    else do
-      src_queue.rollback (element)   # regular error, rollback. It would be retried
-    end
-  else
-    if (res.drop) do                 # processed ok, but drop the item anyway
-      src_queue.commit (element)
-    else do
-      # commit on entry queue and insert into the exit queue, all in one atomic operation
-      # modifications in the payload are conserved
-      move_to_next_queue (element, src_queue)
-    end
-  end
-
-  next_loop
-end
-```
-
-* The `process()` part is user-provided, passed as a function on the initialization of the processor
-* The exact semantics of `move_to_next_queue()` vary depending on the specific type of Processor chosen
-
-## Real, simple example
-
-Here is the simplest possible example: 2 queues connected with a very simple processor. Elements in the source queue are taken, a `passed: true` is added to them and moved to the next queue:
-
-```javascript
-const MQ = require    ('keuss/backends/pl-mongo');
-const PDL = require   ('keuss/Pipeline/DirectLink');
-const async = require ('async');
-
-const factory_opts = {
-  url: 'mongodb://localhost/qeus'
-};
-
-// initialize factory
-MQ (factory_opts, (err, factory) => {
-  if (err) return console.error (err);
-
-  // factory ready, create 2 queues on default pipeline
-  const q_opts = {};
-  const q1 = factory.queue ('test_pl_1', q_opts);
-  const q2 = factory.queue ('test_pl_2', q_opts);
-
-  // tie them up, q1 -> q2
-  const pdl = new PDL (q1, q2);
-
-  pdl.start ((elem, done) => {
-    // pass element to next queue, set payload.passed to true
-    done (null, {
-      update: {
-        $set: {passed: true}
-      }
-    });
-  });
-
-  // insert elements in the entry queue
-  async.timesLimit (111, 3, (n, next) => q1.push ({a:n, b:'see it spin...'}, next));
-
-  // read elements at the outer end
-  async.timesLimit (111, 3, (n, next) => q2.pop ('exit', (err, res) => {
-    console.log ('end point get', res);
-    next ();
-  }));
-});
-```
-
-just run this example and you'll see 111 elements being inserted at q1, being processed at the pdl processor, and then popped from q2
-
-## Pipeline-aware Queues
-
-As stated before only one Keuss Queue backed -`pl-mongo`- is compatible with pipelines. Those are the pipeline-related options available at the backend:
-
-* `pipeline`: specifies the pipeline name for this queue. Only queues within the same pipeline (that is, same mongodb url and same pipeline name) can actually work together in a pipeline. Defaults to `default`
-
-In the above example both queues q1 and q2 are created in a pipeline named 'default'. To use a different one you just change the code into:
-
-```javascript
-const q_opts = {pipeline: 'some_other_pipeline'};
-const q1 = factory.queue ('test_pl_1', q_opts);
-const q2 = factory.queue ('test_pl_2', q_opts);
-```
-
-Also, pipeline-aware queues provide a new operation:
-
-```javascript
-pl_step (id, next_queue, opts, callback)
-```
-
-* `id` is a previously reserved Id
-* `next_queue` is the queue to (atomically) move the item to
-* `opts` are extra options for the operation:
-  * `mature`: Date instance with the not-before timestamp for the item, to be used when inserted into `next_queue`. Defaults to `now()`
-  * `tries`: number of tries for the item, to be used when inserted into next_queue. Defaults to `0`
-  * `payload`: if specified, use this as item's payload when moving to next_queue. This totally substitutes the previous payload
-  * `update`: Optional object containing [mongodb update operations](https://docs.mongodb.com/manual/reference/operator/update/). Those are mapped to be applied to the message's `payload`. For example, in the example above:
-
-    ```javascript
-    done (null, {
-      update: {
-        $set: {passed: true}
-      }
-    });
-    ```
-
-    the '`update` parameter of the second argument to `done()` is passed internally to `pl_step()` as `opts.update`: this would cause the message's `payload.passed` to be set to `true` even if there's no explicit mention of `payload`
-
-The whole `pl_step()` operation is guaranteed to be atomic; this includes applying of `opts.payload` or `opts.update` if present
-
-Also, `opts.payload` takes precedence over `opts.update`: if both are specified only the former is taken into account, and the latter is totally ignored