keuss 1.6.10 → 1.6.13

package/docusaurus/docs/api/queue.md

@@ -1,247 +0,0 @@
----
-id: queue
-title: Queue API
-sidebar_label: Queue
----
-
-### `stats`: Queue stats
-
-```javascript
-q.stats ((err, res) => {
-  ...
-})
-```
-
-* `res` contains usage statistics (elements inserted, elements extracted, paused status).
-
-### `name`: Queue name
-
-```javascript
-var qname = q.name ()
-```
-
-### `type`: Queue type
-
-```javascript
-var qtype = q.type ()
-```
-
-Returns a string with the type of the queue (the type of backend which was used to create it).
-
-### `size`: Queue occupation
-
-```javascript
-q.size ((err, res) => {
-  ...
-})
-```
-
-Returns the number of elements in the queue that are already elligible (that is, excluding scheduled elements with a schedule time in the future).
-
-### `totalSize`: Total queue occupation
-
-```javascript
-q.totalSize ((err, res) => {
-  ...
-})
-```
-
-Returns the number of elements in the queue (that is, including scheduled elements with a schedule time in the future).
-
-### `schedSize`: Size of Scheduled
-
-```javascript
-q.schedSize ((err, res) => {
-  ...
-})
-```
-
-Returns the number of scheduled elements in the queue (that is, those with a schedule time in the future). Returns 0 if the queue does not support scheduling.
-
-### `resvSize`: Reserved elements size
-
-```javascript
-q.resvSize ((err, res) => {
-  ...
-})
-```
-
-Returns the number of reserved elements in the queue. Returns `null` if the queue does not support reserve.
-
-### `pause` / `paused`: Pause/Resume
-
-```javascript
-// pauses the queue
-q.pause (true)
-
-// resumes the queue
-q.pause (false)
-
-// gets paused status of queue
-q.paused ((err, is_paused) => {
-  ...
-})
-```
-
-Pauses/Resumes all consumers on this queue (calls to `pop()`). Producers are not afected (calls to `push()`).
-
-The pause/resume condition is propagated via the signallers, so this affects all consumers, not only those local to the process, if a redis-pubsub or mongo-capped signaller is used.
-
-Also, the paused condition is stored as stats, so any new call to `pop()` will honor it.
-
-### `next_t`: Time of schedule of next message
-
-```javascript
-q.next_t ((err, res) => {
-  ...
-})
-```
-
-Returns a `Date`, or `null` if queue is empty. Queues with no support for schedule/delay always return `null
-
-### `push`: Add element to queue
-
-```javascript
-q.push (payload, [opts,] (err, res) => {
-  ...
-})
-```
-
-Adds payload to the queue and calls passed callback upon completion. Callback's *res* will contain the id assigned to the inserted element, if the backup provides one.
-
-`payload` can be an `object`, `array`, `string`, `number` or `Buffer`
-
-Possible opts:
-
-* `mature`: unix timestamp where the element would be elligible for extraction. It is guaranteed that the element won't be extracted before this time.
-* `delay`: delay in seconds to calculate the mature timestamp, if mature is not provided. For example, a delay=120 guarantees the element won't be extracted until 120 secs have elapsed *at least*.
-* `tries`: value to initialize the retry counter, defaults to 0 (still no retries).
-* `hdrs`: object with scalar-valued keys (ie, string, number or boolean) to be added alongside the `payload` as general purpose headers
-
-:::note
-**mature** and **delay** have no effect if the backend does not support delay/schedule.
-:::
-
-### `pop`: Get element from queue
-
-```javascript
-var tr = q.pop (cid, [opts,] (err, res) => {
-  ...
-})
-```
-
-Obtains an element from the queue. Callback is called with the element obtained if any, or if an error happened. If defined, the operation will wait for `opts.timeout` seconds for an element to appear in the queue before bailing out (with both `err` and `res` being null). However, it immediately returns an id that can be used to cancel the operation at anytime.
-
-`*cid*` is an string that identifies the consumer entity; it is used only for debugging purposes.
-
-Possible opts:
-
-* **timeout**: milliseconds to wait for an elligible element to appear in the queue to be returned. If not defined it will wait forever
-* **reserve**: if `true` the element is only reserved, not completely returned. This means either *ok* or *ko* operations are needed upon the obtained element once processed, otherwise the element will be rolled back (and made available again) at some point in the future (this is only available on backends capable of reserve/commit).
-
-### `cancel`: Cancel a pending Pop
-
-```javascript
-var tr = q.pop (cid, opts, (err, res) => {...});
-.
-.
-.
-q.cancel (tr);
-```
-
-Cancels a pending `pop` operation, identified by the value returned by `pop()`
-
-If no `tr` param is passed, or it is `null`, all pending `pop` operations on the queue are cancelled. Cancelled `pop` operations will get `'cancel'` (a string) in the `error` parameter value of the callback.
-
-### `ok`: Commit a reserved element
-
-```javascript
-q.ok (id, (err, res) => {
-  ...
-})
-```
-
-Commits a reserved element by its id (the id would be assigned to `res._id` on the `res` param of `pop()` operation). This effectively erases the element from the queue.
-
-Alternatively, you can pass the entire `res` object from the `pop()` operation:
-
-```javascript
-var tr = q.pop ('my-consumer-id', {reserve: true}, (err, res) => {
-  // do something with res
-  ...
-
-  // commit it
-  q.ok (res, (err, res) => {
-    ...
-  });
-});
-```
-
-### `ko`: Roll back a reserved element
-
-```javascript
-q.ko (id, next_t, (err, res) => {
-  ...
-})
-```
-
-Rolls back a reserved element by its id (the id would be at `res._id` on the `res` param of `pop()` operation). This effectively makes the element available again at the queue, marking it to be mature at `next_t` (`next_t` being a millsec-unixtime). If no `next_t` is specified or a `null` is passed, `now()` is assumed.
-
-the `res` param of the callback can take the following values:
-* `true` if all went ok and an element was in fact rolled back
-* *nullish* if the element does not exist in the queue
-* `'deadletter'` (as string) if the rollback resulted in the element being moved to deadletter 
-
-As with `ok()`, you can use the entire `res` object as `id`:
-
-```javascript
-var tr = q.pop ('my-consumer-id', {reserve: true}, (err, res) => {
-  // do something with res
-  ...
-
-  // commit or rollback it
-  if (succeed) q.ok (res, (err, res) => {
-    ...
-  })
-  else q.ko (res, (err, res) => {
-    if (res === 'deadletter') {
-      // moved to deadletter
-      ...
-    }
-    ...
-  })
-});
-```
-
-:::important
-You must pass the entire `res` object for the [deadletter](../api/factory#dead-letter) feature to work; even if activated at the factory, `ko()` will not honor deadletter if you only pass the `res._id` as `id`.
-:::
-
-### `remove`: Delete elements from queue by id
-
-```javascript
-q.remove (id, (err, res) => {
-  ...
-})
-```
-
-Remvoes an element from a queue, by using the id returned at insertion time (backends supporting `remove` will return an in upon insertion). A reserved element can not be removed, and will be considered as nonexistent
-
-the `res` param of the callback can take the following values:
-* `true` if all went ok and the element was removed
-* *nullish* if the element does not exist in the queue
-
-### `drain`: Drain queue
-
-```javascript
-q.drain (err => {
-  ...
-})
-```
-
-Drains a queue. This is a needed operation when a backend does read-ahead upon a `pop()`, or buffers `push()` operations for later; in this case, you may want to be sure that all extra elemens read are actually popped, and all pending pushes are committed.
-
-:::warning
-'drain' will immediately inhibit `push()`: any call to `push()` will immediately result in a `'drain'` (a string)  in the `error` parameter value of the callback. The callback will be called when all pending pushes are committed, and all read-ahead on a pop() has been actually popped.
-:::
-Also, `drain()` will also call `cancel()` on the queue immediately before finishing, in case of success.

package/docusaurus/docs/api/signal.md

@@ -1,38 +0,0 @@
----
-id: signal
-title: Signaller API
-sidebar_label: Signaller
----
-## Signaler factory
-
-Signaller factory is passed to queues either in queue creation or in backend init, inside *opts.signaller*. Note that the result for the *new* operation is indeed the factory; the result of the `require` is therefore a *metafactory*.
-
-```javascript
-var signal_redis_pubsub = require ('keuss/signal/redis-pubsub');
-
-var local_redis_opts = {
-  Redis: {
-    port: 6379,
-    host: 'localhost',
-    db: 6
-  }
-};
-
-var f_opts = {
-  signaller: {
-    provider: signal_redis_pubsub,
-    opts: local_redis_opts
-  }
-  .
-  .
-  .
-}
-
-MQ (f_opts, (err, factory) => {
-  // queues created by factory here will use a redis pubsub signaller, hosted at redis at localhost, db 6
-})
-```
-
-:::note
-The signaller has no public api *per se*; it is considered just a piece of infrastructure to glue queues together.
-:::

package/docusaurus/docs/api/stats.md

@@ -1,37 +0,0 @@
----
-id: stats
-title: Stats API
-sidebar_label: Stats
----
-
-## Stats factories
-
-Stats factories are passed to queues either in queue creation or in backend init, inside *opts.signaller*. Note that the result of the `new` operation is indeed the factory; the result of the `require` is therefore a *metafactory*
-
-```javascript
-var local_redis_pubsub = require ('keuss/signal/redis-pubsub');
-
-var local_redis_opts = {
-  Redis: {
-    port: 6379,
-    host: 'localhost',
-    db: 6
-  }
-};
-
-var f_opts = {
-  stats: {
-    provider: signal_redis_pubsub,
-    opts: local_redis_opts
-  }
-  .
-  .
-  .
-}
-
-MQ (f_opts, (err, factory) => {
-  // queues created by factory here will use a redis-backed stats, hosted at redis at localhost, db 6
-})
-```
-
-Stats objects, as of now, store the number of elements inserted and the number of elements extracted; they are created behind the scenes and tied to queue instances, and the stats-related interface is in fact part of the queues' interface.

package/docusaurus/docs/changelog.md

@@ -1,51 +0,0 @@
----
-id: changelog
-title: Changelog
-sidebar_label: Changelog
----
-* v1.6.10
-  * updated deps
-  * add 'deadletter' to stats, for elements moved to deadletter queue
-* v1.6.9
-  * fix glitch in remove item on tape mongo (ps-mongo)
-* v1.6.8
-  * added remove-by-id support 
-* v1.6.7
-  * small fix on deadetter signalling   
-* v1.6.6
-  * added reserve, commit, rollback counters to stats
-  * fixed mongo-driver deprecation
-* v1.6.5
-  * dependencies updated
-* v1.6.4
-  * dependencies updated
-* v1.6.3
-  * added support for headers (along with payload)
-* v1.6.2
-  * payload can be of type object, string, number of buffer
-* v1.6.1
-  * added Docusaurus based documentation
-  * fixed deps' vulnerabilities
-* v1.6.0
-  * added sane defaults for stats and signal for mongodb-based backends (using mongo stats and signal)
-  * added pipeline builder
-  * added ability to create a full pipeline from text (making it trivial to be stored in file)
-* v1.5.12
-  * corrected small pipeline-related issues
-* v1.5.11 (void)
-* v1.5.10
-  * pipelines overhaul
-  * mubsub change to @nodebb/mubsub
-* v1.5.9:
-  * added some complete, meaningful examples
-* v1.5.8
-  * added deadletter support
-* v1.5.7
-  * added resvSize support
-* v1.5.4:
-  * added pause support
-  * deps updated
-* v1.5.3:
-  * use mongodb driver v3.2 (was v2.2)
-* v1.5.2
-  * added bucket-based backends: 2 backends using buckets/buffering on top of mongodb

package/docusaurus/docs/concepts.md

@@ -1,116 +0,0 @@
----
-id: concepts
-title: Concepts
-sidebar_label: Concepts
----
-
-## Queue
-
-A **Queue** is more of an interface, a definition of what it can do. Keuss queues are capable of:
-
-* Insert one element.
-* Schedule an element: insert one element with a not-before datetime; this means, the element will be affectively inserted in the queue, but any operation on the queue ofter that will not take that element into account before the not-before datetime.
-* Get an element, and block for some specified time if no element is available.
-* Reserve an element, and block for some specified time if no element is available.
-* Commit (remove) or rollback (return back) a previously reserved element.
-* Remove an element by its id (which was returned in the insertion)
-* Get element count (it will not include the elements scheduled in the future).
-* Get element count whose not-before datetime is in the future (scheduled elements).
-* Get usage stats: elements inserted, elements extracted.
-
-*Element* here translates to any js object, js array, string, number or `Buffer`. Optionally, a set of headers (in the form of a js object with string, number or boolean values) can be added.
-
-## Bucket
-
-The initial idea for Keuss Queues, transtated the elements inserted in the queue into rows of the backed storage. This makes it easy to inspect the elements values directly in the backend, which is pretty useful when you need to debug things up. Buckets came later, as a way to pack more than one message into a single row of the backend to gain performance. See [Bucked-based backends](usage/buckets).
-
-## Pipeline
-
-A **[pipeline](usage/pipelines/about)** is an enhanced queue that provides an extra operation: pass an element to another queue **atomically**. In an scenario where processors are linked with queues, it is usually a good feature to allow the *'commit element in incoming queue, insert element in the next queue'* to be atomic. This removes chances for race conditions, or message losses.
-
-The pipeline concept is, indeed, an extension of the reserve-commit model; it is so far implemented only atop mongodb, and it is anyway considered as a 'low-level' feature, best used by means of specialized classes to encapsulate the aforementioned processors.
-
-## Processor
-
-A **processor** is an object tied to one or more queues, that controls the flow of messages between them. They are used mainly to define [**pipelines**](usage/pipelines/about). Currently there are 4 specialized classes of processors defined:
-
-* [BaseLink](usage/pipelines/processors#baselink): This is really more of a base definition for the rest of the specialized processors.
-* [DirectLink](usage/pipelines/processors#directlink) (one queue to another).
-* [ChoiceLink](usage/pipelines/processors#choicelink) (one queue to one or more queues).
-* [Sink](usage/pipelines/processors#sink) (endpoint, one queue to none).
-
-## Storage
-
-The **Storage** or **Backend** provides almost-complete queue primitives, fully functional and already usable as is. Keuss comes with 7 backends, with various levels of features and performance:
-
-* *`mongo`*, a mongodb-based backend that provides the full set of queue features, still with decent performance.
-* *`redis-oq`*, backed using an ordered queue on top of redis (made in turn with a sorted set, a hash and some lua). Provides all queue features including reserve-commit-rollback. Noticeable faster than mongodb.
-* *`redis-list`*, backed using a redis list. Does not offer reserve-commit-rollback nor the ability to schedule, but is much faster than redis-oq
-* *`pl-mongo`*, a version of the *`mongo`* backend that provides pipelining capabilities (the queues it produces are also pipelines).
-* *`ps-mongo`*, a version of the *`mongo`* backend where elements are not physically deleted from the collection when extracted; instead, they are just marked as processed and later deleted automatically using a mongodb TTL index.
-* *`bucket-mongo`*, a first attepmt on storing more than one element on each mongodb record in order to break past mongodb I/O limitations. It is very simple, lacking schedule and reserve support. However, it has staggering throughput on a reasonable durability.
-* *`bucket-mongo-safe`*, an evolution of *`bucket-mongo`*, provides both scheduling and reserve support with a performance only a bit below *`bucket-mongo`*.
-
-As mentioned before, persistence and High Availability (HA) depends exclusively on the underliying system: mongodb provides production-grade HA and persistence while using potentially gigantic queues, and with redis one can balance performance and simplicity over reliability and durability, by using standalone redis, redis sentinel or redis cluster. Keuss uses [ioredis](https://github.com/luin/ioredis) as redis driver, which supports all 3 cases.
-
-The following table shows the capabilities of each backend:
-
-backend           | delay/schedule | reserve/commit | pipelining | history | remove | throughput |
-------------------|:--------------:|:--------------:|:----------:|:-------:|:------:|:----------:|
-redis-list        | - | - | - | - | - | ++++
-redis-oq          | x | x | - | - | x | +++
-mongo             | x | x | - | - | x | ++
-pl-mongo          | x | x | x | - | x | +
-ps-mongo          | x | x | - | x | x | ++
-bucket-mongo      | - | - | - | - | - | +++++
-bucket-mongo-safe | x | x | - | - | x | +++++
-
-## Signaller
-
-**Signaller** provides a bus interconnecting all keuss clients, so events can be shared. Keuss provides 3 signallers:
-
-* *`local`* : provides in-proccess messaging, useful only for simple cases or testing
-* *`redis-pubsub`*: uses the pubsub subsystem provided by redis
-* *`mongo-capped`*: uses pubsub on top of a mongodb capped collection, using [@nodebb/mubsub](https://www.npmjs.com/package/@nodebb/mubsub)
-
-So far, the only events published by keuss are:
-
-* *element inserted in queue X*, which allows other clients waiting for elements to be available to wake up and retry. A client will not fire an event if
-  another one of the same type (same client, same queue) was already fired less than 50ms ago.
-* *queue X paused/resumed*.
-
-The use of a signaller provider different from `local` allows the formation of a cluster of clients: all those clients sharing the same signaller object
-(with the same configuration, obviously) will see and share the same set of events and therefore can collaborate (for example, all consumers of a given
-queue *on every machine* will be awaken when an insertion happens *on any machine*)
-
-## Stats
-
-**Stats** provides counters and metrics on queues, shared among keuss clients. The supported stats are:
-
-* Elements put
-* Elements got
-* Elements reserved
-* Elements committed
-* Elements rolled back
-* Paused status
-
-Three options are provided to store the stats:
-
-* *`mem`*: very simple in-process, memory based.
-* *`redis`*: backed by redis hashes. Modifications are buffered in memory and flushed every 100ms.
-* *`mongo`*: backed by mongodb using one object per queue inside a single collection. Modifications are buffered in memory and flushed every 100ms.
-
-The use of a stats provider different from `mem` allows for a shared view of a cluster of clients: all those clients sharing the same stats object
-(with the same configuration, obviously) will see a coherent, aggregated view of the stats (all clients will update the stats)
-
-The stats can also be used as a queue discovery source: existing queues can be recreated from the information stored (in fact, extra information
-needed to ensure this is also stored alongside the actual stats). Keuss does not, at this point, provide any actual *recreate* functionality on
-top of this
-
-## How all fits together
-
-* *`Queues`*, or rather clients to individual queues, are created using a *backend* as factory.
-* *`Backends`* need to be initialized before being used. Exact initialization details depend on each backend.
-* When creating a *`queue`*, a *`signaller`* and a *`stats`* are assigned to it. The actual class/type to be used can be specified at the queue's creation moment, or at the backend initialization moment. By default *`local`* and *`mem`*, respectively, are used for redis-based backends; for mongodb-based backends, *`mongo-capped`* and *`mongo`* are used intead as defaults
-* *`Queues`* are created on-demand, and are never destroyed as far as Keuss is concerned. They do exist as long as the underlying backend kepts them in existence: for example, redis queues dissapear as such when they become empty.
-* *`Pipelines`* are, strictly speaking, just enhanced queues; as such they behave and can be used as a queue. More info on pipelines [here](usage/pipelines/about)

package/docusaurus/docs/examples.md

@@ -1,10 +0,0 @@
----
-id: examples
-title: Examples
-sidebar_label: Examples
----
-
-A set of funcioning examples can be found inside the *examples* directory:
-
-* [webhooks](https://github.com/pepmartinez/keuss/tree/master/examples/webhooks): A small but functionally complete webhook dispatcher sporting retries, reserve-commit, persistence and deadletter
-* [snippets](https://github.com/pepmartinez/keuss/tree/master/examples/snippets): A set of assorted code snippets for various tasks