iotagent-node-lib 4.5.0 → 4.7.0

package/doc/getting-started.md

@@ -1,64 +1,92 @@
-## Getting Started
+# Getting Started
 
-Every IoT Agent which uses the library is different, but the concepts for provisioning IoT devices remain the same
-regardless of protocol.
+-   [Introduction](#introduction)
+-   [IoT Agent settings - `config.js`](#iot-agent-settings---configjs)
+-   [Provisioning a Config Group](#provisioning-a-config-group)
+-   [Provisioning an Individual Device](#provisioning-an-individual-device)
+-   [Receiving measures from devices](#receiving-measures-from-devices)
+    -   [Receiving a measure from a known Device](#receiving-a-measure-from-a-known-device)
+    -   [Receiving a measure from an anonymous Device](#receiving-a-measure-from-an-anonymous-device)
 
-### `config.js` - IoT Agent settings
+## Introduction
+
+In this guide we will be using the IoT Agent JSON (which is the reference IoTAgent using the IoTAgent Library) as an
+example to demonstrate how to provision config groups, devices and how to receive measures from devices.
+
+Be aware that every IoT Agent which uses the library is different, but the concepts for provisioning IoT devices remain
+the same regardless of protocol.
+
+The IoT Agent JSON is a simple IoT Agent which uses JSON payloads to send and receive data. It is a good starting point
+for understanding how an IoT Agent works since it uses JSON payloads to send and receive data.
+
+## IoT Agent settings - `config.js`
 
 The `config.js` holds common information about the interactions between the Agent and the Context Broker. Additional
-custom settings may also be required dependent upon the actual IoT Agent used.
+custom settings may also be required dependent upon the actual IoT Agent used. The following is an example of a typical
+`config.js` file:
 
 ```javascript
 config = {
-    logLevel: "DEBUG",
+    logLevel: 'DEBUG',
     contextBroker: {
-        host: "orion",
-        port: "1026",
+        host: 'orion',
+        port: '1026'
     },
     server: {
         port: 4041,
-        host: "0.0.0.0",
+        host: '0.0.0.0'
     },
     deviceRegistry: {
-        type: "memory",
+        type: 'mongodb'
+    },
+    mongodb: {
+        host: 'localhost',
+        port: '27017',
+        db: 'iotagent'
     },
-    service: "openiot",
-    subservice: "/",
-    providerUrl: "http://iot-agent:4041",
-    defaultType: "Thing",
+    service: 'openiot',
+    subservice: '/',
+    providerUrl: 'http://iot-agent:4041',
+    defaultType: 'Thing'
 };
 ```
 
-In this case the context broker is called `orion` and is listening on port `1026`, the IoT Agent can be provisioned by
-sending requests to port `4041` which is also the port used to receive NGSI requests. The IoT Agent is holding the
-device mappings in memory.
+In this case the context broker hostname is `orion` and is listening on port `1026`, the IoT Agent can be provisioned by
+sending requests to port `4041` which is also the port used to receive NGSI requests. The IoT Agent is using the
+`iotagent` database from a MongoDB instance at `localhost:27017` to store needed information (provisioned groups and
+devices, etc.).
 
 The remaining settings help to define the NGSI interactions - the IoT Agent will be using the `fiware-service=openiot`
 and `fiware-service-path=/`. The default `type`for each created entity is `Thing`, although this can be overridden as
 shown below. Devices will be registered for a period of one month and the IoT Agent will receive registration callbacks
 at the URL `http://iot-agent:4041`.
 
-All configuration settings can also updated using Docker environment variables.
+All configuration settings can also updated using Docker environment variables. You can find more information about the
+available configuration parameters and environment variables in the [Administration Guide](admin.md).
 
-### Provisioning a Service Group
+## Provisioning a Config Group
 
-Settings which are common to a group of devices can be passed to the IoT Agent using the Service API. The
-`fiware-service` and `fiware-service-path` to be used to access the API are defined within the `config.js`. Each service
-may override values previously defined in the configuration if necessary.
+Settings which are common to a group of devices can be passed to the IoT Agent using the Config Group API. Each config
+group may override values previously defined in the global configuration if necessary. When using the config group API,
+the `fiware-service` and `fiware-servicepath` headers will define the service and subservice to which the configuration
+will be applied. Additionally, the `apikey` field is used to identify the configuration group. An example of a basic
+config group is shown below:
 
 ```bash
 curl -iX POST \
-  'http://localhost:4041/iot/services' \
+  'http://localhost:4041/iot/groups' \
   -H 'Content-Type: application/json' \
   -H 'fiware-service: openiot' \
   -H 'fiware-servicepath: /' \
   -d '{
- "services": [
+ "groups": [
    {
      "apikey":      "4jggokgpepnvsb2uv4s40d59ov",
-     "cbHost":     "http://orion:1026",
      "entity_type": "Device",
-     "resource":    "/iot/d",
+     "resource":    "/iot/json",
+     "attributes": [
+       { "object_id": "t", "name": "temperature", "type": "Number" }
+     ]
    }
  ]
 }'
@@ -66,12 +94,18 @@ curl -iX POST \
 
 In this case an `apiKey` for identifying devices has been created and all interactions to the path `/iot/d` which
 present this `apiKey` will be created as entities of `type=Device` rather than using the configuration default of
-`type=Thing`. The service group would usual hold additional attribute mappings, commands and common static attributes as
-well.
+`type=Thing`.
+
+Additionally, the group has defined an attribute mapping for a measurement `t` to be mapped to `temperature` attribute
+when receiving data from devices.
+
+The config group would usual hold additional attribute mappings, commands and common static attributes as well.
 
-### Provisioning an Individual Device
+## Provisioning an Individual Device
 
-Settings which are specific to an individual device can be passed to the IoT Agent using the Device API
+Settings which are specific to an individual device can be passed to the IoT Agent using the Device API. The
+configuration provided in the Device API will override any settings defined in the Config Group API and the global
+configuration as well. An example of a basic device configuration is shown below:
 
 ```bash
 curl -iX POST \
@@ -86,7 +120,7 @@ curl -iX POST \
      "entity_name": "urn:ngsi-ld:Motion:001",
      "entity_type": "Motion",
      "attributes": [
-       { "object_id": "c", "name": "count", "type": "Integer" }
+       { "object_id": "c", "name": "count", "type": "Number" }
      ],
      "static_attributes": [
        { "name":"refStore", "type": "Relationship", "value": "urn:ngsi-ld:Store:001"}
@@ -97,54 +131,81 @@ curl -iX POST \
 '
 ```
 
-This information is combined with the common service group information whenever a request is received at the South port
-of the IoT Agent and used to create or update the relevant entity in the Context Broker.
+The device `motion001` has been provisioned to persist data to the Context Broker as an entity of `type=Motion` (instead
+of the default `type=Thing`). The destination entity is identified by the `entity_name` field, which is set to
+`urn:ngsi-ld:Motion:001`. The device has a single attribute mapping for a measurement `c` to be mapped to `count`
+attribute, additionally to one defined in the group mapping (`temperature`). The device also has a static attribute
+`refStore` which is a relationship to the entity `urn:ngsi-ld:Store:001`.
 
-#### Receiving a measure from a known Device
+This information is combined with the common config group information whenever a measurement is received at the IoT
+Agent and used to create or update the relevant entity in the Context Broker.
 
-For example, imagine we are using the Ultralight IoT Agent and the following request is sent to the South port:
+## Receiving measures from devices
+
+In order to see the complete process from provisioning the groups and devices to receiving measures, we will show how a
+device can send a measure to the IoT Agent. In this case, we will use the IoTA JSON as an example, using the HTTP
+transport protocol. To reproduce the measure sending, you can use the following `curl` commands.
+
+The device measures are sent to the South port of the IoT Agent which is listening in the port `7896`.
+
+### Receiving a measure from a known Device
+
+In this case, the device has been provisioned previously. We will use the `motion001` device defined in the previous
+example. To simulate the device sending a measure, the following request is sent to the South port:
 
 ```bash
 curl -iX POST \
-  'http://localhost:7896/iot/d?k=4jggokgpepnvsb2uv4s40d59ov&i=motion001' \
-  -H 'Content-Type: text/plain' \
-  -d 'c|1'
+  'http://localhost:7896/iot/json?k=4jggokgpepnvsb2uv4s40d59ov&i=motion001' \
+  -H 'Content-Type: application/json' \
+  -d '{"t":23,"c":1}'
 ```
 
-The IoT Agent South port is listening to the path defined in the service group, and the API key is recognized to match.
+The IoT Agent South port is listening to the path defined in the config group, and the API key is recognized to match.
 Because the `device_id` is also recognized, the provisioned device configuration is used and its settings are combined
-with the service group. A mapping has been found to rename the `c` measurement to `count` - the following context entity
-is created in the context broker:
+with the config group.
+
+Mapping has been found to use the `c` measurement as `count` and the `t` measurement as `temperature` attributes values.
+The following context entity is created in the context broker:
 
 ```json
 {
     "id": "urn:ngsi-ld:Motion:001",
     "type": "Motion",
-    "count": { "value": "1", "type": "Integer" },
+    "temperature": { "value": 23, "type": "Number" },
+    "count": { "value": 1, "type": "Number" },
     "refStore": { "value": "urn:ngsi-ld:Store:001", "type": "Relationship" }
 }
 ```
 
-#### Receiving a measure from an anonymous Device
+### Receiving a measure from an anonymous Device
+
+When receiving a measure, it is not necessary to have the device provisioned. In this case, the IoT Agent will use the
+config group configuration to create the device and the entity. This process is called "autoprovision" and it is enabled
+by default in provisioned groups (for further information, review the
+[Autoprovision](api.md#autoprovision-configuration-autoprovision) section in the API documentation).
 
-For example, imagine we are using the Ultralight IoT Agent and a request from an anonymous, unprovisioned device is sent
-to the South port:
+Take as an example the following request from an anonymous device:
 
 ```bash
 curl -iX POST \
-  'http://localhost:7896/iot/d?k=4jggokgpepnvsb2uv4s40d59ov&i=temp001' \
-  -H 'Content-Type: text/plain' \
-  -d 't|1'
+  'http://localhost:7896/iot/json?k=4jggokgpepnvsb2uv4s40d59ov&i=dev001' \
+  -H 'Content-Type: application/json' \
+  -d '{"t":13,"c":4}'
 ```
 
-The IoT Agent South port is listen to the path defined in the service group, and the API key is recognized to match, so
-the Service group configuration will be used. No mappings will be made for the Entity `id` or the attribute names and
-the following entity will be created:
+The IoT Agent South port is listen to the path defined in the config group, and the API key is recognized to match, so
+the config group configuration will be used. No device has been provisioned with the `device_id=dev001`, so the IoT
+Agent will only use the config group configuration.
+
+A new entity will be created in the Context Broker with the `id` `Device:dev001` and the `type` `Device`. Only the `t`
+measurement will be mapped to the `temperature` attribute, as defined in the config group. The remaining measurements
+will be created as attributes with the same name. The following context entity will be created in the context broker:
 
 ```json
 {
-    "id": "temp001",
+    "id": "Device:dev001",
     "type": "Device",
-    "t": { "value": "1", "type": "Number" }
+    "temperature": { "value": 13, "type": "Number" },
+    "c": { "value": 4, "type": "Number" }
 }
 ```

package/doc/requirements.txt

@@ -1,4 +1,4 @@
 mkdocs==1.2.4
 Pygments==2.15.0
 Markdown==3.3.4
-jinja2==3.1.4
+jinja2==3.1.5

package/doc/roadmap.md

@@ -25,10 +25,7 @@ Disclaimer:
 The following list of features are planned to be addressed in the short term, and incorporated in a release of the
 product:
 
--   cgroup literal in configuration groups management API (community)
--   Refactor Append Mode & initial entity ([#1413](https://github.com/telefonicaid/iotagent-node-lib/issues/1413))
--   Native support for NGSI-v2 and LD ingestion ([#1451](https://github.com/telefonicaid/iotagent-node-lib/issues/1451))
--   Remove plugins structure (bidirectional plugin) ([#1413](https://github.com/telefonicaid/iotagent-node-lib/issues/1413))
+-   cgroup literal in configuration groups management API ([#752](https://github.com/telefonicaid/iotagent-node-lib/issues/752))
 -   Add init and improve log traces ([#1452](https://github.com/telefonicaid/iotagent-node-lib/issues/1452))
 -   Remove InMemory registry for Devices and Groups ([#1429](https://github.com/telefonicaid/iotagent-node-lib/issues/1429))
 
@@ -37,7 +34,6 @@ product:
 The following list of features are planned to be addressed in the medium term, typically within the subsequent
 release(s) generated in the next 9 months after the next planned release:
 
--   Allow to add metadata to attributes from measures ([#1453](https://github.com/telefonicaid/iotagent-node-lib/issues/1453))
 -   Cache support ([#1467](https://github.com/telefonicaid/iotagent-node-lib/issues/1467))
 -   MQTT per group advanced configuration (MQTT broker, topics) ([#1454](https://github.com/telefonicaid/iotagent-node-lib/issues/1454))
 -   Subscription based commands ([#1455](https://github.com/telefonicaid/iotagent-node-lib/issues/1455))
@@ -59,6 +55,10 @@ us if you wish to get involved in the implementation or influence the roadmap:
 
 The following list contains all features that were in the roadmap and have already been implemented.
 
+-   Allow to add metadata to attributes from measures ([#1453](https://github.com/telefonicaid/iotagent-node-lib/issues/1453)) ([4.6.0](https://github.com/telefonicaid/iotagent-node-lib/releases/tag/4.0.0))
+-   Refactor Append Mode & initial entity ([#1413](https://github.com/telefonicaid/iotagent-node-lib/issues/1413)) ([4.0.0](https://github.com/telefonicaid/iotagent-node-lib/releases/tag/4.0.0))
+-   Native support for NGSI-v2 and LD ingestion ([#1451](https://github.com/telefonicaid/iotagent-node-lib/issues/1451)) ([4.0.0](https://github.com/telefonicaid/iotagent-node-lib/releases/tag/4.0.0))
+-   Remove plugins structure (bidirectional plugin) ([#1413](https://github.com/telefonicaid/iotagent-node-lib/issues/1413)) ([4.0.0](https://github.com/telefonicaid/iotagent-node-lib/releases/tag/4.0.0))
 -   Refactor entities-NGSI-v2.js module ([#1166](https://github.com/telefonicaid/iotagent-node-lib/issues/1166)) ([3.0.0](https://github.com/telefonicaid/iotagent-node-lib/releases/tag/3.0.0))
 -   Accept JEXL Expressions for entity name in autoprovisioned devices (entityNameExp) ([#1145](https://github.com/telefonicaid/iotagent-node-lib/issues/1145)) ([2.22.0](https://github.com/telefonicaid/iotagent-node-lib/releases/tag/2.22.0))
 -   Improve command functionalities (binary data + expression + mapping) ([2.22.0](https://github.com/telefonicaid/iotagent-node-lib/releases/tag/2.22.0))

package/docker/Mosquitto/Dockerfile

@@ -1,4 +1,4 @@
-ARG  IMAGE_TAG=12.5-slim
+ARG  IMAGE_TAG=12.6-slim
 FROM debian:${IMAGE_TAG}
 
 ARG CLEAN_DEV_TOOLS
@@ -16,7 +16,7 @@ RUN \
     # Install dependencies
     apt-get -y install \
        wget \
-       mosquitto && \
+       mosquitto mosquitto-clients && \
     cp /etc/mosquitto/mosquitto.conf /etc/mosquitto/mosquitto.conf.orig && \
     chmod 755 /bin/startMosquitto.sh && \
     mkdir -p /var/log/mosquitto && \

package/docker/Mosquitto/README.md

@@ -1,13 +1,16 @@
-Thi directory containts the Dockerfile (and associated files) for a container of [mosquitto MQTT broker](https://mosquitto.org).
-This container is provide as a help for users to test with MQTT, but it is just an auxiliary material in this repository.
+Thi directory containts the Dockerfile (and associated files) for a container of
+[mosquitto MQTT broker](https://mosquitto.org). This container is provide as a help for users to test with MQTT, but it
+is just an auxiliary material in this repository.
 
 The following releases matches with eclipse-mosquitto version:
-- 2.1.0 uses mosquitto-2.0.11 from Debian 12
-- 2.0.0 uses mosquitto-2.0.11 from Debian 11
-- 1.6.0 uses mosquitto-1.6.10-1.el7.x86_64 (from Centos7)
-- 1.5.0 uses mosquitto-1.6.10-1.el7.x86_64 (from Centos7)
-- 1.4.0 uses mosquitto-1.6.10-1.el7.x86_64 (from Centos7)
-- 1.3.0 uses mosquitto-1.6.8-1.el7.x86_64 (from Centos7)
-- 1.2.0 uses mosquitto-1.6.7-1.el7.x86_64 (from Centos7)
-- 1.1.0 uses mosquitto-1.5.8-1.el7.x86_64 (from Centos7)
-- 1.0.0 uses mosquitto-1.4.8-1.el7.x86_64 (from Centos7)
+
+-   2.2.0 uses mosquitto-2.0.11 from Debian 12
+-   2.1.0 uses mosquitto-2.0.11 from Debian 12
+-   2.0.0 uses mosquitto-2.0.11 from Debian 11
+-   1.6.0 uses mosquitto-1.6.10-1.el7.x86_64 (from Centos7)
+-   1.5.0 uses mosquitto-1.6.10-1.el7.x86_64 (from Centos7)
+-   1.4.0 uses mosquitto-1.6.10-1.el7.x86_64 (from Centos7)
+-   1.3.0 uses mosquitto-1.6.8-1.el7.x86_64 (from Centos7)
+-   1.2.0 uses mosquitto-1.6.7-1.el7.x86_64 (from Centos7)
+-   1.1.0 uses mosquitto-1.5.8-1.el7.x86_64 (from Centos7)
+-   1.0.0 uses mosquitto-1.4.8-1.el7.x86_64 (from Centos7)

package/lib/commonConfig.js

@@ -157,7 +157,9 @@ function processEnvironmentVariables() {
         'IOTA_FALLBACK_PATH',
         'IOTA_LD_SUPPORT_NULL',
         'IOTA_LD_SUPPORT_DATASET_ID',
-        'IOTA_EXPRESS_LIMIT'
+        'IOTA_EXPRESS_LIMIT',
+        'IOTA_USE_CB_FLOW_CONTROL',
+        'IOTA_STORE_LAST_MEASURE'
     ];
     const iotamVariables = [
         'IOTA_IOTAM_URL',
@@ -350,6 +352,11 @@ function processEnvironmentVariables() {
         config.defaultResource = process.env.IOTA_DEFAULT_RESOURCE;
     }
 
+    // Default transport
+    if (process.env.IOTA_DEFAULT_TRANSPORT !== undefined) {
+        config.defaultTransport = process.env.IOTA_DEFAULT_TRANSPORT;
+    }
+
     // Default explicitAttrs
     if (process.env.IOTA_EXPLICIT_ATTRS !== undefined) {
         config.explicitAttrs = process.env.IOTA_EXPLICIT_ATTRS;
@@ -474,6 +481,16 @@ function processEnvironmentVariables() {
     } else {
         config.expressLimit = config.expressLimit ? config.expressLimit : '1mb';
     }
+    if (process.env.IOTA_USE_CB_FLOW_CONTROL) {
+        config.useCBflowControl = process.env.IOTA_USE_CB_FLOW_CONTROL === 'true';
+    } else {
+        config.useCBflowControl = config.useCBflowControl === true;
+    }
+    if (process.env.IOTA_STORE_LAST_MEASURE) {
+        config.storeLastMeasure = process.env.IOTA_STORE_LAST_MEASURE === 'true';
+    } else {
+        config.storeLastMeasure = config.storeLastMeasure === true;
+    }
 }
 
 function setConfig(newConfig) {
@@ -503,7 +520,9 @@ function getConfigForTypeInformation() {
         multiCore: config.multiCore,
         relaxTemplateValidation: config.relaxTemplateValidation,
         defaultEntityNameConjunction: config.defaultEntityNameConjunction,
-        defaultType: config.defaultType
+        defaultType: config.defaultType,
+        useCBflowControl: config.useCBflowControl,
+        storeLastMeasure: config.storeLastMeasure
     };
     return conf;
 }

package/lib/constants.js

@@ -88,5 +88,8 @@ module.exports = {
     ATTRIBUTE_DEFAULT,
     DATETIME_DEFAULT,
 
+    CONFIGGROUP_TERM: 'groups',
+    CONFIGGROUP_API_PATH: '/iot/groups',
+
     getInitialValueForType
 };

package/lib/fiware-iotagent-lib.js

@@ -45,22 +45,19 @@ const context = {
     op: 'IoTAgentNGSI.Global'
 };
 
+/* eslint-disable-next-line  no-unused-vars */
 function activateStatLogs(newConfig, callback) {
-    if (newConfig.stats && newConfig.stats.interval) {
-        async.series(
-            [
-                apply(statsRegistry.globalLoad, {
-                    deviceCreationRequests: 0,
-                    deviceRemovalRequests: 0,
-                    measureRequests: 0
-                }),
-                apply(statsRegistry.addTimerAction, statsRegistry.logStats)
-            ],
-            callback
-        );
-    } else {
-        callback();
-    }
+    async.series([
+        apply(statsRegistry.globalLoad, {
+            deviceCreationRequests: 0,
+            deviceRemovalRequests: 0,
+            measureRequests: 0,
+            raiseAlarm: 0,
+            releaseAlarm: 0,
+            updateEntityRequestsOk: 0,
+            updateEntityRequestsError: 0
+        })
+    ], callback);
 }
 
 /**

package/lib/jexlTranformsMap.js

@@ -80,7 +80,9 @@ const map = {
     localestring: (d, timezone, options) => new Date(d).toLocaleString(timezone, options),
     now: () => Date.now(),
     hextostring: (val) =>
-        new TextDecoder().decode(new Uint8Array(val.match(/.{1,2}/g).map((byte) => parseInt(byte, 16))))
+        new TextDecoder().decode(new Uint8Array(val.match(/.{1,2}/g).map((byte) => parseInt(byte, 16)))),
+    valuePicker: (val,pick) => Object.entries(val).filter(([, v]) => v === pick).map(([k,]) => k),
+    valuePickerMulti: (val,pick) => Object.entries(val).filter(([, v]) => pick.includes(v)).map(([k,]) => k)
 };
 
 exports.map = map;

package/lib/model/Command.js

@@ -34,8 +34,8 @@ const Command = new Schema({
     creationDate: { type: Date, default: Date.now }
 });
 
-function load(db) {
-    module.exports.model = db.model('Command', Command);
+function load() {
+    module.exports.model = mongoose.model('Command', Command);
     module.exports.internalSchema = Command;
 }
 

package/lib/model/Device.js

@@ -53,11 +53,15 @@ const Device = new Schema({
     autoprovision: Boolean,
     explicitAttrs: Group.ExplicitAttrsType,
     ngsiVersion: String,
-    payloadType: String
+    payloadType: String,
+    useCBflowControl: Boolean,
+    storeLastMeasure: Boolean,
+    lastMeasure: Object,
+    oldCtxt: Object
 });
 
-function load(db) {
-    module.exports.model = db.model('Device', Device);
+function load() {
+    module.exports.model = mongoose.model('Device', Device);
     module.exports.internalSchema = Device;
 }
 

package/lib/model/Group.js

@@ -65,12 +65,14 @@ const Group = new Schema({
     defaultEntityNameConjunction: String,
     ngsiVersion: String,
     entityNameExp: String,
-    payloadType: String
+    payloadType: String,
+    useCBflowControl: Boolean,
+    storeLastMeasure: Boolean
 });
 
-function load(db) {
+function load() {
     Group.index({ apikey: 1, resource: 1 }, { unique: true });
-    module.exports.model = db.model('Group', Group);
+    module.exports.model = mongoose.model('Group', Group);
     module.exports.internalSchema = Group;
 }