iotagent-node-lib 2.20.0 → 2.21.0

package/.readthedocs.yml

@@ -4,4 +4,6 @@ mkdocs:
   configuration: mkdocs.yml
 
 python:
-  version: 3.6
+  version: 3.8
+  install:
+    - requirements: doc/requirements.txt

package/README.md

@@ -20,7 +20,7 @@ platform (authentication and authorization of the channel) and provide other com
 This project is part of [FIWARE](https://www.fiware.org/). For more information check the FIWARE Catalogue entry for the
 [IoT Agents](https://github.com/Fiware/catalogue/tree/master/iot-agents).
 
-| :books: [Documentation](https://iotagent-node-lib.rtfd.io) | :mortar_board: [Academy](https://fiware-academy.readthedocs.io/en/latest/iot-agents/idas) | :dart: [Roadmap](https://github.com/telefonicaid/iotagent-node-lib/blob/master/docs/roadmap.md) |
+| :books: [Documentation](https://iotagent-node-lib.rtfd.io) | :mortar_board: [Academy](https://fiware-academy.readthedocs.io/en/latest/iot-agents/idas) | :dart: [Roadmap](https://github.com/telefonicaid/iotagent-node-lib/blob/master/doc/roadmap.md) |
 | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
 
 
@@ -304,7 +304,7 @@ deleted upon startup.
 
 The IoT Agent Node Library is licensed under [Affero General Public License (GPL) version 3](./LICENSE).
 
-© 2019 Telefonica Investigación y Desarrollo, S.A.U
+© 2022 Telefonica Investigación y Desarrollo, S.A.U
 
 ### Are there any legal issues with AGPL 3.0? Is it safe for me to use?
 

package/doc/advanced-topics.md

@@ -305,8 +305,8 @@ stored in the Context Broker by adding a new attribute to the entity with the sa
 element. By adding the field `explicitAttrs` with `true` value to device or group provision, the IoTAgent rejects the
 measure elements that are not defined in the mappings of device or group provision, persisting only the one defined in
 the mappings of the provision. If `explicitAttrs` is provided both at device and group level, the device level takes
-precedence. Additionally `explicitAttrs` can be used to define which meassures defined in JSON/JEXL array will be
-propagated to NGSI interface.
+precedence. Additionally `explicitAttrs` can be used to define which meassures (identified by their attribute names, not 
+by their object_id) defined in JSON/JEXL array will be propagated to NGSI interface.
 
 The different possibilities are summarized below:
 
@@ -332,8 +332,8 @@ Case 3:
 "explicitAttrs": "['attr1','atrr2']"
 ```
 
-just measures defined in the array will be will be propagated to NGSI interface (note that in this case the value of
-`explicitAttrs` is not a JSON but a string that looks likes a JSON).
+just measures defined in the array (identified by their attribute names, not by their object_id) will be will be 
+propagated to NGSI interface (note that in this case the value of `explicitAttrs` is not a JSON but a string that looks likes a JSON).
 
 Case 4:
 
@@ -346,8 +346,8 @@ depending on the JEXL expression evaluation:
 -   If it evaluates to `true` every measure will be propagated to NGSI interface (as in case 1)
 -   If it evaluates to `false` just measures defined in active, static (plus conditionally TimeInstant) will be
     propagated to NGSI interface (as in case 2)
--   If it evaluates to an array just measures defined in the array will be will be propagated to NGSI interface (as in
-    case 3)
+-   If it evaluates to an array just measures defined in the array (identified by their attribute names, not by their object_id)
+    will be will be propagated to NGSI interface (as in case 3)
 
 ### Configuring operation to persist the data in Context Broker (appendMode)
 

package/doc/api.md

@@ -247,7 +247,7 @@ the API resource fields and the same fields in the database model.
 
 #### Attribute lists
 
-In the device model there are three list of attributes that can be declared: attributes, lazy and commands. All of them
+In the group/device model there are three list of attributes that can be declared: attributes, lazy and commands. All of them
 have the same syntax, an object containing the following attributes:
 
 -   **object_id** (optional): name of the attribute as coming from the device.
@@ -255,7 +255,7 @@ have the same syntax, an object containing the following attributes:
 -   **type** (mandatory): name of the type of the attribute in the target entity.
 -   **metadata** (optional): additional static metadata for the attribute in the target entity. (e.g. `unitCode`)
 
-Some transformation plugins also allow the use of the following optional attributes:
+Some transformation plugins also allow the use of the following optional fields:
 
 -   **expression**: indicates that the value of the target attribute will not be the plain value or the measurement, but
     an expression based on a combination of the reported values. See the
@@ -268,6 +268,15 @@ Some transformation plugins also allow the use of the following optional attribu
 -   **reverse**: add bidirectionality expressions to the attribute. See the **bidirectionality** transformation plugin
     in the [Data Mapping Plugins section](advanced-topics.md#bidirectionality-plugin-bidirectional) for details.
 
+Additionally for commands (which are attributes of type `command`) the following fields are optional:
+
+-   **expression** indicates that the value of the target command will not be the plain value or the command, but an
+    expression based on a combination of the returned values. See the
+    [Expression Language definition](expressionLanguage.md) for details
+-   **payloadType**: indicates how command payload will be transformed before be sent to device. Please have a look to particular 
+    IOTAs documentation for allowed values of this field in each case.
+-   **contentType**: `content-type` header used when send command by HTTP transport (ignored in other kinds of transports)
+
 See the transformation plugins Section for more details.
 
 #### Advice on Attribute defintions

package/doc/expressionLanguage.md

@@ -474,7 +474,10 @@ Current common transformation set:
 | slice: (arr, init, end)          | `arr.slice(init,end);`                                                                                                  |
 | addset: (arr, x)                 | <code>{ return Array.from((new Set(arr)).add(x)) }</code>                                                               |
 | removeset: (arr, x)              | <code>{ let s = new Set(arr); s.delete(x); return Array.from(s) }</code>                                                |
+| touppercase: (val)               | `String(val).toUpperCase()`                                                                                            |
+| tolowercase: (val)               | `String(val).toLowerCase()`                                                                                            |
 
+  
 You have available this [JEXL interactive playground][99] with all the transformations already loaded, in which you can
 test all the functions described above.
 

package/doc/northboundinteractions.md

@@ -163,16 +163,15 @@ The equivalent **NGSI-LD** payload is associated to an update operation (PATCH `
 
 ```json
 {
-     "temperature":  {
-          "type": "Property",
-          "value": "23"
-      },
-      "pressure" :{
-          "type": "Property",
-          "value": "720"
-      }
+    "temperature": {
+        "type": "Property",
+        "value": "23"
+    },
+    "pressure": {
+        "type": "Property",
+        "value": "720"
+    }
 }
-
 ```
 
 #### R1 - UpdateContext (response)
@@ -285,7 +284,8 @@ This is an **NGSI-LD** response to `/ngsi-ld/v1/entities/<entity>`
 ```
 
 In this case, the response to the QueryContext is a list of responses, one for each requested entity, indicating whether
-the information has been retrieved successfully (in the HTTP Status code) and the requested Context information in the body of the response.
+the information has been retrieved successfully (in the HTTP Status code) and the requested Context information in the
+body of the response.
 
 Application level errors can be specified for each entity in this payload.
 
@@ -305,7 +305,7 @@ Context Element, but with the request as a whole.
 
 ### Scenario 1: active attributes
 
-![General ](./img/scenario1.png "Scenario 1: active attributes")
+![General ](./img/scenario1.png 'Scenario 1: active attributes')
 
 In this scenario, the interaction is started by the device, that is going to actively send a piece of data to the
 platform. When the IoTAgent receives the data, it sends it to the Context Broker through a P1 request. The Context
@@ -319,7 +319,7 @@ updating process, and can occur at any time (they are to completely different pr
 
 ### Scenario 2: lazy attributes
 
-![General ](./img/scenario2.png "Scenario 2: lazy attributes")
+![General ](./img/scenario2.png 'Scenario 2: lazy attributes')
 
 This scenario requires that the attributes that are going to be requested are marked as provided by the IoT Agent,
 through a registration process (NGSIv9). Examples of this registration process will be provided in the practical section
@@ -345,7 +345,7 @@ queries (and thus P2 and R2 payloads).
 
 ### Scenario 3: commands
 
-![General ](./img/scenario3.png "Scenario 3: commands")
+![General ](./img/scenario3.png 'Scenario 3: commands')
 
 This scenario requires that the attributes that are going to be requested are marked as provided by the IoT Agent,
 through a registration process (NGSIv9). Examples of this registration process will be provided in the practical section
@@ -359,15 +359,26 @@ use three kinds of attributes:
 -   An attribute will be used as the _input attribute_ (the attribute registered in the Context Provider). This input
     attribute can be thought of as a command issued to the IoTAgent (from here the name of the scenario) whose value is
     the set of arguments of the command. Only updateContext operations will be used to interact with this attributes.
+    Typically this attribute is of type `command`.
 
 -   Another attribute will be used as the _result attribute_. This attribute will be updated from the IoTAgent, and its
     value stored in the Context Broker. This attribute will contain the result of the command (this result can be
     information in case the command was a "information retrieval" command or the result of an action if it was an
-    "actuator command"). Typically, the name of this attribute will be the same of the input attribute, with an
-    additional sufix (`_info`).
+    "actuator command"). Initially its value is empty. Typically, the name of this attribute will be the same of the
+    input attribute, with an additional sufix (`_info`) and the type is `commandResult`.
 
 -   Another attribute with the same characteristics as the later will be used to indicate whether the command has ended
-    successfully or whether an error has been reported.
+    successfully or whether an error has been reported. Typically, the name of this attribute will be the same of the
+    input attribute, with an additional sufix (`_status`) and the type is `commandStatus`. The possible values of this
+    attribute are: `ERROR`, `EXPIRED`, `PENDING`, `DELIVERED`, `UNKNOWN` with the following meanings:
+    -   ERROR: There is a kind of error.
+    -   EXPIRED: This meens that pull command has been expired without be delivered to device according with
+        `pollingExpiration` time defined by config.
+    -   PENDING: In a PUSH command means that command has been sent to device but not device has still not respond. In a
+        PULL command means that command has been stored and device still has no ask for it.
+    -   DELIVERED: The command has been delivered to phisical device.
+    -   OK: The command has been delivered and device has respond.
+    -   UNKNOWN: This is the initial value.
 
 In this scenario, the interaction is also initiated by the User. The user starts the scenario by sending an update
 request P1 to the Context Broker, to the input attribute (1). The Context Broker redirects this same payload to the
@@ -835,7 +846,6 @@ The IoT Agent detects the selected attribute is a command, and replies to the Co
 ```json
 [
     {
-
         "type": "device",
         "id": "Dev0001",
         "switch": {
@@ -854,7 +864,6 @@ The Context Broker, forwards the same response to the user, thus replying the or
 ```json
 [
     {
-
         "type": "device",
         "id": "Dev0001",
         "switch": {
@@ -882,11 +891,11 @@ curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" -
             "isPattern": "false",
             "id": "Dev0001",
             "switch_info": {
-                "type": "command_info",
+                "type": "commandResult",
                 "value": "Switched successfully!"
             },
             "switch_status": {
-                "type": "command_status",
+                "type": "commandStatus",
                 "value": "OK"
             }
         }
@@ -906,11 +915,11 @@ The Context Broker replies to the IoT Agent with a R1 payload (200 OK):
         "type": "device",
         "id": "Dev0001",
         "switch_info": {
-            "type": "command_info",
+            "type": "commandResult",
             "value": ""
         },
-        "switch_status":  {
-            "type": "command_status",
+        "switch_status": {
+            "type": "commandStatus",
             "value": ""
         }
     }
@@ -936,8 +945,8 @@ curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" -
         }
     ],
     "attributes": [
-    	"switch_info",
-    	"switch_status"
+        "switch_info",
+        "switch_status"
     ]
 }' "https://<platform-ip>:10027/v2/op/query"
 ```
@@ -947,20 +956,18 @@ The Context Broker replies with all the desired data, in R2 format (200 OK):
 ```json
 [
     {
-
         "type": "device",
         "id": "Dev0001",
         "switch_info": {
-            "type": "command_info",
+            "type": "commandResult",
             "value": "Switched successfully!"
         },
         "switch_status": {
-            "type": "command_status",
+            "type": "commandStatus",
             "value": "OK"
         }
     }
 ]
-
 ```
 
 ### Scenario 3: commands (error)
@@ -978,11 +985,11 @@ curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" -
             "isPattern": "false",
             "id": "Dev0001",
             "switch_info":{
-                "type": "command_info",
+                "type": "commandResult",
                 "value": "The switch could not be switched due to the following error: switch blocked"
             },
             "switch_status":{
-                "type": "command_status",
+                "type": "commandStatus",
                 "value": "ERROR"
             }
         }
@@ -999,11 +1006,11 @@ In this case, the Context Broker reply with the following response (200 OK):
         "type": "device",
         "id": "Dev0001",
         "switch_info": {
-            "type": "command_info",
+            "type": "commandResult",
             "value": ""
         },
         "switch_status": {
-            "type": "command_status",
+            "type": "commandStatus",
             "value": ""
         }
     }

package/doc/requirements.txt

@@ -0,0 +1,4 @@
+mkdocs==1.2.3
+Pygments==2.9.0
+Markdown==3.3.4
+jinja2==3.0.0

package/{docs → doc}/roadmap.md

@@ -13,7 +13,7 @@ only, and this section may be revised to provide newer information at any time.
 
 Disclaimer:
 
--   This section has been last updated in March 2020. Please take into account its content could be obsolete.
+-   This section has been last updated in March 2022. Please take into account its content could be obsolete.
 -   Note we develop this software in Agile way, so development plan is continuously under review. Thus, this roadmap has
     to be understood as rough plan of features to be done along time which is fully valid only at the time of writing
     it. This roadmap has not be understood as a commitment on features and/or dates.
@@ -25,19 +25,17 @@ Disclaimer:
 The following list of features are planned to be addressed in the short term, and incorporated in a release of the
 product:
 
--   Selectively ignore measure in the southbound interface (community)
--   JEXL support in expressions (community)
 -   cgroup literal in configuration groups management API (community)
 -   Metadata processing improvements
--   Add MongoDB authentication support
+-   Improve command functionalities (binary data + expression + mapping)
 
 ### Medium term
 
 The following list of features are planned to be addressed in the medium term, typically within the subsequent
 release(s) generated in the next 9 months after the next planned release:
 
--   Support for "delta" measures (i.e. "temperature _increased_ in 5 degress" instead of "temperature _is_ 25")
--   Removal support for NGSIv1 (which currently is deprecated)
+-   Accept JEXL Expressions for entity name in autoprovisioned devices (#1145)
+-   Refactor entities-NGSI-v2.js module (#1166)
 
 ### Long term
 
@@ -48,3 +46,20 @@ us if you wish to get involved in the implementation or influence the roadmap:
 -   Incremental introduccion of ECMAScript6 syntax (previous analysis of which sub-set of interesting aspect we want to
     take)
 -   Use the lightweight ingestion mechanism for connection oriented updates implemented in Context Broker
+-   Add support to other transport protocols (BacNET, Modbus, etc)
+
+### Features already completed
+
+The following list contains all features that were in the roadmap and have already been implemented.
+
+-   Support for "delta" measures (i.e. "temperature _increased_ in 5 degress" instead of "temperature _is_ 25")
+-   Allow to handle binary messages ([iota-ul#530](https://github.com/telefonicaid/iotagent-ul/issues/530))
+-   Removal support for NGSIv1 (#966) ([2.18.0](https://github.com/telefonicaid/iotagent-node-lib/releases/tag/2.18.0))
+-   Selectively ignore measure in the southbound interface
+    ([iotagent-json#416](https://github.com/telefonicaid/iotagent-json/issues/416),
+    [iotagent-ul#372](https://github.com/telefonicaid/iotagent-ul/issues/372))
+    ([2.13.0](https://github.com/telefonicaid/iotagent-node-lib/releases/tag/2.13.0))
+-   JEXL support in expressions (#801, #687, #868)
+    ([2.13.0](https://github.com/telefonicaid/iotagent-node-lib/releases/tag/2.13.0))
+-   Add MongoDB authentication support (#844)
+    ([2.12.0](https://github.com/telefonicaid/iotagent-node-lib/releases/tag/2.12.0))

package/doc/usermanual.md

@@ -209,8 +209,8 @@ function setCommandResult(entityName, resource, apikey, commandName, commandResu
 ###### Description
 
 Update the result of a command in the Context Broker. The result of the command has two components: the result of the
-command itself will be represented with the suffix `_result` in the entity while the status is updated in the attribute
-with the `_info` suffix.
+command itself will be represented with the suffix `_info` in the entity while the status is updated in the attribute
+with the `_status` suffix.
 
 ###### Params
 

package/docker/Mosquitto/Dockerfile

@@ -1,19 +1,35 @@
-FROM centos:7.9.2009
+ARG  IMAGE_TAG=11.2-slim
+FROM debian:${IMAGE_TAG}
 
-COPY aclfile /root/
-COPY startMosquitto.sh /bin
+ARG CLEAN_DEV_TOOLS
+ENV CLEAN_DEV_TOOLS ${CLEAN_DEV_TOOLS:-1}
 
 ENV CONGIF_FROM_ENV true
 
-RUN yum update -y && yum install -y wget \
-  && yum install -y epel-release \
-  && yum update -y epel-release \
-  && yum install -y mosquitto \
-  && cp /etc/mosquitto/mosquitto.conf /etc/mosquitto/mosquitto.conf.orig \
-  && chmod 755 /bin/startMosquitto.sh \
-  && mkdir /var/log/mosquitto \
-  && chown mosquitto:mosquitto /var/log/mosquitto \
-  && yum clean all
+COPY aclfile /root/
+COPY startMosquitto.sh /bin
+
+RUN \
+    # Install security updates
+    apt-get -y update && \
+    apt-get -y upgrade && \
+    # Install dependencies
+    apt-get -y install \
+       wget \
+       mosquitto && \
+    cp /etc/mosquitto/mosquitto.conf /etc/mosquitto/mosquitto.conf.orig && \
+    chmod 755 /bin/startMosquitto.sh && \
+    mkdir -p /var/log/mosquitto && \
+    chown mosquitto:mosquitto /var/log/mosquitto && \
+    mkdir -p /var/run/mosquitto/ && \
+    chown mosquitto:mosquitto /var/run/mosquitto && \
+    echo "INFO: Cleaning unused software..." && \
+    apt-get clean && \
+    apt-get -y autoremove --purge && \
+    if [ ${CLEAN_DEV_TOOLS} -eq 0 ] ; then exit 0 ; fi && \
+    # remove the same packages we installed at the beginning to build Orch
+    apt-get -y autoremove --purge \
+        wget
 
 
 EXPOSE 1883

package/docker/Mosquitto/README.md

@@ -2,10 +2,11 @@ Thi directory containts the Dockerfile (and associated files) for a container of
 This container is provide as a help for users to test with MQTT, but it is just an auxiliary material in this repository.
 
 The following releases matches with eclipse-mosquitto version:
-- 1.6.0 uses mosquitto-1.6.10-1.el7.x86_64
-- 1.5.0 uses mosquitto-1.6.10-1.el7.x86_64
-- 1.4.0 uses mosquitto-1.6.10-1.el7.x86_64
-- 1.3.0 uses mosquitto-1.6.8-1.el7.x86_64
-- 1.2.0 uses mosquitto-1.6.7-1.el7.x86_64
-- 1.1.0 uses mosquitto-1.5.8-1.el7.x86_64
-- 1.0.0 uses mosquitto-1.4.8-1.el7.x86_64
+- 2.0.0 uses mosquitto-2.0.11 from Debian 11
+- 1.6.0 uses mosquitto-1.6.10-1.el7.x86_64 (from Centos7)
+- 1.5.0 uses mosquitto-1.6.10-1.el7.x86_64 (from Centos7)
+- 1.4.0 uses mosquitto-1.6.10-1.el7.x86_64 (from Centos7)
+- 1.3.0 uses mosquitto-1.6.8-1.el7.x86_64 (from Centos7)
+- 1.2.0 uses mosquitto-1.6.7-1.el7.x86_64 (from Centos7)
+- 1.1.0 uses mosquitto-1.5.8-1.el7.x86_64 (from Centos7)
+- 1.0.0 uses mosquitto-1.4.8-1.el7.x86_64 (from Centos7)

package/docker/Mosquitto/startMosquitto.sh

@@ -1,7 +1,10 @@
 #!/usr/bin/env bash
 
+echo "INFO: startMosquitto..."
+
 if [ "${CONGIF_FROM_ENV}" = true ] ; then
     cp /etc/mosquitto/mosquitto.conf.orig /etc/mosquitto/mosquitto.conf
+    sed -i 's/log_dest file \/var\/log\/mosquitto\/mosquitto.log/log_dest stderr/g' /etc/mosquitto/mosquitto.conf
     echo "log_timestamp true" >> /etc/mosquitto/mosquitto.conf
     echo "log_timestamp_format %Y-%m-%dT%H:%M:%S" >> /etc/mosquitto/mosquitto.conf
     echo 'listener 9001' >> /etc/mosquitto/mosquitto.conf
@@ -18,4 +21,9 @@ if [ "${CONGIF_FROM_ENV}" = true ] ; then
     fi
 fi
 
+echo "INFO: content /etc/mosquitto/mosquitto.conf: "
+cat /etc/mosquitto/mosquitto.conf
+
+echo "INFO: start: startMosquitto -c /etc/mosquitto/mosquitto.conf"
+
 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

package/lib/fiware-iotagent-lib.js

@@ -314,6 +314,7 @@ exports.listDevices = deviceService.listDevices;
 exports.getDevice = deviceService.getDevice;
 exports.getDeviceSilently = deviceService.getDeviceSilently;
 exports.getDeviceByName = deviceService.getDeviceByName;
+exports.getDeviceByNameAndType = deviceService.getDeviceByNameAndType;
 exports.getDevicesByAttribute = deviceService.getDevicesByAttribute;
 exports.mergeDeviceWithConfiguration = deviceService.mergeDeviceWithConfiguration;
 exports.retrieveDevice = deviceService.retrieveDevice;

package/lib/jexlTranformsMap.js

@@ -65,7 +65,9 @@ const map = {
         let s = new Set(arr);
         s.delete(x);
         return Array.from(s);
-    }
+    },
+    touppercase: (val) => String(val).toUpperCase(),
+    tolowercase: (val) => String(val).toLowerCase()
 };
 
 exports.map = map;

package/lib/plugins/expressionPlugin.js

@@ -30,6 +30,8 @@ const jexlParser = require('./jexlParser');
 const config = require('../commonConfig');
 /* eslint-disable no-unused-vars */
 const logger = require('logops');
+const errors = require('../errors');
+const constants = require('../constants');
 const context = {
     op: 'IoTAgentNGSI.expressionPlugin'
 };
@@ -42,6 +44,22 @@ function setJEXLTransforms(transformationMap) {
     jexlParser.setTransforms(transformationMap);
 }
 
+function applyExpression(expression, context, typeInformation) {
+    let parser = legacyParser;
+    if (checkJexl(typeInformation)) {
+        parser = jexlParser;
+    }
+    return parser.applyExpression(expression, context, typeInformation);
+}
+
+function extractContext(attributeList, typeInformation) {
+    let parser = legacyParser;
+    if (checkJexl(typeInformation)) {
+        parser = jexlParser;
+    }
+    return parser.extractContext(attributeList);
+}
+
 function mergeAttributes(attrList1, attrList2) {
     const finalCollection = _.clone(attrList1);
     const additionalItems = [];
@@ -70,26 +88,26 @@ function mergeAttributes(attrList1, attrList2) {
     return finalCollection.concat(additionalItems);
 }
 
-function update(entity, typeInformation, callback) {
-    function checkJexl(typeInformation) {
-        if (
-            config.getConfig().defaultExpressionLanguage === 'jexl' &&
-            typeInformation.expressionLanguage &&
-            typeInformation.expressionLanguage !== 'legacy'
-        ) {
-            return true;
-        } else if (config.getConfig().defaultExpressionLanguage === 'jexl' && !typeInformation.expressionLanguage) {
-            return true;
-        } else if (
-            config.getConfig().defaultExpressionLanguage === 'legacy' &&
-            typeInformation.expressionLanguage &&
-            typeInformation.expressionLanguage === 'jexl'
-        ) {
-            return true;
-        }
-        return false;
+function checkJexl(typeInformation) {
+    if (
+        config.getConfig().defaultExpressionLanguage === 'jexl' &&
+        typeInformation.expressionLanguage &&
+        typeInformation.expressionLanguage !== 'legacy'
+    ) {
+        return true;
+    } else if (config.getConfig().defaultExpressionLanguage === 'jexl' && !typeInformation.expressionLanguage) {
+        return true;
+    } else if (
+        config.getConfig().defaultExpressionLanguage === 'legacy' &&
+        typeInformation.expressionLanguage &&
+        typeInformation.expressionLanguage === 'jexl'
+    ) {
+        return true;
     }
+    return false;
+}
 
+function update(entity, typeInformation, callback) {
     function processEntityUpdateNgsi2(attributes) {
         let parser = legacyParser;
         if (checkJexl(typeInformation)) {
@@ -114,15 +132,31 @@ function update(entity, typeInformation, callback) {
 
     try {
         logger.debug(context, 'expressionPlugin entity %j', entity);
-        let attsArray = utils.extractAttributesArrayFromNgsi2Entity(entity);
-        attsArray = processEntityUpdateNgsi2(attsArray);
-        entity = utils.createNgsi2Entity(entity.id, entity.type, attsArray, true);
+        const attsArray = utils.extractAttributesArrayFromNgsi2Entity(entity);
+        // Exclude processing all attr expressions when current attr is of type 'commandStatus' or 'commandResult'
+        const attsArrayFiltered = attsArray.filter((obj) => {
+            return ![constants.COMMAND_STATUS, constants.COMMAND_RESULT].includes(obj.type);
+        });
+        const attsArrayCmd = attsArray.filter((obj) => {
+            // just attr of type 'commandStatus' or 'commandResult'
+            return [constants.COMMAND_STATUS, constants.COMMAND_RESULT].includes(obj.type);
+        });
+        let attsArrayFinal = [];
+        if (attsArrayFiltered.length > 0) {
+            attsArrayFinal = processEntityUpdateNgsi2(attsArrayFiltered);
+        }
+        attsArrayFinal = attsArrayFinal.concat(attsArrayCmd);
+        entity = utils.createNgsi2Entity(entity.id, entity.type, attsArrayFinal, true);
 
         callback(null, entity, typeInformation);
     } catch (e) {
+        logger.error(context, 'expressionPlugin error %j procesing entity %j', e, entity);
         callback(e);
     }
 }
 
 exports.update = update;
 exports.setJEXLTransforms = setJEXLTransforms;
+exports.applyExpression = applyExpression;
+exports.extractContext = extractContext;
+exports.checkJexl = checkJexl;

package/lib/plugins/multiEntity.js

@@ -30,7 +30,7 @@ const _ = require('underscore');
 const constants = require('../constants');
 const legacyParser = require('./expressionParser');
 const jexlParser = require('./jexlParser');
-const config = require('../commonConfig');
+const expressionPlugin = require('./expressionPlugin');
 /* eslint-disable-next-line  no-unused-vars */
 const logger = require('logops');
 /* eslint-disable-next-line  no-unused-vars */
@@ -41,25 +41,6 @@ const utils = require('./pluginUtils');
 /* eslint-disable-next-line  no-unused-vars */
 const aliasPlugin = require('./attributeAlias');
 
-function checkJexl(typeInformation) {
-    if (
-        config.getConfig().defaultExpressionLanguage === 'jexl' &&
-        typeInformation.expressionLanguage &&
-        typeInformation.expressionLanguage !== 'legacy'
-    ) {
-        return true;
-    } else if (config.getConfig().defaultExpressionLanguage === 'jexl' && !typeInformation.expressionLanguage) {
-        return true;
-    } else if (
-        config.getConfig().defaultExpressionLanguage === 'legacy' &&
-        typeInformation.expressionLanguage &&
-        typeInformation.expressionLanguage === 'jexl'
-    ) {
-        return true;
-    }
-    return false;
-}
-
 function hasEntityName(item) {
     return item.entity_name;
 }
@@ -226,7 +207,7 @@ function propagateTimestamp(entity, entities) {
 
 function updateAttribute(entity, typeInformation, callback) {
     let parser = legacyParser;
-    if (checkJexl(typeInformation)) {
+    if (expressionPlugin.checkJexl(typeInformation)) {
         parser = jexlParser;
     }
     const attsArray = utils.extractAttributesArrayFromNgsi2Entity(entity);

package/lib/services/devices/deviceRegistryMemory.js

@@ -149,13 +149,19 @@ function getDevice(id, service, subservice, callback) {
     }
 }
 
-function getByName(name, service, subservice, callback) {
+function getByNameAndType(name, type, service, subservice, callback) {
     const devices = _.values(registeredDevices[service]);
     let device;
 
     for (let i = 0; i < devices.length; i++) {
         if (devices[i].name === name) {
-            device = devices[i];
+            if (type) {
+                if (devices[i].type === type) {
+                    device = devices[i];
+                }
+            } else {
+                device = devices[i];
+            }
         }
     }
 
@@ -166,6 +172,10 @@ function getByName(name, service, subservice, callback) {
     }
 }
 
+function getByName(name, service, subservice, callback) {
+    getByNameAndType(name, null, service, subservice, callback);
+}
+
 function update(device, callback) {
     registeredDevices[device.service][device.id] = deepClone(device);
     callback(null, device);
@@ -208,4 +218,5 @@ exports.list = listDevices;
 exports.get = getDevice;
 exports.getSilently = getDevice;
 exports.getByName = getByName;
+exports.getByNameAndType = getByNameAndType;
 exports.clear = clear;

package/lib/services/devices/deviceRegistryMongoDB.js

@@ -243,15 +243,18 @@ function getDevice(id, service, subservice, callback) {
     });
 }
 
-function getByName(name, service, servicepath, callback) {
+function getByNameAndType(name, type, service, servicepath, callback) {
     context = fillService(context, { service, subservice: servicepath });
-    logger.debug(context, 'Looking for device with name [%s].', name);
-
-    const query = Device.model.findOne({
-        name,
-        service,
+    let optionsQuery = {
+        name: name,
+        service: service,
         subservice: servicepath
-    });
+    };
+    if (type) {
+        optionsQuery.type = type;
+    }
+    logger.debug(context, 'Looking for device with [%j].', optionsQuery);
+    const query = Device.model.findOne(optionsQuery);
 
     query.select({ __v: 0 });
 
@@ -270,6 +273,10 @@ function getByName(name, service, servicepath, callback) {
     });
 }
 
+function getByName(name, service, servicepath, callback) {
+    getByNameAndType(name, null, service, servicepath, callback);
+}
+
 /**
  * Updates the given device into the database.
  * updated.
@@ -361,4 +368,5 @@ exports.list = alarmsInt(constants.MONGO_ALARM, listDevices);
 exports.get = alarmsInt(constants.MONGO_ALARM, getDevice);
 exports.getSilently = getDevice;
 exports.getByName = alarmsInt(constants.MONGO_ALARM, getByName);
+exports.getByNameAndType = alarmsInt(constants.MONGO_ALARM, getByNameAndType);
 exports.clear = alarmsInt(constants.MONGO_ALARM, clear);

package/lib/services/devices/deviceService.js

@@ -214,7 +214,13 @@ function findConfigurationGroup(deviceObj, callback) {
     } else {
         config
             .getGroupRegistry()
-            .findTypeSilently(deviceObj.service, deviceObj.subservice, deviceObj.type, deviceObj.apikey, handlerGroupFindByType);
+            .findTypeSilently(
+                deviceObj.service,
+                deviceObj.subservice,
+                deviceObj.type,
+                deviceObj.apikey,
+                handlerGroupFindByType
+            );
     }
 }
 
@@ -513,6 +519,18 @@ function getDeviceByName(deviceName, service, subservice, callback) {
     config.getRegistry().getByName(deviceName, service, subservice, callback);
 }
 
+/**
+ * Retrieve a device from the registry based on its entity name and type
+ *
+ * @param {String} deviceName       Name of the entity associated to a device.
+ * @param {String} deviceType       Type of the entity associated to a device.
+ * @param {String} service          Service the device belongs to.
+ * @param {String} subservice       Division inside the service.
+ */
+function getDeviceByNameAndType(deviceName, deviceType, service, subservice, callback) {
+    config.getRegistry().getByNameAndType(deviceName, deviceType, service, subservice, callback);
+}
+
 /**
  * Retrieve a device from the registry based on the value of a given attribute.
  *
@@ -581,7 +599,12 @@ function findOrCreate(deviceId, group, callback) {
                     callback(error, device, group);
                 });
             } else {
-                logger.info(context, 'Device %j not provisioned due autoprovision is disabled by its conf %j', newDevice, group);
+                logger.info(
+                    context,
+                    'Device %j not provisioned due autoprovision is disabled by its conf %j',
+                    newDevice,
+                    group
+                );
                 callback(new errors.DeviceNotFound(deviceId));
             }
         } else {
@@ -632,6 +655,7 @@ exports.getDevice = intoTrans(context, checkRegistry)(getDevice);
 exports.getDeviceSilently = intoTrans(context, checkRegistry)(getDeviceSilently);
 exports.getDevicesByAttribute = intoTrans(context, checkRegistry)(getDevicesByAttribute);
 exports.getDeviceByName = intoTrans(context, checkRegistry)(getDeviceByName);
+exports.getDeviceByNameAndType = intoTrans(context, checkRegistry)(getDeviceByNameAndType);
 exports.register = intoTrans(context, registerDevice);
 exports.updateRegister = intoTrans(context, updateRegisterDevice);
 exports.unregister = intoTrans(context, unregisterDevice);

package/lib/services/ngsi/entities-NGSI-LD.js

@@ -410,8 +410,9 @@ function addLinkedEntities(typeInformation, json) {
  * @param {Object} typeInformation  Configuration information for the device.
  */
 function removeHiddenAttrsFromMultiEntity(entities, typeInformation) {
-    var explicitAttrsList = [];
+    let explicitAttrsList;
     if (typeInformation.explicitAttrs) {
+        explicitAttrsList = [];
         explicitAttrsList.push('type');
         explicitAttrsList.push('id');
         if (typeof typeInformation.explicitAttrs === 'boolean') {
@@ -437,7 +438,7 @@ function removeHiddenAttrsFromMultiEntity(entities, typeInformation) {
             });
         }
     }
-    if (explicitAttrsList.length > 0) {
+    if (explicitAttrsList && explicitAttrsList.length >= 0) {
         entities.forEach((entity) => {
             const hidden = _.difference(_.keys(entity), explicitAttrsList);
             hidden.forEach((attr) => {
@@ -457,8 +458,9 @@ function removeHiddenAttrsFromMultiEntity(entities, typeInformation) {
 function removeHiddenAttrs(result, typeInformation) {
     delete result.id;
     delete result.type;
-    var explicitAttrsList = [];
+    let explicitAttrsList;
     if (typeInformation.explicitAttrs && typeof typeInformation.explicitAttrs === 'boolean') {
+        explicitAttrsList = [];
         if (typeInformation.timestamp) {
             explicitAttrsList.push(constants.TIMESTAMP_ATTRIBUTE);
         }
@@ -478,7 +480,7 @@ function removeHiddenAttrs(result, typeInformation) {
         const res = jexlParser.parse(typeInformation.explicitAttrs, ctx);
         explicitAttrsList = res;
     }
-    if (explicitAttrsList.length > 0) {
+    if (explicitAttrsList && explicitAttrsList.length >= 0) {
         const hidden = _.difference(_.keys(result), explicitAttrsList);
         hidden.forEach((attr) => {
             delete result[attr];

package/lib/services/ngsi/entities-NGSI-v2.js

@@ -321,8 +321,9 @@ function sendQueryValueNgsi2(entityName, attributes, typeInformation, token, cal
  * @param {Object} typeInformation  Configuration information for the device.
  */
 function removeHiddenAttrsFromMultiEntity(entities, typeInformation) {
-    var explicitAttrsList = [];
+    let explicitAttrsList;
     if (typeInformation.explicitAttrs) {
+        explicitAttrsList = [];
         explicitAttrsList.push('type');
         explicitAttrsList.push('id');
         if (typeof typeInformation.explicitAttrs === 'boolean') {
@@ -349,10 +350,10 @@ function removeHiddenAttrsFromMultiEntity(entities, typeInformation) {
             });
         }
     }
-    if (explicitAttrsList.length > 0) {
-        logger.debug(context, 'removeHiddenAttrsFromMultiEntity %s', explicitAttrsList);
+    if (explicitAttrsList && explicitAttrsList.length >= 0) {
         entities.forEach((entity) => {
             const hidden = _.difference(_.keys(entity), explicitAttrsList);
+            logger.debug(context, 'removeHiddenAttrsFromMultiEntity %s from entity %s', hidden, entity);
             hidden.forEach((attr) => {
                 delete entity[attr];
             });
@@ -369,8 +370,9 @@ function removeHiddenAttrsFromMultiEntity(entities, typeInformation) {
 function removeHiddenAttrs(result, typeInformation) {
     delete result.id;
     delete result.type;
-    var explicitAttrsList = [];
+    let explicitAttrsList;
     if (typeInformation.explicitAttrs && typeof typeInformation.explicitAttrs === 'boolean') {
+        explicitAttrsList = [];
         if (typeInformation.timestamp) {
             explicitAttrsList.push(constants.TIMESTAMP_ATTRIBUTE);
         }
@@ -390,9 +392,9 @@ function removeHiddenAttrs(result, typeInformation) {
         const res = jexlParser.applyExpression(typeInformation.explicitAttrs, ctx, typeInformation);
         explicitAttrsList = res;
     }
-    if (explicitAttrsList.length > 0) {
-        logger.debug(context, 'removeHiddenAttrs %s', explicitAttrsList);
+    if (explicitAttrsList && explicitAttrsList.length >= 0) {
         const hidden = _.difference(_.keys(result), explicitAttrsList);
+        logger.debug(context, 'removeHiddenAttrs %s', hidden);
         hidden.forEach((attr) => {
             delete result[attr];
         });

package/lib/services/ngsi/ngsiService.js

@@ -177,7 +177,7 @@ function executeWithDeviceInformation(operationFunction) {
 
 /**
  * Update the result of a command in the Context Broker. The result of the command has two components: the result
- * of the command itself will be represented with the sufix '_result' in the entity while the status is updated in the
+ * of the command itself will be represented with the sufix '_info' in the entity while the status is updated in the
  * attribute with the '_status' sufix.
  *
  * @param {String} entityName           Name of the entity holding the command.
@@ -234,7 +234,7 @@ function setCommandResult(
         }
 
         if (commandInfo.length === 1) {
-            exports.update(entityName, resource, apikey, attributes, typeInformation, callback);
+            exports.update(entityName, typeInformation.type, apikey, attributes, typeInformation, callback);
         } else {
             callback(new errors.CommandNotFound(commandName));
         }

package/lib/services/northBound/contextServer-NGSI-v2.js

@@ -159,29 +159,32 @@ function generateUpdateActionsNgsi2(req, contextElement, callback) {
         callback(null, updateActions);
     }
 
-    deviceService.getDeviceByName(entityId, req.headers['fiware-service'], req.headers['fiware-servicepath'], function (
-        error,
-        deviceObj
-    ) {
-        if (error) {
-            callback(error);
-        } else {
-            async.waterfall(
-                [
-                    apply(deviceService.findConfigurationGroup, deviceObj),
-                    apply(
-                        deviceService.mergeDeviceWithConfiguration,
-                        ['lazy', 'internalAttributes', 'active', 'staticAttributes', 'commands', 'subscriptions'],
-                        [null, null, [], [], [], [], []],
-                        deviceObj
-                    ),
-                    splitUpdates,
-                    createActionsArray
-                ],
-                callback
-            );
+    deviceService.getDeviceByNameAndType(
+        entityId,
+        entityType,
+        req.headers['fiware-service'],
+        req.headers['fiware-servicepath'],
+        function (error, deviceObj) {
+            if (error) {
+                callback(error);
+            } else {
+                async.waterfall(
+                    [
+                        apply(deviceService.findConfigurationGroup, deviceObj),
+                        apply(
+                            deviceService.mergeDeviceWithConfiguration,
+                            ['lazy', 'internalAttributes', 'active', 'staticAttributes', 'commands', 'subscriptions'],
+                            [null, null, [], [], [], [], []],
+                            deviceObj
+                        ),
+                        splitUpdates,
+                        createActionsArray
+                    ],
+                    callback
+                );
+            }
         }
-    });
+    );
 }
 
 /** Express middleware to manage incoming update requests using NGSIv2.
@@ -238,8 +241,7 @@ function defaultQueryHandlerNgsi2(id, type, service, subservice, attributes, cal
         type,
         id
     };
-
-    deviceService.getDeviceByName(id, service, subservice, function (error, ngsiDevice) {
+    deviceService.getDeviceByNameAndType(id, type, service, subservice, function (error, ngsiDevice) {
         if (error) {
             callback(error);
         } else {
@@ -285,8 +287,9 @@ function handleNotificationNgsi2(req, res, next) {
                 }
             }
         }
-        deviceService.getDeviceByName(
+        deviceService.getDeviceByNameAndType(
             dataElement.id,
+            dataElement.type,
             req.headers['fiware-service'],
             req.headers['fiware-servicepath'],
             function (error, device) {
@@ -448,6 +451,7 @@ function handleQueryNgsi2(req, res, next) {
         }
 
         deviceService.findConfigurationGroup(device, function (error, group) {
+            logger.debug(context, 'finishQueryForDevice %j', group);
             const executeCompleteAttributes = apply(completeAttributes, attributes, group);
             const executeQueryHandler = apply(
                 actualHandler,
@@ -474,8 +478,9 @@ function handleQueryNgsi2(req, res, next) {
 
         if (contextEntity.id) {
             getFunction = apply(
-                deviceService.getDeviceByName,
+                deviceService.getDeviceByNameAndType,
                 contextEntity.id,
+                contextEntity.type,
                 req.headers['fiware-service'],
                 req.headers['fiware-servicepath']
             );
@@ -504,7 +509,7 @@ function handleQueryNgsi2(req, res, next) {
             } else {
                 deviceList = [innerDevice];
             }
-
+            logger.debug(context, 'handleFindDevice from %j', deviceList);
             async.map(
                 deviceList,
                 async.apply(finishQueryForDevice, attributes, contextEntity, actualHandler),

package/lib/services/northBound/contextServerUtils.js

@@ -91,7 +91,7 @@ function executeUpdateSideEffects(device, id, type, service, subservice, attribu
                     ];
 
                     sideEffects.push(
-                        apply(ngsi.update, device.name, device.resource, device.apikey, newAttributes, device)
+                        apply(ngsi.update, device.name, device.type, device.apikey, newAttributes, device)
                     );
                 }
             }

package/lib/services/northBound/deviceProvisioningServer.js

@@ -184,6 +184,8 @@ function attributeToProvisioningAPIFormat(attribute) {
         entity_name: attribute.entity_name,
         entity_type: attribute.entity_type,
         mqtt: attribute.mqtt,
+        payloadType: attribute.payloadType,
+        contentType: attribute.contentType,
         metadata: attribute.metadata
     };
 }

package/lib/templates/createDevice.json

@@ -173,6 +173,18 @@
                   "type": "string",
                   "pattern": "^([^<>();'=\"]+)*$"
                 },
+                "expression": {
+                  "description": "Optional expression for command transformation",
+                  "type": "string"
+                },
+                "payloadType": {
+                    "description": "Payload type",
+                    "type": "string"
+                },
+                "contentType": {
+                    "description": "Content type",
+                    "type": "string"
+                },
                 "mqtt": {
                   "description": "Mqtt properties",
                   "type": "object",

package/lib/templates/updateDevice.json

@@ -158,6 +158,18 @@
             "type": "string",
             "pattern": "^([^<>();'=\"]+)*$"
           },
+          "expression": {
+              "description": "Optional expression for command transformation",
+              "type": "string"
+          },
+          "payloadType": {
+              "description": "Payload type",
+              "type": "string"
+          },
+          "contentType": {
+              "description": "Content type",
+              "type": "string"
+          },
           "mqtt": {
               "description": "Mqtt properties",
               "type": "object",

package/package.json

@@ -2,7 +2,7 @@
   "name": "iotagent-node-lib",
   "license": "AGPL-3.0-only",
   "description": "IoT Agent library to interface with NGSI Context Broker",
-  "version": "2.20.0",
+  "version": "2.21.0",
   "homepage": "https://github.com/telefonicaid/iotagent-node-lib",
   "keywords": [
     "fiware",
@@ -42,17 +42,17 @@
     "watch": "watch 'npm test && npm run lint' ./lib ./test"
   },
   "dependencies": {
-    "async": "2.6.2",
+    "async": "2.6.4",
     "body-parser": "~1.19.0",
     "express": "~4.16.4",
     "got": "~11.8.2",
     "jexl": "2.3.0",
     "jison": "0.4.18",
     "logops": "2.1.2",
-    "moment": "~2.24.0",
+    "moment": "~2.29.2",
     "moment-timezone": "~0.5.25",
-    "mongodb": "3.6.8",
-    "mongoose": "5.7.7",
+    "mongodb": "3.6.12",
+    "mongoose": "5.7.14",
     "query-string": "6.5.0",
     "revalidator": "~0.3.1",
     "underscore": "~1.12.1",

package/test/unit/ngsiv2/examples/contextRequests/updateContextExpressionPlugin35.json

@@ -0,0 +1,2 @@
+{
+}

package/test/unit/ngsiv2/examples/contextRequests/updateContextExpressionPlugin36.json

@@ -0,0 +1 @@
+{"myattr":{"type": "String","value": "location"}}

package/test/unit/ngsiv2/expressions/jexlBasedTransformations-test.js

@@ -319,7 +319,7 @@ const iotAgentConfig = {
                     expression: "{coordinates: [lon,lat], type: 'Point'}"
                 }
             ],
-            explicitAttrs: '[ myattr ]'
+            explicitAttrs: "[ 'myattr' ]"
         },
         GPS6: {
             commands: [],
@@ -345,6 +345,30 @@ const iotAgentConfig = {
                 }
             ],
             explicitAttrs: true
+        },
+        GPS7: {
+            commands: [],
+            type: 'GPS',
+            lazy: [],
+            static: [
+                {
+                    name: 'color',
+                    type: 'string',
+                    value: 'blue'
+                }
+            ],
+            active: [
+                {
+                    name: 'price',
+                    type: 'number'
+                },
+                {
+                    name: 'location',
+                    type: 'geo:json',
+                    expression: "{coordinates: [lon,lat], type: 'Point'}"
+                }
+            ],
+            explicitAttrs: '[ ]'
         }
     },
     service: 'smartgondor',
@@ -1180,7 +1204,7 @@ describe('Java expression language (JEXL) based transformations plugin', functio
                 .patch(
                     '/v2/entities/gps1/attrs',
                     utils.readExampleFile(
-                        './test/unit/ngsiv2/examples/contextRequests/updateContextExpressionPlugin34.json'
+                        './test/unit/ngsiv2/examples/contextRequests/updateContextExpressionPlugin36.json'
                     )
                 )
                 .query({ type: 'GPS' })
@@ -1230,6 +1254,43 @@ describe('Java expression language (JEXL) based transformations plugin', functio
             });
         });
     });
+
+    describe('When there is an extra TimeInstant sent by the device to be removed by jexl expression with context but with empty explicitAttrs', function () {
+        // Case: Expression which results is sent as a new attribute
+        const values = [
+            {
+                name: 'lat',
+                type: 'Number',
+                value: 52
+            },
+            {
+                name: 'lon',
+                type: 'Number',
+                value: 13
+            },
+            {
+                name: 'myattr',
+                type: 'String',
+                value: 'location'
+            },
+            {
+                name: 'TimeInstant',
+                type: 'DateTime',
+                value: '2015-08-05T07:35:01.468+00:00'
+            }
+        ];
+
+        beforeEach(function () {
+            nock.cleanAll();
+        });
+
+        it('should calculate them and remove non-explicitAttrs by jexl expression with context from the payload ', function (done) {
+            iotAgentLib.update('gps1', 'GPS7', '', values, function (error) {
+                should.not.exist(error);
+                done();
+            });
+        });
+    });
 });
 
 describe('Java expression language (JEXL) based transformations plugin - Timestamps', function () {