iotagent-node-lib 3.2.0 → 3.4.0

package/doc/api.md

@@ -19,8 +19,7 @@
     -   [Measurement persistence options](#measurement-persistence-options)
         -   [Autoprovision configuration (autoprovision)](#autoprovision-configuration-autoprovision)
         -   [Explicitly defined attributes (explicitAttrs)](#explicitly-defined-attributes-explicitattrs)
-        -   [Configuring operation to persist the data in Context Broker (appendMode)](#configuring-operation-to-persist-the-data-in-context-broker-appendmode)
-        -   [Differences between `autoprovision`, `explicitAttrs` and `appendMode`](#differences-between-autoprovision-explicitattrs-and-appendmode)
+        -   [Differences between `autoprovision`, `explicitAttrs`](#differences-between-autoprovision-explicitattrs)
     -   [Expression language support](#expression-language-support)
         -   [Examples of JEXL expressions](#examples-of-jexl-expressions)
         -   [Available functions](#available-functions)
@@ -31,7 +30,6 @@
         -   [Multientity measurement transformation support (`object_id`)](#multientity-measurement-transformation-support-object_id)
     -   [Timestamp Compression](#timestamp-compression)
     -   [Timestamp Processing](#timestamp-processing)
-    -   [Bidirectionality plugin (bidirectional)](#bidirectionality-plugin-bidirectional)
     -   [Overriding global Context Broker host](#overriding-global-context-broker-host)
     -   [Multitenancy, FIWARE Service and FIWARE ServicePath](#multitenancy-fiware-service-and-fiware-servicepath)
     -   [Secured access to the Context Broker](#secured-access-to-the-context-broker)
@@ -58,7 +56,7 @@
     -   [Miscellaneous API](#miscellaneous-api)
         -   [Log operations](#log-operations)
             -   [Modify Loglevel `PUT /admin/log`](#modify-loglevel-put-adminlog)
-            -   [Retrieve log level `PUT /admin/log`](#retrieve-log-level-put-adminlog)
+            -   [Retrieve log level `GET /admin/log`](#retrieve-log-level-get-adminlog)
         -   [About operations](#about-operations)
             -   [List IoTA Information `GET /iot/about`](#list-iota-information-get-iotabout)
 
@@ -141,6 +139,16 @@ subservice mapping, security information and attribute configuration can be spec
 relaying on the config group configuration. The specific parameters that can be configured for a given device are
 described in the [Device datamodel](#device-datamodel) section.
 
+If devices are not pre-registered, they will be automatically created when a measure arrives to the IoT Agent - this
+process is known as autoprovisioning. The IoT Agent will create an empty device with the group `apiKey` and `type` - the
+associated document created in database doesn't include config group parameters (in particular, `timestamp`,
+`explicitAttrs`, `active` or `attributes`, `static` and `lazy` attributes and commands). The IoT Agent will also create
+the entity in the Context Broker if it does not exist yet.
+
+This behavior allows that autoprovisioned parameters can freely established modifying the device information after
+creation using the provisioning API. However, note that if a device (autoprovisioned or not) doesn't have these
+parameters defined at device level in database, the parameters are inherit from config group parameters.
+
 ## Entity attributes
 
 In the config group/device model there are four list of attributes with different purpose to configure how the
@@ -175,22 +183,21 @@ All of them have the same syntax, a list of objects with the following attribute
 -   **type** (mandatory): name of the type of the attribute in the target entity.
 -   **metadata** (optional): additional static metadata for the attribute in the target entity. (e.g. `unitCode`)
 
-Some transformation plugins also allow the use of the following optional fields:
+Some advanced features also allow the use of the following optional fields:
 
 -   **expression**: indicates that the value of the target attribute will not be the plain value or the measurement, but
     an expression based on a combination of the reported values. See the
     [Expression Language definition](#expression-language-support) for details
--   **skipValue**: indicates that if the result of applying `expression` to a measure is equal to the value of `skipValue` then
-    the attribute corresponding to the measure is not sent to CB. By default if `skipValue` is not defined then is
-    considered as `null` (i.e. if the result of apply `expression` results in `null` then corresponding attribute is not
-    sent to CB). It is only used if `expression` is provided (otherwise is ignored).
+-   **skipValue**: indicates that if the result of applying `expression` to a measure is equal to the value of
+    `skipValue` then the attribute corresponding to the measure is not sent to CB. In other words, this field **is not
+    an expression**, it is a value that is compared with the result of applying `expression` to a measure. By default if
+    `skipValue` is not defined then is considered as `null` (i.e. if the result of apply `expression` results in `null`
+    then corresponding attribute is not sent to CB). It is only used if `expression` is provided (otherwise is ignored).
 -   **entity_name**: the presence of this attribute indicates that the value will not be stored in the original device
     entity but in a new entity with an ID given by this attribute. The type of this additional entity can be configured
     with the `entity_type` attribute. If no type is configured, the device entity type is used instead. Entity names can
     be defined as expressions, using the [Expression Language definition](#expression-language-support).
 -   **entity_type**: configures the type of an alternative entity.
--   **reverse**: add bidirectionality expressions to the attribute. See the **bidirectionality** transformation plugin
-    in the [Data Mapping Plugins section](development.md#bidirectionality-plugin-bidirectional) for details.
 
 Additionally for commands (which are attributes of type `command`) the following fields are optional:
 
@@ -360,15 +367,13 @@ used should be taken from those defined by
 
 ## Measurement persistence options
 
-There are 3 different options to configure how the IoTAgent stores the measures received from the devices, depending on
+There are 2 different options to configure how the IoTAgent stores the measures received from the devices, depending on
 the following parameters:
 
 -   `autoprovision`: If the device is not provisioned, the IoTAgent will create a new device and entity for it.
 -   `explicitAttrs`: If the measure element (object_id) is not defined in the mappings of the device or config group
     provision, the measure is stored in the Context Broker by adding a new attribute to the entity with the same name of
     the undefined measure element.
--   `appendMode`: It configures the request to the Context Broker to update the entity every time a new measure arrives.
-    It have implications depending if the entity is already created or not in the Context Broker.
 
 ### Autoprovision configuration (autoprovision)
 
@@ -376,7 +381,8 @@ By default, when a measure arrives to the IoTAgent, if the `device_id` does not
 IoTA creates a new device and a new entity according to the config group. Defining the field `autoprovision` to `false`
 when provisioning the config group, the IoTA to reject the measure at the southbound, allowing only to persist the data
 to devices that are already provisioned. It makes no sense to use this field in device provisioning since it is intended
-to avoid provisioning devices (and for it to be effective, it would have to be provisional).
+to avoid provisioning devices (and for it to be effective, it would have to be provisional). Further information can be
+found in section [Devices](#devices)
 
 ### Explicitly defined attributes (explicitAttrs)
 
@@ -440,15 +446,7 @@ depending on the JEXL expression evaluation:
 -   If it evaluates to an array just measures defined in the array (identified by their attribute names, not by their
     object_id) will be will be propagated to NGSI interface (as in case 3)
 
-### Configuring operation to persist the data in Context Broker (appendMode)
-
-This is a flag that can be enabled by activating the parameter `appendMode` in the configuration file or by using the
-`IOTA_APPEND_MODE` environment variable (more info
-[here](https://github.com/telefonicaid/iotagent-node-lib/blob/master/doc/installationguide.md)). If this flag is
-activated, the update requests to the Context Broker will be performed always with APPEND type, instead of the default
-UPDATE. This have implications in the use of attributes with Context Providers, so this flag should be used with care.
-
-### Differences between `autoprovision`, `explicitAttrs` and `appendMode`
+### Differences between `autoprovision`, `explicitAttrs`
 
 Since those configuration parameters are quite similar, this section is intended to clarify the relation between them.
 
@@ -460,16 +458,6 @@ related to the **southbound**.
 What `explicitAttrs` does is to filter from the southbound the parameters that are not explicitly defined in the device
 provision or config group. That also would avoid propagating the measures to the Context Broker.
 
-The default way the agent updates the information into the Context Broker is by using an update request. If
-`appendMode=true`, the IoTA will use an append request instead of an update one. This means it will store the attributes
-even if they are not present in the entity. This seems the same functionality that the one provided by `autoprovision`,
-but it is a different concept since the scope of this config is to setup how the IoT interacts with the context broker,
-this is something related to the **northbound**.
-
-Note that, even creating a config group with `autoprovision=true` and `explicitAttrs=true`, if you do not provision
-previously the entity in the Context Broker (having all attributes to be updated), it would fail if `appendMode=false`.
-For further information check the issue [#1301](https://github.com/telefonicaid/iotagent-node-lib/issues/1301).
-
 ## Expression language support
 
 The IoTAgent Library provides an expression language for measurement transformation and other purposes. This expression
@@ -495,6 +483,9 @@ expression. In all cases the following data is available to all expressions:
 -   `subservice`: device subservice
 -   `staticAttributes`: static attributes defined in the device or config group
 
+Additionally, for attribute expressions (`expression`, `entity_name`) and `entityNameExp` measures are avaiable in the
+**context** used to evaluate them.
+
 ### Examples of JEXL expressions
 
 The following table shows expressions and their expected outcomes taking into account the following measures at
@@ -683,6 +674,10 @@ will check which ones contain expressions whose variables are present in the rec
 whose variables are covered, their expressions will be executed with the received values, and their values updated in
 the Context Broker.
 
+If as a result of apply expression `undefined` or `null` value is obtained then any of them will not progress to entity
+attribute. Have into acount that some numeric operations results (like nonexistent \* 2) are a kind of null with a
+number type but NaN value, which will also not progress to entity attribute.
+
 E.g.: if a device with the following provisioning information is created in the IoT Agent:
 
 ```json
@@ -760,7 +755,7 @@ following to CB:
 
 ### Multientity measurement transformation support (`object_id`)
 
-To allow support for measurement transformation in combination with multi entity plugin, where the same attribute is
+To allow support for measurement transformation in combination with multi entity feature, where the same attribute is
 generated for different entities out of different incoming attribute values (i.e. `object_id`), we introduced support
 for `object_id` in the expression context.
 
@@ -854,58 +849,10 @@ it in queries (and viceversa, receive the extended one in queries and return it
 
 ## Timestamp Processing
 
-The IOTA processes the entity attributes looking for a `TimeInstant` attribute. If one is found, for NGSI v2, the plugin
+The IOTA processes the entity attributes looking for a `TimeInstant` attribute. If one is found, for NGSI v2, then it
 adds a `TimeInstant` attribute as metadata for every other attribute in the same request. With NGSI-LD, the Standard
 `observedAt` property-of-a-property is used instead.
 
-## Bidirectionality plugin (bidirectional)
-
-This plugin allows the devices with composite values an expression to update the original values in the devices when the
-composite expressions are updated in the Context Broker. This behavior is achieved through the use of subscriptions.
-
-IoTAs using this plugins should also define a notification handler to handle incoming values. This handler will be
-intercepted by the plugin, so the mapped values are included in the updated notification.
-
-When a device is provisioned with bidirectional attributes, the IoTAgent subscribes to changes in that attribute. When a
-change notification for that attribute arrives to the IoTA, it applies the transformation defined in the device
-provisioning payload to the notification, and calls the underlying notification handler with the transformed entity
-including the `value` along with any `metadata`, and in the case of an NGSI-LD bidirectional attribute a `datasetId` if
-provided.
-
-The following `attributes` section shows an example of the plugin configuration (using `IOTA_AUTOCAST=false` to avoid
-translation from geo:point to geo:json)
-
-```json
-      "attributes": [
-        {
-          "name":"location",
-          "type":"geo:point",
-          "expression": "latitude, longitude",
-          "reverse": [
-            {
-              "object_id":"longitude",
-              "type": "Number",
-              "expression": "location | split(', ')[0] | parsefloat()"
-            },
-            {
-              "object_id":"latitude",
-              "type": "Number",
-              "expression": "location | split(', ')[1] | parsefloat()"
-            }
-          ]
-        }
-      ],
-```
-
-For each attribute that would have bidirectionality, a new field `reverse` must be configured. This field will contain
-an array of fields that will be created based on the notifications content. The expression notification can contain any
-attribute of the same entity as the bidirectional attribute; declaring them in the expressions will add them to the
-subscription payload.
-
-For each attribute in the `reverse` array, an expression must be defined to calculate its value based on the
-notification attributes. This value will be passed to the underlying protocol with the `object_id` name. Details about
-how the value is then progressed to the device are protocol-specific.
-
 ## Overriding global Context Broker host
 
 **cbHost**: Context Broker host URL. This option can be used to override the global CB configuration for specific types
@@ -1666,7 +1613,7 @@ _**Response code**_
 -   `200` `OK` if successful.
 -   `500` `SERVER ERROR` if there was any error not contemplated above.
 
-#### Retrieve log level `PUT /admin/log`
+#### Retrieve log level `GET /admin/log`
 
 _**Response code**_
 

package/doc/deprecated.md

@@ -21,7 +21,10 @@ A list of deprecated features and the version in which they were deprecated foll
 -   Support to NGSI-LD v1.3 in iotagent-node-lib 2.25.0 (finally removed in 2.26.0)
 -   Support groups (provision) statically defined by configuration
 -   Support to in-memory registry (i.e.`deviceRegistry.type=memory`)
+-   eventType configuration (finally removed in 3.0.0)
 -   Support to legacy expressions (finally removed in 3.2.0)
+-   Bidirectinal pluging (finally removed in 3.4.0)
+-   appendMode configuration (`IOTA_APPEND_MODE` env var) (finally removed in 3.4.0)
 
 The use of Node.js v14 is highly recommended.
 
@@ -41,13 +44,16 @@ information in the case you want to use old versions:
 
 The following table provides information about the last iotagent-node-lib version supporting currently removed features:
 
-| **Removed feature**    | **Last iotagent-node-lib version supporting feature** | **That version release date** |
-| ---------------------- | ----------------------------------------------------- | ----------------------------- |
-| NGSI v1 API            | 2.17.0                                                | August 30th, 2021             |
-| Support to Node.js v4  | 2.8.1                                                 | December 19th, 2018           |
-| Support to Node.js v6  | 2.9.0                                                 | May 22nd, 2019                |
-| Support to Node.js v8  | 2.12.0                                                | April 7th, 2020               |
-| Support to Node.js v10 | 2.15.0                                                | February 18th, 2021           |
-| Support to Node.js v12 | 2.24.0                                                | September 2nd, 2022           |
-| Support to NGSI-LD 1.3 | 2.25.0                                                | January 24th, 2023            |
-| Support to Legacy Expressions | 3.1.0                                          | April 25th, 2023              |
+| **Removed feature**                                   | **Last iotagent-node-lib version supporting feature** | **That version release date** |
+| ----------------------------------------------------- | ----------------------------------------------------- | ----------------------------- |
+| NGSI v1 API                                           | 2.17.0                                                | August 30th, 2021             |
+| Support to Node.js v4                                 | 2.8.1                                                 | December 19th, 2018           |
+| Support to Node.js v6                                 | 2.9.0                                                 | May 22nd, 2019                |
+| Support to Node.js v8                                 | 2.12.0                                                | April 7th, 2020               |
+| Support to Node.js v10                                | 2.15.0                                                | February 18th, 2021           |
+| Support to Node.js v12                                | 2.24.0                                                | September 2nd, 2022           |
+| Support to NGSI-LD 1.3                                | 2.25.0                                                | January 24th, 2023            |
+| eventType configuration                               | 2.26.0                                                | March 15th, 2023              |
+| Support to Legacy Expressions                         | 3.1.0                                                 | April 25th, 2023              |
+| bidirectional plugin                                  | 3.3.0                                                 | August 24th, 2023             |
+| appendMode configuration (`IOTA_APPEND_MODE` env var) | 3.3.0                                                 | August 24th, 2023             |

package/doc/{architecture.md → devel/architecture.md}

@@ -1,6 +1,6 @@
 ## Architecture
 
-The following section defines the archtecture and message flow which is common to all IoT Agents which use the library.
+The following section defines the architecture and message flow which is common to all IoT Agents which use the library.
 
 ### Device to NGSI Mapping
 
@@ -32,7 +32,7 @@ basis preprovisioning the devices). Device measures can have three different beh
 The following sequence diagram shows the different NGSI interactions an IoT Agent makes with the Context Broker,
 explained in the following subsections (using the example of a OMA Lightweight M2M device).
 
-![General ](./img/ngsiInteractions.png "NGSI Interactions")
+![General ](../img/ngsiInteractions.png 'NGSI Interactions')
 
 Be aware that the IoT Agents are only required to support NGSI10 operations `updateContext` and `queryContext` in their
 standard formats (currently in JSON format; XML deprecated) but will not answer to NGSI9 operations (or NGSI convenience
@@ -254,7 +254,7 @@ the concrete IoT Agent implementations will be to map between the native device
 The following figure offers a graphical example of how a COAP IoT Agent work, ordered from the registration of the
 device to a command update to the device.
 
-![General ](./img/iotAgentLib.png "Architecture Overview")
+![General ](../img/iotAgentLib.png 'Architecture Overview')
 
 ### The `TimeInstant` element
 

package/doc/{Contribution.md → devel/contribution-guidelines.md}

@@ -4,21 +4,22 @@
 
 Before we get started, here are a few things we expect from you (and that you should expect from others):
 
-* Be kind and thoughtful in your conversations around this project. We all come from different backgrounds and
-  projects, which means we likely have different perspectives on "how open source is done." Try to listen to others
-  rather than convince them that your way is correct.
-* Please ensure that your contribution passes all tests. If there are test failures, you will need to address them
-  before we can merge your contribution.
-* When adding content, please consider if it is widely valuable. Please don't add references or links to things you or
-  your employer have created as others will do so if they appreciate it.
-* When reporting a vulnerability on the software, please, put in contact with IoT Agent Node Lib repository maintainers in order to discuss it 
-  in a private way.
+-   Be kind and thoughtful in your conversations around this project. We all come from different backgrounds and
+    projects, which means we likely have different perspectives on "how open source is done." Try to listen to others
+    rather than convince them that your way is correct.
+-   Please ensure that your contribution passes all tests. If there are test failures, you will need to address them
+    before we can merge your contribution.
+-   When adding content, please consider if it is widely valuable. Please don't add references or links to things you or
+    your employer have created as others will do so if they appreciate it.
+-   When reporting a vulnerability on the software, please, put in contact with IoT Agent Node Lib repository
+    maintainers in order to discuss it in a private way.
 
 ## How to contribute
 
-If you'd like to contribute, start by searching through the [issues](https://github.com/telefonicaid/iotagent-node-lib/issues) and
-[pull requests](https://github.com/telefonicaid/iotagent-node-lib/pulls) to see whether someone else has raised a similar idea or
-question.
+If you'd like to contribute, start by searching through the
+[issues](https://github.com/telefonicaid/iotagent-node-lib/issues) and
+[pull requests](https://github.com/telefonicaid/iotagent-node-lib/pulls) to see whether someone else has raised a
+similar idea or question.
 
 If you don't see your idea listed, and you think it fits into the goals of this guide, do one of the following:
 
@@ -28,36 +29,43 @@ If you don't see your idea listed, and you think it fits into the goals of this
 
 ### Pull Request protocol
 
-As explained in ([FIWARE Contribution Requirements](https://fiware-requirements.readthedocs.io/en/latest)) 
-contributions are done using a pull request (PR). The detailed "protocol" used in such PR is described below:
-
-* Direct commits to master branch (even single-line modifications) are not allowed. Every modification has to come as a PR
-* In case the PR is implementing/fixing a numbered issue, the issue number has to be referenced in the body of the PR at creation time
-* Anybody is welcome to provide comments to the PR (either direct comments or using the review feature offered by GitHub)
-* Use *code line comments* instead of *general comments*, for traceability reasons (see comments lifecycle below)
-* Comments lifecycle
-  * Comment is created, initiating a *comment thread*
-  * New comments can be added as responses to the original one, starting a discussion
-  * After discussion, the comment thread ends in one of the following ways:
-    * `Fixed in <commit hash>` in case the discussion involves a fix in the PR branch (which commit hash is 
-       included as reference)
-    * `NTC`, if finally nothing needs to be done (NTC = Nothing To Change)
- * PR can be merged when the following conditions are met:
-    * All comment threads are closed
-    * All the participants in the discussion have provided a `LGTM` general comment (LGTM = Looks good to me)
- * Self-merging is not allowed (except in rare and justified circumstances)
+As explained in ([FIWARE Contribution Requirements](https://fiware-requirements.readthedocs.io/en/latest)) contributions
+are done using a pull request (PR). The detailed "protocol" used in such PR is described below:
+
+-   Direct commits to master branch (even single-line modifications) are not allowed. Every modification has to come as
+    a PR
+-   In case the PR is implementing/fixing a numbered issue, the issue number has to be referenced in the body of the PR
+    at creation time
+-   Anybody is welcome to provide comments to the PR (either direct comments or using the review feature offered by
+    GitHub)
+-   Use _code line comments_ instead of _general comments_, for traceability reasons (see comments lifecycle below)
+-   Comments lifecycle
+    -   Comment is created, initiating a _comment thread_
+    -   New comments can be added as responses to the original one, starting a discussion
+    -   After discussion, the comment thread ends in one of the following ways:
+        -   `Fixed in <commit hash>` in case the discussion involves a fix in the PR branch (which commit hash is
+            included as reference)
+        -   `NTC`, if finally nothing needs to be done (NTC = Nothing To Change)
+-   PR can be merged when the following conditions are met:
+    -   All comment threads are closed
+    -   All the participants in the discussion have provided a `LGTM` general comment (LGTM = Looks good to me)
+-   Self-merging is not allowed (except in rare and justified circumstances)
 
 Some additional remarks to take into account when contributing with new PRs:
 
-* PR must include not only code contributions, but their corresponding pieces of documentation (new or modifications to existing one) and tests
-* PR modifications must pass full regression based on existing test (unit, functional, memory, e2e) in addition to whichever new test added due to the new functionality
-* PR should be of an appropriated size that makes review achievable. Too large PRs could be closed with a "please, redo the work in smaller pieces" without any further discussing
+-   PR must include not only code contributions, but their corresponding pieces of documentation (new or modifications
+    to existing one) and tests
+-   PR modifications must pass full regression based on existing test (unit, functional, memory, e2e) in addition to
+    whichever new test added due to the new functionality
+-   PR should be of an appropriated size that makes review achievable. Too large PRs could be closed with a "please,
+    redo the work in smaller pieces" without any further discussing
 
 ## Community
 
 Discussions about the Open Source Guides take place on this repository's
-[Issues](https://github.com/telefonicaid/iotagent-node-lib/issues) and [Pull Requests](https://github.com/telefonicaid/iotagent-node-lib/pulls)
-sections. Anybody is welcome to join these conversations.
+[Issues](https://github.com/telefonicaid/iotagent-node-lib/issues) and
+[Pull Requests](https://github.com/telefonicaid/iotagent-node-lib/pulls) sections. Anybody is welcome to join these
+conversations.
 
 Wherever possible, do not take these conversations to private channels, including contacting the maintainers directly.