iotagent-node-lib 3.2.0 → 3.3.0

package/.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,134 @@
+name: 'Bug report'
+description: Create a report to help us improve
+labels: ['bug']
+
+body:
+  - type: markdown
+    attributes:
+      value: Thanks for taking the time to fill out this bug report!
+
+  - type: input
+    id: version
+    attributes:
+      label: IoT Agent Node Lib version the issue has been seen with
+      description: |
+        Do not submit bug reports about anything but the two most recently released *major* systemd versions upstream!
+        If there have been multiple stable releases for that major version, please consider updating to a recent one before reporting an issue.
+        See https://github.com/telefonicaid/iotagent-node-lib/releases/ for the list of most recent releases.
+      placeholder: '3.2.0'
+    validations:
+      required: true
+
+  - type: dropdown
+    id: port
+    attributes:
+      label: Bound or port used (API interaction)
+      description: Select with part of the API is motivating this issue creation.
+      multiple: true
+      options:
+        - 'Southbound (Devices data API)'
+        - 'Northbound (Provision API and NGSI Interactions)'
+        - 'Other'
+    validations:
+      required: true
+
+  - type: dropdown
+    id: ngsi-version
+    attributes:
+      label: NGSI version
+      description: Please chose the NGSI version configured
+      multiple: true
+      options:
+        - 'NGSIv2'
+        - 'NGSI-LD'
+        - 'Mixed Mode'
+        - 'Other'
+    validations:
+      required: true
+
+  - type: dropdown
+    id: containerized
+    attributes:
+      label: Are you running a container?
+      description: In case you are using a container, please select the image type the issue was seen on
+      options:
+        - 'Yes, I am using a contaner (Docker, Kubernetes...)'
+        - 'No, I am running it natively'
+    validations:
+      required: false
+
+  - type: dropdown
+    id: image-type
+    attributes:
+      label: Image type
+      description: In case you are using a container, please select the image type the issue was seen on
+      options:
+        - normal
+        - pm2
+        - distroless
+    validations:
+      required: false
+      
+  - type: textarea
+    id: expected-behaviour
+    attributes:
+      label: Expected behaviour you didn't see
+    validations:
+      required: false
+
+  - type: textarea
+    id: unexpected-behaviour
+    attributes:
+      label: Unexpected behaviour you saw
+    validations:
+      required: false
+
+  - type: textarea
+    id: steps-to-reproduce
+    attributes:
+      label: Steps to reproduce the problem
+      description: Please fill this field with all the required information and steps to reproduce the issue described above. 
+        Consider including curl requests if required
+      placeholder: |
+        ```sh
+        # Get IoTA version
+        `curl -iX GET 'http://localhost:4041/iot/about'`
+        ```
+    validations:
+      required: false
+
+  - type: textarea
+    id: config
+    attributes:
+      label: Configs
+      description: |
+        Please provide details about your environment vars or config file by copying and pasting the enviroments variables 
+        or the config.js file.
+      value: |
+        ```yaml
+        environment:
+            - "IOTA_CB_HOST=orion"
+            - "IOTA_CB_PORT=1026"
+            - "IOTA_NORTH_PORT=4041"
+            - "IOTA_REGISTRY_TYPE=mongodb"
+            - "IOTA_MONGO_HOST=mongodb"
+            - "IOTA_MONGO_PORT=27017"
+            - "IOTA_MONGO_DB=iotagent-json"
+            - "IOTA_HTTP_PORT=7896"
+            - "IOTA_PROVIDER_URL=http://iot-agent:4041"
+        ```
+
+    validations:
+      required: false
+
+  - type: textarea
+    id: additional-information
+    attributes:
+      label: Log output
+      description: |
+        Please paste the output log here, ideally when generated in debug mode (try setting the `IOTA_LOG_LEVEL=debug` environment variable).
+        For very long copy/pasted data consider using a service like https://gist.github.com/. Where copy/paste is not possible, a photo of the screen might do too, but text is always much preferred.
+      placeholder: Paste your log here
+      render: sh
+    validations:
+      required: false

package/.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,16 @@
+blank_issues_enabled: true
+contact_links:
+  - name: IoT Agent documentation
+    url: https://iotagent-node-lib.readthedocs.io/en/latest/
+    about: check the documentation before asking help or reporting a bug.
+  - name: Community Support
+    url: https://stackoverflow.com/questions/ask?tags=iot-agent+fiware
+    about: |
+      Volunteers from the Open Source FIWARE community monitor the 𝚏𝚒𝚠𝚊𝚛𝚎 tag
+      and related tags over on Stack Overflow. For 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀 𝗼𝗻𝗹𝘆 you can ask questions
+      over on Stack Overflow provided you follow their rules and guidance on posting and respond to any feedback given.
+      
+      Stack Overflow is a crowd-sourced question and answer website for professional and enthusiast programmers. It is not affliated with this
+      software and 𝗱𝗼𝗲𝘀 𝗻𝗼𝘁 𝗮𝗻𝘀𝘄𝗲𝗿 𝗰𝘂𝘀𝘁𝗼𝗺𝗲𝗿 𝘀𝗲𝗿𝘃𝗶𝗰𝗲 𝗿𝗲𝗹𝗮𝘁𝗲𝗱 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀.
+      
+      Do not raise a Stack Overflow question for bug reports or feature requests, stick with opening an issue directly within this repository.

package/.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,55 @@
+name: Feature request
+description: Suggest an improvement
+labels: ["Feature Request"]
+
+body:
+  - type: markdown
+    attributes:
+      value: Thanks for taking the time to fill out this feature request!
+
+  - type: dropdown
+    id: type
+    attributes:
+      label: Component
+      description: Please chose components related to this feature request.
+      multiple: true
+      options:
+        - 'New feature'
+        - 'Feature enhancement'
+        - 'Techdeb'
+        - 'Other'
+    validations:
+      required: false
+
+  - type: textarea
+    id: description
+    attributes:
+      label: Is your feature request related to a problem? Please describe
+      description: A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+    validations:
+      required: false
+
+  - type: textarea
+    id: solution
+    attributes:
+      label: Describe the solution you'd like
+      description: A clear and concise description of what you want to happen.
+    validations:
+      required: false
+
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: Describe alternatives you've considered
+      description: A clear and concise description of any alternative solutions or features you've considered.
+    validations:
+      required: false
+
+  - type: input
+    id: version
+    attributes:
+      label: The systemd version you checked that didn't have the feature you are asking for
+      description: If this is not the most recently released upstream version, then please check first if it has that feature already.
+      placeholder: '3.2.0'
+    validations:
+      required: false

package/.github/advanced-issue-labeler.yml

@@ -0,0 +1,30 @@
+#
+# This file is used by .github/workflows/issue-labeler.yml
+# Here, you can configure which dropdown fields are used
+# to assing a label (keys) to the issue and the name of the label (name)
+#
+# More info: https://github.com/marketplace/actions/advanced-issue-labeler
+#
+
+policy:
+  - template: [bug_report.yml]
+    section:
+      - id: [ngsi-version]
+        block-list: ['None', 'Other', 'Mixed Mode']
+        label:
+          - name: 'NGSIv2'
+            keys: ['NGSIv2']
+          - name: 'NGSI-LD'
+            keys: ['NGSI-LD']
+
+  - template: [feature_request.yml]
+    section:
+      - id: [type]
+        block-list: ['None', 'Other']
+        label:
+          - name: 'Feature request'
+            keys: ['New feature']
+          - name: 'Enhancement'
+            keys: ['Feature enhancement']
+          - name: 'Techdeb'
+            keys: ['Techdeb']

package/.github/workflows/issue-labeler.yml

@@ -0,0 +1,43 @@
+#
+# This workflow is used to asign labels to issues created through the templates. 
+# There is another file associated that maps a issue label name to each field on the 
+# issue template. You can find that file in /.github/advanced-issue-labeler.yml 
+#
+# More info: https://github.com/marketplace/actions/advanced-issue-labeler
+#
+
+name: Issue labeler
+on:
+  issues:
+    types: [ opened ]
+
+permissions:
+  contents: read
+
+jobs:
+  label-component:
+    runs-on: ubuntu-latest
+
+    permissions:
+      # required for all workflows
+      issues: write
+
+    strategy:
+      matrix:
+        template: [ bug_report.yml, feature_request.yml ]
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Parse issue form
+        uses: stefanbuck/github-issue-parser@v3
+        id: issue-parser
+        with:
+          template-path: .github/ISSUE_TEMPLATE/${{ matrix.template }}
+
+      - name: Set labels based on policy
+        uses: redhat-plumbers-in-action/advanced-issue-labeler@v2
+        with:
+          issue-form: ${{ steps.issue-parser.outputs.jsonString }}
+          template: ${{ matrix.template }}
+          token: ${{ secrets.GITHUB_TOKEN }}

package/doc/api.md

@@ -19,8 +19,7 @@
     -   [Measurement persistence options](#measurement-persistence-options)
         -   [Autoprovision configuration (autoprovision)](#autoprovision-configuration-autoprovision)
         -   [Explicitly defined attributes (explicitAttrs)](#explicitly-defined-attributes-explicitattrs)
-        -   [Configuring operation to persist the data in Context Broker (appendMode)](#configuring-operation-to-persist-the-data-in-context-broker-appendmode)
-        -   [Differences between `autoprovision`, `explicitAttrs` and `appendMode`](#differences-between-autoprovision-explicitattrs-and-appendmode)
+        -   [Differences between `autoprovision`, `explicitAttrs`](#differences-between-autoprovision-explicitattrs)
     -   [Expression language support](#expression-language-support)
         -   [Examples of JEXL expressions](#examples-of-jexl-expressions)
         -   [Available functions](#available-functions)
@@ -31,7 +30,6 @@
         -   [Multientity measurement transformation support (`object_id`)](#multientity-measurement-transformation-support-object_id)
     -   [Timestamp Compression](#timestamp-compression)
     -   [Timestamp Processing](#timestamp-processing)
-    -   [Bidirectionality plugin (bidirectional)](#bidirectionality-plugin-bidirectional)
     -   [Overriding global Context Broker host](#overriding-global-context-broker-host)
     -   [Multitenancy, FIWARE Service and FIWARE ServicePath](#multitenancy-fiware-service-and-fiware-servicepath)
     -   [Secured access to the Context Broker](#secured-access-to-the-context-broker)
@@ -141,6 +139,16 @@ subservice mapping, security information and attribute configuration can be spec
 relaying on the config group configuration. The specific parameters that can be configured for a given device are
 described in the [Device datamodel](#device-datamodel) section.
 
+If devices are not pre-registered, they will be automatically created when a measure arrives to the IoT Agent - this
+process is known as autoprovisioning. The IoT Agent will create an empty device with the group `apiKey` and `type` - the
+associated document created in database doesn't include config group parameters (in particular, `timestamp`,
+`explicitAttrs`, `active` or `attributes`, `static` and `lazy` attributes and commands). The IoT Agent will also create
+the entity in the Context Broker if it does not exist yet.
+
+This behavior allows that autoprovisioned parameters can freely established modifying the device information after
+creation using the provisioning API. However, note that if a device (autoprovisioned or not) doesn't have these
+parameters defined at device level in database, the parameters are inherit from config group parameters.
+
 ## Entity attributes
 
 In the config group/device model there are four list of attributes with different purpose to configure how the
@@ -180,17 +188,16 @@ Some transformation plugins also allow the use of the following optional fields:
 -   **expression**: indicates that the value of the target attribute will not be the plain value or the measurement, but
     an expression based on a combination of the reported values. See the
     [Expression Language definition](#expression-language-support) for details
--   **skipValue**: indicates that if the result of applying `expression` to a measure is equal to the value of `skipValue` then
-    the attribute corresponding to the measure is not sent to CB. By default if `skipValue` is not defined then is
-    considered as `null` (i.e. if the result of apply `expression` results in `null` then corresponding attribute is not
-    sent to CB). It is only used if `expression` is provided (otherwise is ignored).
+-   **skipValue**: indicates that if the result of applying `expression` to a measure is equal to the value of
+    `skipValue` then the attribute corresponding to the measure is not sent to CB. In other words, this field **is not
+    an expression**, it is a value that is compared with the result of applying `expression` to a measure. By default if
+    `skipValue` is not defined then is considered as `null` (i.e. if the result of apply `expression` results in `null`
+    then corresponding attribute is not sent to CB). It is only used if `expression` is provided (otherwise is ignored).
 -   **entity_name**: the presence of this attribute indicates that the value will not be stored in the original device
     entity but in a new entity with an ID given by this attribute. The type of this additional entity can be configured
     with the `entity_type` attribute. If no type is configured, the device entity type is used instead. Entity names can
     be defined as expressions, using the [Expression Language definition](#expression-language-support).
 -   **entity_type**: configures the type of an alternative entity.
--   **reverse**: add bidirectionality expressions to the attribute. See the **bidirectionality** transformation plugin
-    in the [Data Mapping Plugins section](development.md#bidirectionality-plugin-bidirectional) for details.
 
 Additionally for commands (which are attributes of type `command`) the following fields are optional:
 
@@ -360,15 +367,13 @@ used should be taken from those defined by
 
 ## Measurement persistence options
 
-There are 3 different options to configure how the IoTAgent stores the measures received from the devices, depending on
+There are 2 different options to configure how the IoTAgent stores the measures received from the devices, depending on
 the following parameters:
 
 -   `autoprovision`: If the device is not provisioned, the IoTAgent will create a new device and entity for it.
 -   `explicitAttrs`: If the measure element (object_id) is not defined in the mappings of the device or config group
     provision, the measure is stored in the Context Broker by adding a new attribute to the entity with the same name of
     the undefined measure element.
--   `appendMode`: It configures the request to the Context Broker to update the entity every time a new measure arrives.
-    It have implications depending if the entity is already created or not in the Context Broker.
 
 ### Autoprovision configuration (autoprovision)
 
@@ -376,7 +381,8 @@ By default, when a measure arrives to the IoTAgent, if the `device_id` does not
 IoTA creates a new device and a new entity according to the config group. Defining the field `autoprovision` to `false`
 when provisioning the config group, the IoTA to reject the measure at the southbound, allowing only to persist the data
 to devices that are already provisioned. It makes no sense to use this field in device provisioning since it is intended
-to avoid provisioning devices (and for it to be effective, it would have to be provisional).
+to avoid provisioning devices (and for it to be effective, it would have to be provisional). Further information can be
+found in section [Devices](#devices)
 
 ### Explicitly defined attributes (explicitAttrs)
 
@@ -440,15 +446,7 @@ depending on the JEXL expression evaluation:
 -   If it evaluates to an array just measures defined in the array (identified by their attribute names, not by their
     object_id) will be will be propagated to NGSI interface (as in case 3)
 
-### Configuring operation to persist the data in Context Broker (appendMode)
-
-This is a flag that can be enabled by activating the parameter `appendMode` in the configuration file or by using the
-`IOTA_APPEND_MODE` environment variable (more info
-[here](https://github.com/telefonicaid/iotagent-node-lib/blob/master/doc/installationguide.md)). If this flag is
-activated, the update requests to the Context Broker will be performed always with APPEND type, instead of the default
-UPDATE. This have implications in the use of attributes with Context Providers, so this flag should be used with care.
-
-### Differences between `autoprovision`, `explicitAttrs` and `appendMode`
+### Differences between `autoprovision`, `explicitAttrs`
 
 Since those configuration parameters are quite similar, this section is intended to clarify the relation between them.
 
@@ -460,16 +458,6 @@ related to the **southbound**.
 What `explicitAttrs` does is to filter from the southbound the parameters that are not explicitly defined in the device
 provision or config group. That also would avoid propagating the measures to the Context Broker.
 
-The default way the agent updates the information into the Context Broker is by using an update request. If
-`appendMode=true`, the IoTA will use an append request instead of an update one. This means it will store the attributes
-even if they are not present in the entity. This seems the same functionality that the one provided by `autoprovision`,
-but it is a different concept since the scope of this config is to setup how the IoT interacts with the context broker,
-this is something related to the **northbound**.
-
-Note that, even creating a config group with `autoprovision=true` and `explicitAttrs=true`, if you do not provision
-previously the entity in the Context Broker (having all attributes to be updated), it would fail if `appendMode=false`.
-For further information check the issue [#1301](https://github.com/telefonicaid/iotagent-node-lib/issues/1301).
-
 ## Expression language support
 
 The IoTAgent Library provides an expression language for measurement transformation and other purposes. This expression
@@ -495,6 +483,9 @@ expression. In all cases the following data is available to all expressions:
 -   `subservice`: device subservice
 -   `staticAttributes`: static attributes defined in the device or config group
 
+Additionally, for attribute expressions (`expression`, `entity_name`) and `entityNameExp` measures are avaiable in the
+**context** used to evaluate them.
+
 ### Examples of JEXL expressions
 
 The following table shows expressions and their expected outcomes taking into account the following measures at
@@ -858,54 +849,6 @@ The IOTA processes the entity attributes looking for a `TimeInstant` attribute.
 adds a `TimeInstant` attribute as metadata for every other attribute in the same request. With NGSI-LD, the Standard
 `observedAt` property-of-a-property is used instead.
 
-## Bidirectionality plugin (bidirectional)
-
-This plugin allows the devices with composite values an expression to update the original values in the devices when the
-composite expressions are updated in the Context Broker. This behavior is achieved through the use of subscriptions.
-
-IoTAs using this plugins should also define a notification handler to handle incoming values. This handler will be
-intercepted by the plugin, so the mapped values are included in the updated notification.
-
-When a device is provisioned with bidirectional attributes, the IoTAgent subscribes to changes in that attribute. When a
-change notification for that attribute arrives to the IoTA, it applies the transformation defined in the device
-provisioning payload to the notification, and calls the underlying notification handler with the transformed entity
-including the `value` along with any `metadata`, and in the case of an NGSI-LD bidirectional attribute a `datasetId` if
-provided.
-
-The following `attributes` section shows an example of the plugin configuration (using `IOTA_AUTOCAST=false` to avoid
-translation from geo:point to geo:json)
-
-```json
-      "attributes": [
-        {
-          "name":"location",
-          "type":"geo:point",
-          "expression": "latitude, longitude",
-          "reverse": [
-            {
-              "object_id":"longitude",
-              "type": "Number",
-              "expression": "location | split(', ')[0] | parsefloat()"
-            },
-            {
-              "object_id":"latitude",
-              "type": "Number",
-              "expression": "location | split(', ')[1] | parsefloat()"
-            }
-          ]
-        }
-      ],
-```
-
-For each attribute that would have bidirectionality, a new field `reverse` must be configured. This field will contain
-an array of fields that will be created based on the notifications content. The expression notification can contain any
-attribute of the same entity as the bidirectional attribute; declaring them in the expressions will add them to the
-subscription payload.
-
-For each attribute in the `reverse` array, an expression must be defined to calculate its value based on the
-notification attributes. This value will be passed to the underlying protocol with the `object_id` name. Details about
-how the value is then progressed to the device are protocol-specific.
-
 ## Overriding global Context Broker host
 
 **cbHost**: Context Broker host URL. This option can be used to override the global CB configuration for specific types

package/doc/deprecated.md

@@ -22,6 +22,8 @@ A list of deprecated features and the version in which they were deprecated foll
 -   Support groups (provision) statically defined by configuration
 -   Support to in-memory registry (i.e.`deviceRegistry.type=memory`)
 -   Support to legacy expressions (finally removed in 3.2.0)
+-   Bidirectinal pluging (to be removed in 3.4.0)
+-   appendMode (to be removed in 3.4.0)
 
 The use of Node.js v14 is highly recommended.
 
@@ -51,3 +53,5 @@ The following table provides information about the last iotagent-node-lib versio
 | Support to Node.js v12 | 2.24.0                                                | September 2nd, 2022           |
 | Support to NGSI-LD 1.3 | 2.25.0                                                | January 24th, 2023            |
 | Support to Legacy Expressions | 3.1.0                                          | April 25th, 2023              |
+| bidirrectional plugin | 3.3.0                                                  | August 24th, 2023             |
+| appendMode | 3.3.0                                                             | August 24th, 2023             |