iotagent-node-lib 4.5.0 → 4.7.0

package/.github/workflows/ci.yml

@@ -51,7 +51,6 @@ jobs:
     strategy:
       matrix:
         node-version:
-          - 14.x
           - 16.x
           - 18.x
     steps:

package/Changelog

@@ -0,0 +1,12 @@
+4.7.0 (February 3rd, 2025)
+
+- Add: store (and recover) previous jexlctxt and make available in current jexlctxt (#1690)
+- Add: option to force to use CB flow control with new API field useCBflowControl at group and device device level  (#1420)
+- Add: useCBflowControl config setting (IOTA_CB_FLOW_CONTROL env var) to set CB flow control behaviour at instance level (#1420)
+- Add: allow remove last measure in device
+- Add: store last measure in device (by id, apikey, service and subservice) and new API field storeLastMeasure at group and device levels (#1669)
+- Add: storeLastMeasure config setting  (IOTA_STORE_LAST_MEASURE env var) to set default store last measure behaviour at instance level (#1669)
+- Fix: set polling and transport for autoprovisioned devices
+- Upgrade express dep from 4.19.2 to 4.21.2
+- Upgrade mongodb devdep from 4.17.1 to 4.17.2
+- Upgrade mongoose dep from 5.13.20 to 8.9.5 (solving vulnerabilies CVE-2024-53900 and CVE-2025-23061) (#1674)

package/README.md

@@ -1,4 +1,4 @@
-# FIWARE IoT Agent Node.js Library
+# FIWARE IoT Agent Node Library
 
 [![FIWARE IoT Agents](https://nexus.lab.fiware.org/static/badges/chapters/iot-agents.svg)](https://www.fiware.org/developers/catalogue/)
 [![License: APGL](https://img.shields.io/github/license/telefonicaid/iotagent-node-lib.svg)](https://opensource.org/licenses/AGPL-3.0)
@@ -10,300 +10,95 @@
 ![Status](https://nexus.lab.fiware.org/static/badges/statuses/iot-node-lib.svg)
 [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/4671/badge)](https://bestpractices.coreinfrastructure.org/projects/4671)
 
-This project aims to provide a Node.js module to enable IoT Agent developers to build custom agents for their devices
-that can easily connect to NGSI Context Brokers (such as [Orion](https://github.com/telefonicaid/fiware-orion)).
+An IoT Agent is a component that lets groups of IoT devices send their data to a NGSI Context Broker (such as
+[Orion Context Broker](https://github.com/telefonicaid/fiware-orion)) using their own native protocols and translating
+them into NGSI.
 
-An IoT Agent is a component that lets groups of devices send their data to and be managed from a FIWARE NGSI Context
-Broker using their own native protocols. IoT Agents should also be able to deal with security aspects of the FIWARE
-platform (authentication and authorization of the channel) and provide other common services to the device programmer.
+The **IoT Agent Node Lib** provides simple standardized REST API for registering, provisioning, discovering and managing
+devices and groups of devices. It also ensures that each IoT Agent can configure its device communications using a
+common vocabulary regardless of the payload, syntax or transport protocol used by the device itself
 
-This project is part of [FIWARE](https://www.fiware.org/). For more information check the FIWARE Catalogue entry for the
-[IoT Agents](https://github.com/Fiware/catalogue/tree/master/iot-agents).
-
-| :books: [Documentation](https://iotagent-node-lib.rtfd.io) | :mortar_board: [Academy](https://fiware-academy.readthedocs.io/en/latest/iot-agents/idas) | :dart: [Roadmap](https://github.com/telefonicaid/iotagent-node-lib/blob/master/doc/roadmap.md) |
-| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
-
-## Index
-
--   [Background](#background)
--   [Install](#install)
--   [API](#api)
--   [Usage](#usage)
--   [Testing](#testing)
-    -   [Agent Console](#agent-console)
-    -   [Agent tester](#agent-tester)
--   [Licence](#licence)
+It also provides a common configuration framework driven by a `config.js` configuration file and a series of common ENV
+variables, configuring some flags for common functionality provided by the **IoT Agent node lib** (e.g. for contecting
+to a Conext Broker or for authenticating with an Identity Manager).Some of common utility functions provided by the
+**IoT Agent node lib** include:
 
-## Background
-
-The main concept of the **IoT Agent node library** is to provide a common framework for provisioning IoT devices,
-allowing each individual IoT Agent to access standardized mapping data for devices and to offer a series common utility
-functions.
-
--   For southbound communications, the library listens to changes in context entities and raises callbacks for the IoT
-    Agent to process.
--   For northbound communications, the library offers an interface which accepts structured input data so that all NGSI
-    communications are left to the library.
+-   For the device communication (southbound), the library listens to changes in context entities and raises callbacks
+    for the IoT Agent to process. It also handles the commands sent by the Context Broker to the devices.
+-   For the context broker communications (northbound), the library offers an interface which persists data from the
+    device in the Context Broker and accepts NGSI data from the Context Broker to be sent to the device.
 -   Standardized OAuth2-based security is available to enable each IoT Agent to connect to several common Identity
     Managers (e.g. Keystone and Keyrock) so that communications can be restricted to trusted components.
 
-Each individual IoT Agent offers is driven by a `config.js` configuration file contains explicit custom settings based
-on the protocol and payload the IoT Agent is translating. It will also contain some common flags for common
-functionality provided by the IoT Agent node lin (e.g. for contecting to a conext broker or for authentication). The
-**IoT Agent node library** offers a standard API for provisioning devices and ensures that each IoT Agent can configure
-its device communications using a common vocabulary regardless of the payload, syntax or transport protocol used by the
-device itself.
-
-## Install
-
-The **IoT Agent node library** is not a standalone product and should be added as a dependency to `package.json` of the
-IoT Agent
-
-```json
-...
-"dependencies": {
-    "iotagent-node-lib": "*",
-}
-```
-
-In order to use the library within your own IoT Agent, you must first you require it before use:
-
-```javascript
-const iotagentLib = require('iotagent-node-lib');
-```
-
-Information about how to configure the Library can be found at the corresponding section of the
-[Installation & Administration Guide](doc/admin.md).
-
-## Usage
-
-This library has no packaging or build processes. The [Getting Started](./doc/getting-started.md) is a good place to
-start. You can also review the [API documentation](./doc/api.md) for a full list of the available functions.
-
-If you plan to use the library in your own IoT Agent, you should read the [Developer Guide](./doc/devel/development.md).
-You can also review the [Architecture](./doc/devel/architecture.md) documentation and
-[Northbound API](./doc/devel/northbound-api.md).
-
-The following features are listed as [deprecated](doc/deprecated.md).
-
-## API
-
-The **IoT Agent node library** offers a simple REST API which provides common functionality to access, provision and
-decommission devices. [API](doc/api.md).
-
-## Testing
-
-Contributions to development can be found [here](doc/devel/development.md) - additional contributions are welcome.
-
-If you are whishing to test the library, or include new tests (either as part of a contribution or as a new feature or
-as a bug report), you can use the functional tests suite included in the project. The tests are described using a JSON
-file. You can find more information about the test suite in the [Functional Tests Guide](test/functional/README.md).
-
-### Agent Console
-
-A command-line client to experiment with the library is packed with it. The command-line client can be started using the
-following command:
-
-```console
-bin/agentConsole.js
-```
-
-The client offers an API similar to the one offered by the library: it can start and stop an IoT agent, register and
-unregister devices, send measures mimicking the device and receive updates of the device data. Take into account that,
-by default, the console uses the same `config.js` file than the IoT Agent.
-
-The command-line client creates a console that offers the following options:
-
-```text
-stressInit
-
-	Start recording a stress batch.
-
-stressCommit <delay> <times> <threads> <initTime>
-
-	Executes the recorded batch as many times as requested, with delay (ms) between commands.
-	The "threads" parameter indicates how many agents will repeat that same sequence. The "initTime" (ms)
-	parameter indicates the mean of the random initial waiting times for each agent.
-
-exit
-
-	Exit from the command-line.
-
-start
-
-	Start the IoT Agent
-
-stop
-
-	Stop the IoT Agent
-
-register <id> <type>
-
-	Register a new device in the IoT Agent. The attributes to register will be extracted from the
-	type configuration
-
-unregister <id> <type>
-
-	Unregister the selected device
-
-showConfig
-
-	Show the current configuration file
-
-config <newConfig>
-
-	Change the configuration file to a new one
-
-updatevalue <deviceId> <deviceType> <attributes>
-
-	Update a device value in the Context Broker. The attributes should be triads with the following
-	format: "name/type/value" sepparated by commas.
-
-listdevices
-
-	List all the devices that have been registered in this IoT Agent session
-```
-
-### Agent tester
-
-#### Command-line testing
-
-The library also offers a Context Broker and IoT Agent client that can be used to:
-
--   Simulate operations to the Context Broker used by the IoT Agent, triggering Context Provider forwardings for lazy
-    attributes and checking the appropriate values for active ones.
--   Simulate operations to the Device Provisioning API and Configuration API of the IoT Agent.
-
-The tester can be started with the following command, from the root folder of the project:
-
-```console
-bin/iotAgentTester.js
-```
-
-From the command-line, the `help` command can be used to show a description of the currently supported features. These
-are the following:
-
-```text
-stressInit
-
-	Start recording a stress batch.
-
-stressCommit <delay> <times> <threads> <initTime>
-
-	Executes the recorded batch as many times as requested, with delay (ms) between commands.
-	The "threads" parameter indicates how many agents will repeat that same sequence. The "initTime" (ms)
-	parameter indicates the mean of the random initial waiting times for each agent.
-
-exit
-
-	Exit from the command-line.
-
-update <entity> <type> <attributes>
-
-	Update the values of the defined set of attributes, using the following format: name#type=value(|name#type=value)*
-
-append <entity> <type> <attributes>
-
-	Append a new Entity with the defined set of attributes, using the following format: name:type=value(,name:type=value)*
-
-query <entity> <type>
-
-	Get all the information on the selected object.
-
-queryAttr <entity> <type> <attributes>
-
-	Get information on the selected object for the selected attributes.
-
-discover <entity> <type>
-
-	Get all the context providers for a entity and type.
-
-configCb <host> <port> <service> <subservice>
-
-	Config a new host and port for the remote Context Broker.
-
-showConfigCb
-
-	Show the current configuration of the client for the Context Broker.
-
-configIot <host> <port> <service> <subservice>
-
-	Config a new host and port for the remote IoT Agent.
-
-showConfigIot
-
-	Show the current configuration of the client for the IoT Agent.
-
-provision <filename>
-
-	Provision a new device using the Device Provisioning API. The device configuration is
-	read from the file specified in the "filename" parameter.
-
-provisionGroup <template> <data> <type>
-
-	Provision a group of devices with the selected template, taking the information needed to
-	fill the template from a CSV with two columns, DEVICE_ID and DEVICE_NAME. The third parameter, type
-	will be used to replace the DEVICE_TYPE field in the template. All the devices will be provisioned
-	to the same IoT Agent, once the templates have been fulfilled.
-
-listProvisioned
-
-	List all the provisioned devices in an IoT Agent.
-
-removeProvisioned <deviceId>
-
-	Remove the selected provisioned device from the IoT Agent, specified by its Device ID.
-
-addGroup <filename>
-
-	Add a new device group to the specified IoT Agent through the Configuration API. The
-	body is taken from the file specified in the "filename" parameter.
-
-listGroups
-
-	List all the device groups created in the selected IoT Agent for the configured service
-
-removeGroup <apiKey> <resource>
+This project is part of [FIWARE](https://www.fiware.org/). For more information check the FIWARE Catalogue entry for the
+[IoT Agents](https://github.com/Fiware/catalogue/tree/master/iot-agents).
 
-	Remove the device group corresponding to the current configured subservice.
+| :books: [Documentation](https://iotagent-node-lib.rtfd.io) | :mortar_board: [Academy](https://fiware-academy.readthedocs.io/en/latest/iot-agents/idas) | :dart: [Roadmap](https://github.com/telefonicaid/iotagent-node-lib/blob/master/doc/roadmap.md) |
+| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
 
-authenticate <host> <port> <user> <password> <service>
+## Content
 
-	Authenticates to the given authentication server, and use the token in subsequent requests.
+-   [Documentation](#documentation)
+    -   [User documentation](#user-documentation)
+    -   [Development documentation](#development-documentation)
+-   [IoT Agents available](#iot-agents-available)
+-   [Install and usage](#install-and-usage)
+-   [License](#license)
 
-setProtocol <protocol>
+## Documentation
 
-	Sets the protocol to use in the requests (http or https). Defaults to http.
+This repository contains the common user documentation across all IoT Agents. For particular documentation, you can
+check the specific documentation for each IoT Agent. You can see the a list of available IoT Agents in the
+[IoT Agents available](#iot-agents-available) section.
 
-configMigration <host> <port> <originDb>
+### User documentation
 
-	Sets the configuration for a migration between a C++ IoTA and a Node.js one.
+-   [Getting started](doc/getting-started.md)
+-   [IoT Agent API](doc/api.md)
+-   [Administration manual](doc/admin.md)
+-   [Deprecated features](doc/deprecated.md)
+-   [Roadmap](doc/roadmap.md)
 
-showConfigMigration
+### Development documentation
 
-	Shows the current migration configuration.
+-   [Development manual](doc/devel/development.md)
+-   [Contributing guide](doc/devel/contribution-guidelines.md)
+-   [Architecture](doc/devel/architecture.md)
+-   [North Port - NGSI Interactions](doc/devel/northboundinteractions.md)
 
-addProtocols <protocols>
+## IoT Agents available
 
-	Add a protocol translation table, in the following format:
-		protocolOrigin1=protocolTarget1;protocolOrigin2=protocolTarget2...
+The following IoT Agents using the **IoT Agent Node Lib** are available:
 
+-   [IoT Agent JSON](https://github.com/telefonicaid/iotagent-json) - a bridge between HTTP/MQTT messaging (with a JSON
+    payload) and NGSI
+-   [IoT Agent Ultralight](https://github.com/telefonicaid/iotagent-ul) - a bridge between HTTP/MQTT messaging
+-   [IoT Agent LWM2M](https://github.com/telefonicaid/lightweightm2m-iotagent) - a bridge between the
+    [Lightweight M2M](https://www.omaspecworks.org/what-is-oma-specworks/iot/lightweight-m2m-lwm2m/) protocol and NGSI
+    (with an UltraLight2.0 payload) and NGSI
+-   [IoT Agent for Sigfox](https://github.com/telefonicaid/sigfox-iotagent) - a bridge between the
+    [Sigfox](https://www.sigfox.com/en) protocol and NGSI
+-   [IoT Agent for LoRaWAN](https://github.com/Atos-Research-and-Innovation/IoTagent-LoRaWAN) - a bridge between the
+    [LoRaWAN](https://www.thethingsnetwork.org/docs/lorawan/) protocol and NGSI
+-   [IoT Agent for OPC-UA](https://github.com/Engineering-Research-and-Development/iotagent-opcua) - a bridge between
+    the [OPC Unified Architecture](http://www.opcua.us/) protocol and NGSI
+-   [IoT Agent for ISOXML](https://github.com/FIWARE/iotagent-isoxml) - a bridge between the ISOXML/ADAPT protocol for
+    agricultural machinery and NGSI
 
-migrate <targetDb> <service> <subservice>
+## Install and usage
 
-	Migrate all the devices and services for the selected service and subservice into the
-	specified Mongo database. To perform the migration for all the services or all the
-	subservices, use the "*" value.
-```
+The **IoT Agent node library** is not a standalone product. If you plan to install and use any of the IoT Agents
+available, you should follow the installation instructions for each IoT Agent (find the link in the
+[previous section](#iot-agents-available)). You can find the common API provided by the **IoT Agent node library** under
+[API](doc/api.md) documentation.
 
-The agent session stores transient configuration data about the target Context Broker and the target IoT Agent. This
-configuration is independent, and can be checked with the `showConfigCb` and `showConfigIot` commands, respectively.
-Their values can be changed with the `configCb` and `configIot` commands respectively. The new configurations will be
-deleted upon startup.
+If you plan to use the IoT Agent node library in your own project or IoT Agent, you should follow the
+[Developer manual](doc/devel/development.md), which includes the installation instructions and the usage of the library.
 
----
+Information about how to configure the IoT agent or the library can be found at the corresponding section of the
+[Administration manual](doc/admin.md).
 
-## Licence
+## License
 
 The IoT Agent Node Library is licensed under [Affero General Public License (GPL) version 3](./LICENSE).
 

package/config.js

@@ -77,7 +77,9 @@ var config = {
     providerUrl: 'http://192.168.56.1:4041',
     deviceRegistrationDuration: 'P1M',
     defaultType: 'Thing',
-    expressLimit: '1Mb'
+    expressLimit: '1Mb',
+    useCBflowControl: false,
+    storeLastMeasure: false
 };
 
 module.exports = config;

package/doc/README.md

@@ -4,7 +4,7 @@
 
 -   [Getting started](getting-started.md)
 -   [IoT Agent API](api.md)
--   [Installation and administration manual](admin.md)
+-   [Administration manual](admin.md)
 -   [Deprecated features](deprecated.md)
 -   [Roadmap](roadmap.md)
 

package/doc/admin.md

@@ -5,7 +5,6 @@
         -   [loglevel](#loglevel)
         -   [contextBroker](#contextbroker)
         -   [server](#server)
-        -   [stats](#stats)
         -   [authentication](#authentication)
         -   [deviceRegistry](#deviceregistry)
         -   [mongodb](#mongodb)
@@ -126,9 +125,9 @@ allowing the computer to interpret the rest of the data with more clarity and de
 ```
 
 Under mixed mode, **NGSI v2** payloads are used for context broker communications by default, but this payload may also
-be switched to **NGSI LD** at service group or device provisioning time using the `ngsiVersion` field in the
-provisioning API. The `ngsiVersion` field switch may be added at either group or device level, with the device level
-overriding the group setting.
+be switched to **NGSI LD** at group or device provisioning time using the `ngsiVersion` field in the provisioning API.
+The `ngsiVersion` field switch may be added at either group or device level, with the device level overriding the group
+setting.
 
 #### `server`
 
@@ -159,17 +158,6 @@ support nulls or multi-attribute requests if they are encountered.
 }
 ```
 
-#### `stats`
-
-It configures the periodic collection of statistics. Use `interval` in milliseconds to set the time between stats
-writings.
-
-```javascript
-stats: {
-    interval: 100;
-}
-```
-
 #### `authentication`
 
 Stores the authentication data, for use in retrieving tokens for devices with a trust token (required in scenarios with
@@ -318,7 +306,8 @@ added `agentPath`:
 
 #### `types`
 
-See **Type Configuration** in the [Configuration API](#configurationapi) section below.
+This parameter includes additional groups configuration as described into the
+[Config group API](api.md#config-group-api) section.
 
 #### `service`
 
@@ -356,7 +345,7 @@ if this flag is activated:
 
 #### `defaultResource`
 
-default string to use as resource for the registration of new Configurations (if no resource is provided).
+default string to use as resource for the registration of new config groups (if no resource is provided).
 
 #### `defaultKey`
 
@@ -427,7 +416,38 @@ IotAgents, as all Express applications that use the body-parser middleware, have
 size that the application will handle. This default limit for ioiotagnets are 1Mb. So, if your IotAgent receives a
 request with a body that exceeds this limit, the application will throw a “Error: Request entity too large”.
 
-The 1Mb default can be changed setting the `expressLimit` configuration parameter (or equivalente `IOTA_EXPRESS_LIMIT` environment variable).
+The 1Mb default can be changed setting the `expressLimit` configuration parameter (or equivalente `IOTA_EXPRESS_LIMIT`
+environment variable).
+
+#### `storeLastMeasure`
+
+If this flag is activated, last measure arrived to Device IoTAgent without be processed will be stored in Device under
+`lastMeasure` field (composed of sub-fields `timestamp` and `measure` for the measure itself, in multi-measure format).
+This flag is overwritten by `storeLastMeasure` flag in group or device. This flag is disabled by default.
+
+For example in a device document stored in MongoDB will be extended with a subdocument named lastMeasure like this:
+
+```json
+{
+    "lastMeasure": {
+        "timestamp": "2025-01-09T10:35:33.079Z",
+        "measure": [
+            [
+                {
+                    "name": "level",
+                    "type": "Text",
+                    "value": 33
+                }
+            ]
+        ]
+    }
+}
+```
+
+#### `useCBflowControl`
+
+If this flag is activated, when iotAgent invokes Context Broker will use [flowControl option](https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/admin/perf_tuning.md#updates-flow-control-mechanism). This flag is overwritten by
+`useCBflowControl` flag in group or device. This flag is disabled by default.
 
 ### Configuration using environment variables
 
@@ -491,6 +511,8 @@ overrides.
 | IOTA_DEFAULT_ENTITY_NAME_CONJUNCTION | `defaultEntityNameConjunction`  |
 | IOTA_RELAX_TEMPLATE_VALIDATION       | `relaxTemplateValidation`       |
 | IOTA_EXPRESS_LIMIT                   | `expressLimit`                  |
+| IOTA_STORE_LAST_MEASURE              | `storeLastMeasure`              |
+| IOTA_CB_FLOW_CONTROL                 | `useCBflowControl`              |
 
 Note: