@itentialopensource/adapter-kafkav2 0.23.4 → 0.24.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/.eslintrc.js CHANGED
@@ -9,6 +9,7 @@ module.exports = {
9
9
  'json'
10
10
  ],
11
11
  parserOptions: {
12
+ ecmaVersion: 2020,
12
13
  sourceType: 'module'
13
14
  },
14
15
  rules: {
@@ -0,0 +1 @@
1
+ []
package/AUTH.md CHANGED
@@ -1,13 +1,13 @@
1
1
  ## Authenticating Kafka v2 Adapter
2
2
 
3
- This document will go through the steps for authenticating the Kafka v2 adapter with. Properly configuring the properties for an adapter in IAP is critical for getting the adapter online. You can read more about adapter authentication <a href="https://docs.itential.com/opensource/docs/authentication" target="_blank">HERE</a>.
3
+ This document will go through the steps for authenticating the Kafka v2 adapter with. Properly configuring the properties for an adapter in Itential Platform is critical for getting the adapter online. You can read more about adapter authentication <a href="https://docs.itential.com/opensource/docs/authentication" target="_blank">HERE</a>.
4
4
 
5
5
  ### Library Authentication
6
6
  The Kafka v2 adapter supports SASL Authentication for Kafka v2 server. If you change authentication methods, you should change this section accordingly and merge it back into the adapter repository.
7
7
 
8
8
  STEPS
9
9
  1. Ensure you have access to a Kafka v2 server and that it is running
10
- 2. Follow the steps in the README.md to import the adapter into IAP if you have not already done so
10
+ 2. Follow the steps in the README.md to import the adapter into Itential Platform if you have not already done so
11
11
  3. Use the properties below for the ```properties.client.sasl``` field
12
12
  4. Kafka v2 adapter supports both PLAIN and SCRAM-SHA authentication mechanism
13
13
 
@@ -33,6 +33,6 @@ For more details on sasl authentication follow the README.md
33
33
 
34
34
  ### Troubleshooting
35
35
  - Make sure you copied over the correct username and password.
36
- - Turn on debug level logs for the adapter in IAP Admin Essentials.
36
+ - Turn on debug level logs for the adapter in Itential Platform Admin Essentials.
37
37
  - Investigate the logs
38
38
  - Credentials should be ** masked ** by the adapter so make sure you verify the username and password - including that there are erroneous spaces at the front or end.
package/ENHANCE.md CHANGED
@@ -2,7 +2,7 @@
2
2
 
3
3
  ### Adding a Second Instance of an Adapter
4
4
 
5
- You can add a second instance of this adapter without adding new code on the file system. To do this go into the IAP Admin Essentials and add a new service config for this adapter. The two instances of the adapter should have unique ids. In addition, they should point to different instances (unique host and port) of the other system.
5
+ You can add a second instance of this adapter without adding new code on the file system. To do this go into the Itential Platform Admin Essentials and add a new service config for this adapter. The two instances of the adapter should have unique ids. In addition, they should point to different instances (unique host and port) of the other system.
6
6
 
7
7
  ### Adding Adapter Calls
8
8
 
@@ -18,7 +18,7 @@ npm run adapter:update
18
18
 
19
19
  If you do not have a Swagger or OpenAPI document, you can use a Postman Collection and convert that to an OpenAPI document using APIMatic and then follow the first process.
20
20
 
21
- If you want to manually update the adapter that can also be done the key thing is to make sure you update all of the right files. Within the entities directory you will find 1 or more entities. You can create a new entity or add to an existing entity. Each entity has an action.json file, any new call will need to be put in the action.json file. It will also need to be added to the enum for the ph_request_type in the appropriate schema files. Once this configuration is complete you will need to add the call to the adapter.js file and, in order to make it available as a workflow task in IAP, it should also be added to the pronghorn.json file. You can optionally add it to the unit and integration test files. There is more information on how to work on each of these files in the <a href="https://docs.itential.com/opensource/docs/adapters" target="_blank">Adapter Technical Resources</a> on our Documentation Site.
21
+ If you want to manually update the adapter that can also be done the key thing is to make sure you update all of the right files. Within the entities directory you will find 1 or more entities. You can create a new entity or add to an existing entity. Each entity has an action.json file, any new call will need to be put in the action.json file. It will also need to be added to the enum for the ph_request_type in the appropriate schema files. Once this configuration is complete you will need to add the call to the adapter.js file and, in order to make it available as a workflow task in Itential Platform, it should also be added to the pronghorn.json file. You can optionally add it to the unit and integration test files. There is more information on how to work on each of these files in the <a href="https://docs.itential.com/opensource/docs/adapters" target="_blank">Adapter Technical Resources</a> on our Documentation Site.
22
22
 
23
23
  ```text
24
24
  Files to update
@@ -32,7 +32,7 @@ Files to update
32
32
 
33
33
  ### Adding Adapter Properties
34
34
 
35
- While changing adapter properties is done in the service instance configuration section of IAP, adding properties has to be done in the adapter. To add a property you should edit the propertiesSchema.json with the proper information for the property. In addition, you should modify the sampleProperties to have the new property in it.
35
+ While changing adapter properties is done in the service instance configuration section of Itential Platform, adding properties has to be done in the adapter. To add a property you should edit the propertiesSchema.json with the proper information for the property. In addition, you should modify the sampleProperties to have the new property in it.
36
36
 
37
37
  ```text
38
38
  Files to update
package/README.md CHANGED
@@ -1,16 +1,16 @@
1
1
  # Kafkav2 Adapter
2
2
 
3
- This adapter is used to integrate the Itential Automation Platform (IAP) with the Kafka System. The documenation for kafkajs is available at [https://kafka.js.org/docs/getting-started]. The adapter utilizes the kafkjs library to provide the integrations that are deemed pertinent to IAP. This Readme file is intended to provide information on this adapter.
3
+ This adapter is used to integrate the Itential Platform with the Kafka System. The documenation for kafkajs is available at [https://kafka.js.org/docs/getting-started]. The adapter utilizes the kafkjs library to provide the integrations that are deemed pertinent to Itential Platform. This Readme file is intended to provide information on this adapter.
4
4
 
5
5
  >**Note**: It is possible that some integrations will be supported through the Kafka adapter while other integrations will not.
6
6
 
7
7
  Itential provides information on all of its product adapters in the Customer Knowledge Base. Information in the [Customer Knowledge Base](https://itential.atlassian.net/servicedesk/customer/portals) is consistently maintained and goes through documentation reviews. As a result, it should be the first place to go for information.
8
8
 
9
- For custom built adapters, it is a starting point to understand what you have built, provide the information for you to be able to update the adapter, and assist you with deploying the adapter into IAP.
9
+ For custom built adapters, it is a starting point to understand what you have built, provide the information for you to be able to update the adapter, and assist you with deploying the adapter into Itential Platform.
10
10
 
11
11
  ## Versioning
12
12
 
13
- Itential Product adapters utilize SemVer for versioning. The current version of the adapter can be found in the `package.json` file or viewed in the IAP GUI on the System page. For Open Source Adapters, the versions available can be found in the [Itential OpenSource Repository](https://www.npmjs.com/search?q=itentialopensource%2Fadapter).
13
+ Itential Product adapters utilize SemVer for versioning. The current version of the adapter can be found in the `package.json` file or viewed in the Itential Platform GUI on the System page. For Open Source Adapters, the versions available can be found in the [Itential OpenSource Repository](https://www.npmjs.com/search?q=itentialopensource%2Fadapter).
14
14
 
15
15
  ## Release History
16
16
 
@@ -77,7 +77,7 @@ npm install
77
77
 
78
78
  ## Installing an Itential Product Adapter
79
79
 
80
- 1. Set up the name space location in your IAP node_modules.
80
+ 1. Set up the name space location in your Itential Platform node_modules.
81
81
 
82
82
  ```json
83
83
  cd /opt/pronghorn/current/node_modules
@@ -85,7 +85,7 @@ if the @itentialopensource directory does not exist, create it:
85
85
  mkdir @itentialopensource
86
86
  ```
87
87
 
88
- 1. Clone the adapter into your IAP environment.
88
+ 1. Clone the adapter into your Itential Platform environment.
89
89
 
90
90
  ```json
91
91
  cd \@itentialopensource
@@ -102,7 +102,7 @@ npm install
102
102
  1. Add the adapter properties for Kafka (created from Adapter Builder) to the `properties.json` file for your Itential build. You will need to change the credentials and possibly the host information below.
103
103
  [Kafka sample properties](sampleProperties.json)
104
104
 
105
- 1. Restart IAP
105
+ 1. Restart Itential Platform
106
106
 
107
107
  ```json
108
108
  systemctl restart pronghorn
@@ -110,7 +110,7 @@ systemctl restart pronghorn
110
110
 
111
111
  ## Adapter Properties and Descriptions
112
112
 
113
- This section defines **all** the properties that are available for the adapter, including detailed information on what each property is for. If you are not using certain capabilities with this adapter, you do not need to define all of the properties. An example of how the properties for this adapter can be used with tests or IAP are provided in the **Installation** section.
113
+ This section defines **all** the properties that are available for the adapter, including detailed information on what each property is for. If you are not using certain capabilities with this adapter, you do not need to define all of the properties. An example of how the properties for this adapter can be used with tests or Itential Platform are provided in the **Installation** section.
114
114
 
115
115
  ```json
116
116
  {
@@ -157,6 +157,14 @@ Sample SSL and SASL properties that go under client props. In the below example
157
157
  }
158
158
  ```
159
159
 
160
+ Sample socket properties
161
+ ```json
162
+ "socketOptions": {
163
+ "keepAlive": true,
164
+ "keepAliveInterval": 45000,
165
+ "socketTimeout": 20000
166
+ }
167
+ ```
160
168
 
161
169
  ### Topic Properties
162
170
 
@@ -311,6 +319,7 @@ The following properties are used to define the Kafka Client. These properties a
311
319
  | retry -> retries | Max number of retries per call. default: 5 |
312
320
  | ssl | Object, options to be passed to the tls broker sockets, ex. { rejectUnauthorized: false }.|
313
321
  | sasl | Object, SASL authentication configuration (Currently, supports PLAIN, SCRAM-SHA-256, SCRAM-SHA-512), ex. { mechanism: 'plain', username: 'foo', password: 'bar' }.|
322
+ | socketOptions | Object, Socket Options: Configuration for socket behavior, including keep-alive, interval, and timeout settings.|
314
323
 
315
324
  For all client config options see [Client Config](https://kafka.js.org/docs/configuration)
316
325
 
@@ -337,13 +346,13 @@ Refer to kafkajs docs for all consumer options [Consumer](https://kafka.js.org/d
337
346
 
338
347
  ## Parsing Properties
339
348
 
340
- The `parseMessage` property allows the user to define how they want the Kafka message to be published to IAP's event system. If `parseMessage` is set to true or omitted, the value of the Kafka message will be parsed as either an object or string and wrapped in an outer object. The wrapper object's key can be defined with the property `wrapMessage`, or the default value `payload` can be used if omitted. If `parseMessage` is set to false, the entire kafka payload, including metadata, would be returned and the message itself would need to be transformed at a later point.
349
+ The `parseMessage` property allows the user to define how they want the Kafka message to be published to Itential Platform's event system. If `parseMessage` is set to true or omitted, the value of the Kafka message will be parsed as either an object or string and wrapped in an outer object. The wrapper object's key can be defined with the property `wrapMessage`, or the default value `payload` can be used if omitted. If `parseMessage` is set to false, the entire kafka payload, including metadata, would be returned and the message itself would need to be transformed at a later point.
341
350
 
342
351
  ## Turning off stream if WorkflowEngine or OperationsManager is down
343
352
 
344
- `check_iap_apps` (boolean) and `iap_apps_check_interval` (integer) are used when the user wants to turn off stream if WorkflowEngine or OperationsManager is down. If `check_iap_apps` is set to true, by default, the adapter will check the status of both WorkflowEngine and OperationsManager at a defined interval. If any IAP apps are down, the consumer will be paused until the apps are active again. If the user does not want to healthcheck OperationsManager, set `check_ops_manager_status` to false. Similarly, if the user does not want to check WorkflowEngine status, set `check_wfe_status` to false.
353
+ `check_iap_apps` (boolean) and `iap_apps_check_interval` (integer) are used when the user wants to turn off stream if WorkflowEngine or OperationsManager is down. If `check_iap_apps` is set to true, by default, the adapter will check the status of both WorkflowEngine and OperationsManager at a defined interval. If any Itential Platform apps are down, the consumer will be paused until the apps are active again. If the user does not want to healthcheck OperationsManager, set `check_ops_manager_status` to false. Similarly, if the user does not want to check WorkflowEngine status, set `check_wfe_status` to false.
345
354
 
346
- `iap_apps_check_interval` (default 30000ms - 30s) allows the user to set the frequency in which to run IAP app healtcheck.
355
+ `iap_apps_check_interval` (default 30000ms - 30s) allows the user to set the frequency in which to run Itential Platform app healtcheck.
347
356
 
348
357
  ## Send message sample payload
349
358
 
@@ -360,9 +369,9 @@ Note that this is an array, so it can have multiple topics.
360
369
  ```
361
370
  For all options of messages array please follow [kafkajs documentation](https://kafka.js.org/docs/producing#producing-messages)
362
371
 
363
- ## Triggering jobs in IAP from a kafka message
372
+ ## Triggering jobs in Itential Platform from a kafka message
364
373
 
365
- Once you have the adapter configured and online in an IAP instance, you can create triggers in Operations Manager to kick off jobs when a kafka message is consumed. The adapter listens for messages on the subscribed topics and publishes to an event. This event is picked up by the Operations Manager. If you have created triggers for this specific topic, a job will get triggered. For a step-by-step example follow the next sub-section.
374
+ Once you have the adapter configured and online in an Itential Platform instance, you can create triggers in Operations Manager to kick off jobs when a kafka message is consumed. The adapter listens for messages on the subscribed topics and publishes to an event. This event is picked up by the Operations Manager. If you have created triggers for this specific topic, a job will get triggered. For a step-by-step example follow the next sub-section.
366
375
 
367
376
  ### Listen to a topic called test-topic and trigger a workflow called test
368
377
 
package/SUMMARY.md CHANGED
@@ -1,8 +1,8 @@
1
1
  ## Overview
2
- This adapter is used to integrate the Itential Automation Platform (IAP) with the Kafka System. The adapter utilizes the KafkaJS client to integrate with Apache Kafka. The ReadMe file is intended to provide information on this adapter it is generated from various other Markdown files.
2
+ This adapter is used to integrate the Itential Platform with the Kafka System. The adapter utilizes the KafkaJS client to integrate with Apache Kafka. The ReadMe file is intended to provide information on this adapter it is generated from various other Markdown files.
3
3
 
4
4
  >**Note**: It is possible that some integrations will be supported through the KafkaV2 adapter while other integrations will not.
5
5
 
6
6
  Itential provides information on all of its product adapters in the Customer Knowledge Base. Information in the <a href="https://itential.atlassian.net/servicedesk/customer/portals" target="_blank">Customer Knowledge Base</a> is consistently maintained and goes through documentation reviews. As a result, it should be the first place to go for information.
7
7
 
8
- For opensourced and custom built adapters, the ReadMe is a starting point to understand what you have built, provide the information for you to be able to update the adapter, and assist you with deploying the adapter into IAP.
8
+ For opensourced and custom built adapters, the ReadMe is a starting point to understand what you have built, provide the information for you to be able to update the adapter, and assist you with deploying the adapter into Itential Platform.
package/SYSTEMINFO.md CHANGED
@@ -16,7 +16,7 @@ Apache Kafka is an open-source distributed event streaming platform used by thou
16
16
  "Store streams of data safely in a distributed, durable, fault-tolerant cluster."
17
17
 
18
18
  ## Why Integrate
19
- The Kafka v2 adapter from Itential is used to integrate the Itential Automation Platform (IAP) with Kafka. With this adapter you have the ability to perform operations with Kafka on items such as:
19
+ The Kafka v2 adapter from Itential is used to integrate the Itential Platform with Kafka. With this adapter you have the ability to perform operations with Kafka on items such as:
20
20
 
21
21
  - Produce/Publish messages onto message topics for others to consume
22
22
  - Listen, receive and process messages placed on topics by other systems
package/TAB1.md CHANGED
@@ -1,9 +1,9 @@
1
1
  # Overview
2
- This adapter is used to integrate the Itential Automation Platform (IAP) with the Kafka System. The adapter utilizes the KafkaJS client to integrate with Apache Kafka. The ReadMe file is intended to provide information on this adapter it is generated from various other Markdown files.
2
+ This adapter is used to integrate the Itential Platform with the Kafka System. The adapter utilizes the KafkaJS client to integrate with Apache Kafka. The ReadMe file is intended to provide information on this adapter it is generated from various other Markdown files.
3
3
 
4
4
 
5
5
  ## Details
6
- The Kafka v2 adapter from Itential is used to integrate the Itential Automation Platform (IAP) with Kafka. With this adapter you have the ability to perform operations with Kafka on items such as:
6
+ The Kafka v2 adapter from Itential is used to integrate the Itential Platform with Kafka. With this adapter you have the ability to perform operations with Kafka on items such as:
7
7
 
8
8
  - Produce/Publish messages onto message topics for others to consume
9
9
  - Listen, receive and process messages placed on topics by other systems
package/TAB2.md CHANGED
@@ -11,14 +11,14 @@
11
11
  ## Specific Adapter Information
12
12
  ### Authentication
13
13
 
14
- This document will go through the steps for authenticating the Kafka v2 adapter with. Properly configuring the properties for an adapter in IAP is critical for getting the adapter online. You can read more about adapter authentication <a href="https://docs.itential.com/opensource/docs/authentication" target="_blank">HERE</a>.
14
+ This document will go through the steps for authenticating the Kafka v2 adapter with. Properly configuring the properties for an adapter in Itential Platform is critical for getting the adapter online. You can read more about adapter authentication <a href="https://docs.itential.com/opensource/docs/authentication" target="_blank">HERE</a>.
15
15
 
16
16
  #### Library Authentication
17
17
  The Kafka v2 adapter supports SASL Authentication for Kafka v2 server. If you change authentication methods, you should change this section accordingly and merge it back into the adapter repository.
18
18
 
19
19
  STEPS
20
20
  1. Ensure you have access to a Kafka v2 server and that it is running
21
- 2. Follow the steps in the README.md to import the adapter into IAP if you have not already done so
21
+ 2. Follow the steps in the README.md to import the adapter into Itential Platform if you have not already done so
22
22
  3. Use the properties below for the ```properties.client.sasl``` field
23
23
  4. Kafka v2 adapter supports both PLAIN and SCRAM-SHA authentication mechanism
24
24
 
@@ -44,12 +44,12 @@ For more details on sasl authentication follow the README.md
44
44
 
45
45
  #### Troubleshooting
46
46
  - Make sure you copied over the correct username and password.
47
- - Turn on debug level logs for the adapter in IAP Admin Essentials.
47
+ - Turn on debug level logs for the adapter in Itential Platform Admin Essentials.
48
48
  - Investigate the logs
49
49
  - Credentials should be ** masked ** by the adapter so make sure you verify the username and password - including that there are erroneous spaces at the front or end.
50
50
  ### Sample Properties
51
51
 
52
- Sample Properties can be used to help you configure the adapter in the Itential Automation Platform. You will need to update connectivity information such as the host, port, protocol and credentials.
52
+ Sample Properties can be used to help you configure the adapter in the Itential Platform. You will need to update connectivity information such as the host, port, protocol and credentials.
53
53
 
54
54
  ```json
55
55
  "properties": {
@@ -69,6 +69,16 @@ Sample Properties can be used to help you configure the adapter in the Itential
69
69
  ],
70
70
  "clientId": "my-app",
71
71
  "logLevel": "INFO",
72
+ "connectionTimeout": 1000,
73
+ "requestTimeout": 30000,
74
+ "enforceRequestTimeout": true,
75
+ "retry": {
76
+ "maxRetryTime": 30000,
77
+ "initialRetryTime": 300,
78
+ "factor": 0.2,
79
+ "multiplier": 2,
80
+ "retries": 5
81
+ },
72
82
  "ssl": {
73
83
  "enableTrace": true,
74
84
  "ca": "/path/to/crt.pem",
@@ -78,6 +88,11 @@ Sample Properties can be used to help you configure the adapter in the Itential
78
88
  "username": "my-user",
79
89
  "password": "my-password",
80
90
  "mechanism": "scram-sha-512"
91
+ },
92
+ "socketOptions": {
93
+ "keepAlive": true,
94
+ "keepAliveInterval": 45000,
95
+ "socketTimeout": 20000
81
96
  }
82
97
  },
83
98
  "producer": {
package/adapter.js CHANGED
@@ -17,11 +17,16 @@
17
17
  const fs = require('fs-extra');
18
18
  const path = require('path');
19
19
  const util = require('util');
20
+ const net = require('net');
21
+ const tls = require('tls');
20
22
  /* Fetch in the other needed components for the this Adaptor */
21
23
  // const EventEmitterCl = require('events').EventEmitter;
22
24
  const AdapterBaseCl = require(path.join(__dirname, 'adapterBase.js'));
23
25
  const DBUtil = require(path.join(__dirname, 'utils', 'dbUtil.js'));
24
26
 
27
+ const DEFAULT_KEEP_ALIVE_INTERVAL = 60000; // 60s
28
+ const DEFAULT_SOCKET_TIMEOUT = 30000; // 30s
29
+
25
30
  const axios = require('axios');
26
31
 
27
32
  let needRestart = false;
@@ -216,6 +221,51 @@ function formatErrorObject(origin, type, variables, sysCode, sysRes, stack) {
216
221
  return errorObject;
217
222
  }
218
223
 
224
+ /**
225
+ * @summary Creates a socket factory for KafkaJS to handle custom socket connections.
226
+ *
227
+ * @function createSocketFactory
228
+ * @param {Object} socketOptions - Configuration options for the socket connection.
229
+ * @param {Boolean} useSSL - Determines whether SSL should be used for the connection.
230
+ *
231
+ * @return {Function} - A socket factory function that establishes a connection to Kafka brokers.
232
+ */
233
+ function createSocketFactory(socketOptions, useSSL) {
234
+ return (options) => {
235
+ console.log('KafkaJS Socket Options:', options); // See what KafkaJS is actually passing
236
+
237
+ // Destructure properties from the options object
238
+ const { host, port, ssl } = options;
239
+ const brokerAddress = `${host}:${port}`;
240
+ console.log(
241
+ `Connecting to Kafka broker: ${brokerAddress} (${host}:${port}) ${useSSL ? 'with SSL' : 'without SSL'
242
+ }`
243
+ );
244
+
245
+ const socket = useSSL
246
+ ? tls.connect({ host, port, rejectUnauthorized: false }) // Use TLS for SSL
247
+ : net.createConnection({ host, port }); // Use Net for Non-SSL
248
+
249
+ const keepAliveEnabled = socketOptions && socketOptions.keepAlive !== undefined
250
+ ? socketOptions.keepAlive
251
+ : true;
252
+ const keepAliveInterval = socketOptions && socketOptions.keepAliveInterval !== undefined
253
+ ? socketOptions.keepAliveInterval
254
+ : DEFAULT_KEEP_ALIVE_INTERVAL; // 60s default
255
+ const socketTimeout = socketOptions && socketOptions.socketTimeout !== undefined
256
+ ? socketOptions.socketTimeout
257
+ : DEFAULT_SOCKET_TIMEOUT; // 30s default
258
+
259
+ socket.setKeepAlive(keepAliveEnabled, keepAliveInterval);
260
+ socket.setTimeout(socketTimeout);
261
+
262
+ socket.on('error', (err) => console.error(`Socket error with ${brokerAddress}:`, err));
263
+ socket.on('timeout', () => console.warn(`Socket timeout with ${brokerAddress}`));
264
+
265
+ return socket;
266
+ };
267
+ }
268
+
219
269
  /**
220
270
  * This is the adapter/interface into Kafka
221
271
  */
@@ -645,6 +695,14 @@ class Kafkav2 extends AdapterBaseCl {
645
695
  log.info('EMITTED ONLINE ON STUB MODE');
646
696
  return;
647
697
  }
698
+ const combinedProps = this.props.client || {};
699
+ // Extract socket options with default values
700
+ const socketOptions = this.props.client.socketOptions || null;
701
+ const useSSL = this.props.client.ssl || false; // Determine SSL usage
702
+
703
+ if (socketOptions) {
704
+ combinedProps.socketFactory = createSocketFactory(socketOptions, useSSL);
705
+ }
648
706
 
649
707
  if (!needRestart) {
650
708
  try {
@@ -653,7 +711,7 @@ class Kafkav2 extends AdapterBaseCl {
653
711
  this.props.client.logLevel = logLevelEnum;
654
712
  }
655
713
  // Create a kafka client
656
- const combinedProps = this.props.client || {};
714
+
657
715
  if (this.props.client.ssl && this.props.client.ssl.ca) {
658
716
  combinedProps.ssl.ca = [fs.readFileSync(this.props.client.ssl.ca, 'utf-8')];
659
717
  }
package/package.json CHANGED
@@ -1,12 +1,11 @@
1
1
  {
2
2
  "name": "@itentialopensource/adapter-kafkav2",
3
- "version": "0.23.4",
3
+ "version": "0.24.0",
4
4
  "description": "Itential adapter to connect to kafka",
5
5
  "main": "adapter.js",
6
- "wizardVersion": "2.35.0",
7
- "engineVersion": "1.40.2",
6
+ "wizardVersion": "2.44.7",
7
+ "engineVersion": "1.69.14",
8
8
  "scripts": {
9
- "artifactize": "npm i && node utils/packModificationScript.js",
10
9
  "preinstall": "node utils/setup.js",
11
10
  "deinstall": "node utils/removeHooks.js",
12
11
  "lint": "node --max_old_space_size=4096 ./node_modules/eslint/bin/eslint.js . --ext .json --ext .js",
@@ -20,10 +19,9 @@
20
19
  },
21
20
  "keywords": [
22
21
  "Itential",
23
- "IAP",
22
+ "Itential Platform",
24
23
  "Automation",
25
24
  "Integration",
26
- "App-Artifacts",
27
25
  "Adapter",
28
26
  "Notification",
29
27
  "Messaging",
@@ -43,26 +41,26 @@
43
41
  "author": "Itential",
44
42
  "homepage": "https://gitlab.com/itentialopensource/adapters/adapter-kafkav2",
45
43
  "dependencies": {
46
- "ajv": "^8.17.1",
47
- "avro-schema-registry": "^2.1.0",
48
- "avsc": "^5.4.21",
49
- "axios": "^1.8.2",
50
- "fs-extra": "^11.2.0",
51
- "json-query": "^2.2.2",
44
+ "ajv": "8.17.1",
45
+ "avro-schema-registry": "2.1.0",
46
+ "avsc": "5.4.21",
47
+ "axios": "1.9.0",
48
+ "fs-extra": "11.3.0",
49
+ "json-query": "2.2.2",
52
50
  "kafkajs": "2.2.3",
53
- "mongodb": "^6.2.0",
54
- "readline-sync": "^1.4.10",
55
- "uuid": "^3.0.1"
51
+ "mongodb": "4.17.2",
52
+ "readline-sync": "1.4.10",
53
+ "uuid": "3.0.1",
54
+ "mocha": "10.8.2",
55
+ "winston": "3.17.0"
56
56
  },
57
57
  "devDependencies": {
58
- "chai": "^4.3.7",
59
- "eslint": "^8.44.0",
60
- "eslint-config-airbnb-base": "^15.0.0",
61
- "eslint-plugin-import": "^2.27.5",
62
- "eslint-plugin-json": "^3.1.0",
63
- "mocha": "^10.7.0",
64
- "testdouble": "^3.18.0",
65
- "winston": "^3.13.1"
58
+ "chai": "4.5.0",
59
+ "eslint": "8.57.0",
60
+ "eslint-config-airbnb-base": "15.0.0",
61
+ "eslint-plugin-import": "2.31.0",
62
+ "eslint-plugin-json": "3.1.0",
63
+ "testdouble": "3.18.0"
66
64
  },
67
65
  "private": false
68
66
  }
@@ -21,16 +21,18 @@
21
21
  ]
22
22
  }
23
23
  },
24
- "parseMessage" :{
24
+ "parseMessage": {
25
25
  "type": "boolean",
26
26
  "description": "Whether or not to prase the message as JSON when consumed by IAP",
27
- "default" : true
27
+ "default": true
28
28
  },
29
- "wrapMessage" : {
29
+ "wrapMessage": {
30
30
  "type": "string",
31
31
  "description": "A key to wrap the parsed JSON message",
32
- "examples" : [ "message"],
33
- "default" : "payload"
32
+ "examples": [
33
+ "message"
34
+ ],
35
+ "default": "payload"
34
36
  },
35
37
  "interval_time": {
36
38
  "type": "integer",
@@ -79,6 +81,26 @@
79
81
  },
80
82
  "sasl": {
81
83
  "type": "object"
84
+ },
85
+ "socketOptions": {
86
+ "type": "object",
87
+ "properties": {
88
+ "keepAlive": {
89
+ "type": "boolean",
90
+ "description": "Whether to enable keep-alive on the socket."
91
+ },
92
+ "keepAliveInterval": {
93
+ "type": "integer",
94
+ "minimum": 0,
95
+ "description": "Interval (in milliseconds) for keep-alive packets."
96
+ },
97
+ "socketTimeout": {
98
+ "type": "integer",
99
+ "minimum": 0,
100
+ "description": "Timeout (in milliseconds) for socket inactivity before closing."
101
+ }
102
+ },
103
+ "additionalProperties": false
82
104
  }
83
105
  }
84
106
  },
@@ -37,6 +37,11 @@
37
37
  "username": "my-user",
38
38
  "password": "my-password",
39
39
  "mechanism": "scram-sha-512"
40
+ },
41
+ "socketOptions": {
42
+ "keepAlive": true,
43
+ "keepAliveInterval": 45000,
44
+ "socketTimeout": 20000
40
45
  }
41
46
  },
42
47
  "producer": {
@@ -14,11 +14,11 @@ const winston = require('winston');
14
14
  const { expect } = require('chai');
15
15
  const { use } = require('chai');
16
16
  const td = require('testdouble');
17
+ const log = require('../../utils/logger');
17
18
 
18
19
  const anything = td.matchers.anything();
19
20
 
20
21
  // stub and attemptTimeout are used throughout the code so set them here
21
- let logLevel = 'none';
22
22
  const stub = true;
23
23
  const isRapidFail = false;
24
24
  const isSaveMockData = false;
@@ -58,43 +58,6 @@ global.pronghornProps = {
58
58
 
59
59
  global.$HOME = `${__dirname}/../..`;
60
60
 
61
- // set the log levels that Pronghorn uses, spam and trace are not defaulted in so without
62
- // this you may error on log.trace calls.
63
- const myCustomLevels = {
64
- levels: {
65
- spam: 6,
66
- trace: 5,
67
- debug: 4,
68
- info: 3,
69
- warn: 2,
70
- error: 1,
71
- none: 0
72
- }
73
- };
74
-
75
- // need to see if there is a log level passed in
76
- process.argv.forEach((val) => {
77
- // is there a log level defined to be passed in?
78
- if (val.indexOf('--LOG') === 0) {
79
- // get the desired log level
80
- const inputVal = val.split('=')[1];
81
-
82
- // validate the log level is supported, if so set it
83
- if (Object.hasOwnProperty.call(myCustomLevels.levels, inputVal)) {
84
- logLevel = inputVal;
85
- }
86
- }
87
- });
88
-
89
- // need to set global logging
90
- global.log = winston.createLogger({
91
- level: logLevel,
92
- levels: myCustomLevels.levels,
93
- transports: [
94
- new winston.transports.Console()
95
- ]
96
- });
97
-
98
61
  /**
99
62
  * Runs the common asserts for test
100
63
  */
@@ -14,11 +14,11 @@ const execute = require('child_process').execSync;
14
14
  const { expect } = require('chai');
15
15
  const { use } = require('chai');
16
16
  const td = require('testdouble');
17
+ const log = require('../../utils/logger');
17
18
 
18
19
  const anything = td.matchers.anything();
19
20
 
20
21
  // stub and attemptTimeout are used throughout the code so set them here
21
- let logLevel = 'none';
22
22
  const stub = true;
23
23
  const isRapidFail = false;
24
24
  const attemptTimeout = 120000;
@@ -53,43 +53,6 @@ global.pronghornProps = {
53
53
 
54
54
  global.$HOME = `${__dirname}/../..`;
55
55
 
56
- // set the log levels that Pronghorn uses, spam and trace are not defaulted in so without
57
- // this you may error on log.trace calls.
58
- const myCustomLevels = {
59
- levels: {
60
- spam: 6,
61
- trace: 5,
62
- debug: 4,
63
- info: 3,
64
- warn: 2,
65
- error: 1,
66
- none: 0
67
- }
68
- };
69
-
70
- // need to see if there is a log level passed in
71
- process.argv.forEach((val) => {
72
- // is there a log level defined to be passed in?
73
- if (val.indexOf('--LOG') === 0) {
74
- // get the desired log level
75
- const inputVal = val.split('=')[1];
76
-
77
- // validate the log level is supported, if so set it
78
- if (Object.hasOwnProperty.call(myCustomLevels.levels, inputVal)) {
79
- logLevel = inputVal;
80
- }
81
- }
82
- });
83
-
84
- // need to set global logging
85
- global.log = winston.createLogger({
86
- level: logLevel,
87
- levels: myCustomLevels.levels,
88
- transports: [
89
- new winston.transports.Console()
90
- ]
91
- });
92
-
93
56
  /**
94
57
  * Runs the error asserts for the test
95
58
  */
@@ -17,13 +17,13 @@ const { expect } = require('chai');
17
17
  const { use } = require('chai');
18
18
  const td = require('testdouble');
19
19
  const Ajv = require('ajv');
20
+ const log = require('../../utils/logger');
20
21
 
21
22
  const ajv = new Ajv({ allErrors: true, unknownFormats: 'ignore' });
22
23
 
23
24
  const anything = td.matchers.anything();
24
25
 
25
26
  // stub and attemptTimeout are used throughout the code so set them here
26
- let logLevel = 'none';
27
27
  const stub = true;
28
28
  const isRapidFail = false;
29
29
  const attemptTimeout = 120000;
@@ -58,43 +58,6 @@ global.pronghornProps = {
58
58
 
59
59
  global.$HOME = `${__dirname}/../..`;
60
60
 
61
- // set the log levels that Pronghorn uses, spam and trace are not defaulted in so without
62
- // this you may error on log.trace calls.
63
- const myCustomLevels = {
64
- levels: {
65
- spam: 6,
66
- trace: 5,
67
- debug: 4,
68
- info: 3,
69
- warn: 2,
70
- error: 1,
71
- none: 0
72
- }
73
- };
74
-
75
- // need to see if there is a log level passed in
76
- process.argv.forEach((val) => {
77
- // is there a log level defined to be passed in?
78
- if (val.indexOf('--LOG') === 0) {
79
- // get the desired log level
80
- const inputVal = val.split('=')[1];
81
-
82
- // validate the log level is supported, if so set it
83
- if (Object.hasOwnProperty.call(myCustomLevels.levels, inputVal)) {
84
- logLevel = inputVal;
85
- }
86
- }
87
- });
88
-
89
- // need to set global logging
90
- global.log = winston.createLogger({
91
- level: logLevel,
92
- levels: myCustomLevels.levels,
93
- transports: [
94
- new winston.transports.Console()
95
- ]
96
- });
97
-
98
61
  /**
99
62
  * Runs the error asserts for the test
100
63
  */
@@ -0,0 +1,44 @@
1
+ const customLevels = {
2
+ spam: 6,
3
+ trace: 5,
4
+ debug: 4,
5
+ info: 3,
6
+ warn: 2,
7
+ error: 1,
8
+ none: 0
9
+ };
10
+
11
+ function parseArgs(argv = process.argv) {
12
+ let properties = null;
13
+ let logLevel = 'none';
14
+ let maxCalls = 5;
15
+ let host = null;
16
+
17
+ argv.forEach((val) => {
18
+ if (val.startsWith('--PROPS=')) {
19
+ // get the properties
20
+ const inputVal = val.split('=')[1];
21
+ properties = JSON.parse(inputVal);
22
+ } else if (val.startsWith('--LOG=')) {
23
+ // get the desired log level
24
+ const level = val.split('=')[1];
25
+ // validate the log level is supported, if so set it
26
+ if (Object.hasOwnProperty.call(customLevels, level)) {
27
+ logLevel = level;
28
+ }
29
+ } else if (val.startsWith('--MAXCALLS=')) {
30
+ const override = parseInt(val.split('=')[1], 10);
31
+ if (!Number.isNaN(override) && override > 0) {
32
+ maxCalls = override;
33
+ }
34
+ } else if (val.startsWith('--HOST=')) {
35
+ [, host] = val.split('=');
36
+ }
37
+ });
38
+
39
+ return {
40
+ properties, logLevel, maxCalls, host
41
+ };
42
+ }
43
+
44
+ module.exports = { parseArgs };
@@ -0,0 +1,26 @@
1
+ // utils/logger.js
2
+ const winston = require('winston');
3
+ const { parseArgs } = require('./argParser');
4
+
5
+ const customLevels = {
6
+ spam: 6,
7
+ trace: 5,
8
+ debug: 4,
9
+ info: 3,
10
+ warn: 2,
11
+ error: 1,
12
+ none: 0
13
+ };
14
+
15
+ // Only set global logger if it doesn't already exist (i.e., not provided by app)
16
+ if (!global.log) {
17
+ const { logLevel = 'info' } = parseArgs();
18
+
19
+ global.log = winston.createLogger({
20
+ level: logLevel,
21
+ levels: customLevels,
22
+ transports: [new winston.transports.Console()]
23
+ });
24
+ }
25
+
26
+ module.exports = global.log;