@itentialopensource/adapter-kafkav2 0.23.4 → 0.24.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.eslintrc.js +1 -0
- package/.topics-Kafkav2.json +1 -0
- package/AUTH.md +3 -3
- package/ENHANCE.md +3 -3
- package/README.md +21 -12
- package/SUMMARY.md +2 -2
- package/SYSTEMINFO.md +1 -1
- package/TAB1.md +2 -2
- package/TAB2.md +19 -4
- package/adapter.js +59 -1
- package/package.json +21 -23
- package/propertiesSchema.json +27 -5
- package/sampleProperties.json +5 -0
- package/test/integration/adapterTestIntegration.js +1 -38
- package/test/unit/adapterBaseTestUnit.js +1 -38
- package/test/unit/adapterTestUnit.js +1 -38
- package/utils/argParser.js +44 -0
- package/utils/logger.js +26 -0
package/.eslintrc.js
CHANGED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
[]
|
package/AUTH.md
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
## Authenticating Kafka v2 Adapter
|
|
2
2
|
|
|
3
|
-
This document will go through the steps for authenticating the Kafka v2 adapter with. Properly configuring the properties for an adapter in
|
|
3
|
+
This document will go through the steps for authenticating the Kafka v2 adapter with. Properly configuring the properties for an adapter in Itential Platform is critical for getting the adapter online. You can read more about adapter authentication <a href="https://docs.itential.com/opensource/docs/authentication" target="_blank">HERE</a>.
|
|
4
4
|
|
|
5
5
|
### Library Authentication
|
|
6
6
|
The Kafka v2 adapter supports SASL Authentication for Kafka v2 server. If you change authentication methods, you should change this section accordingly and merge it back into the adapter repository.
|
|
7
7
|
|
|
8
8
|
STEPS
|
|
9
9
|
1. Ensure you have access to a Kafka v2 server and that it is running
|
|
10
|
-
2. Follow the steps in the README.md to import the adapter into
|
|
10
|
+
2. Follow the steps in the README.md to import the adapter into Itential Platform if you have not already done so
|
|
11
11
|
3. Use the properties below for the ```properties.client.sasl``` field
|
|
12
12
|
4. Kafka v2 adapter supports both PLAIN and SCRAM-SHA authentication mechanism
|
|
13
13
|
|
|
@@ -33,6 +33,6 @@ For more details on sasl authentication follow the README.md
|
|
|
33
33
|
|
|
34
34
|
### Troubleshooting
|
|
35
35
|
- Make sure you copied over the correct username and password.
|
|
36
|
-
- Turn on debug level logs for the adapter in
|
|
36
|
+
- Turn on debug level logs for the adapter in Itential Platform Admin Essentials.
|
|
37
37
|
- Investigate the logs
|
|
38
38
|
- Credentials should be ** masked ** by the adapter so make sure you verify the username and password - including that there are erroneous spaces at the front or end.
|
package/ENHANCE.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
### Adding a Second Instance of an Adapter
|
|
4
4
|
|
|
5
|
-
You can add a second instance of this adapter without adding new code on the file system. To do this go into the
|
|
5
|
+
You can add a second instance of this adapter without adding new code on the file system. To do this go into the Itential Platform Admin Essentials and add a new service config for this adapter. The two instances of the adapter should have unique ids. In addition, they should point to different instances (unique host and port) of the other system.
|
|
6
6
|
|
|
7
7
|
### Adding Adapter Calls
|
|
8
8
|
|
|
@@ -18,7 +18,7 @@ npm run adapter:update
|
|
|
18
18
|
|
|
19
19
|
If you do not have a Swagger or OpenAPI document, you can use a Postman Collection and convert that to an OpenAPI document using APIMatic and then follow the first process.
|
|
20
20
|
|
|
21
|
-
If you want to manually update the adapter that can also be done the key thing is to make sure you update all of the right files. Within the entities directory you will find 1 or more entities. You can create a new entity or add to an existing entity. Each entity has an action.json file, any new call will need to be put in the action.json file. It will also need to be added to the enum for the ph_request_type in the appropriate schema files. Once this configuration is complete you will need to add the call to the adapter.js file and, in order to make it available as a workflow task in
|
|
21
|
+
If you want to manually update the adapter that can also be done the key thing is to make sure you update all of the right files. Within the entities directory you will find 1 or more entities. You can create a new entity or add to an existing entity. Each entity has an action.json file, any new call will need to be put in the action.json file. It will also need to be added to the enum for the ph_request_type in the appropriate schema files. Once this configuration is complete you will need to add the call to the adapter.js file and, in order to make it available as a workflow task in Itential Platform, it should also be added to the pronghorn.json file. You can optionally add it to the unit and integration test files. There is more information on how to work on each of these files in the <a href="https://docs.itential.com/opensource/docs/adapters" target="_blank">Adapter Technical Resources</a> on our Documentation Site.
|
|
22
22
|
|
|
23
23
|
```text
|
|
24
24
|
Files to update
|
|
@@ -32,7 +32,7 @@ Files to update
|
|
|
32
32
|
|
|
33
33
|
### Adding Adapter Properties
|
|
34
34
|
|
|
35
|
-
While changing adapter properties is done in the service instance configuration section of
|
|
35
|
+
While changing adapter properties is done in the service instance configuration section of Itential Platform, adding properties has to be done in the adapter. To add a property you should edit the propertiesSchema.json with the proper information for the property. In addition, you should modify the sampleProperties to have the new property in it.
|
|
36
36
|
|
|
37
37
|
```text
|
|
38
38
|
Files to update
|
package/README.md
CHANGED
|
@@ -1,16 +1,16 @@
|
|
|
1
1
|
# Kafkav2 Adapter
|
|
2
2
|
|
|
3
|
-
This adapter is used to integrate the Itential
|
|
3
|
+
This adapter is used to integrate the Itential Platform with the Kafka System. The documenation for kafkajs is available at [https://kafka.js.org/docs/getting-started]. The adapter utilizes the kafkjs library to provide the integrations that are deemed pertinent to Itential Platform. This Readme file is intended to provide information on this adapter.
|
|
4
4
|
|
|
5
5
|
>**Note**: It is possible that some integrations will be supported through the Kafka adapter while other integrations will not.
|
|
6
6
|
|
|
7
7
|
Itential provides information on all of its product adapters in the Customer Knowledge Base. Information in the [Customer Knowledge Base](https://itential.atlassian.net/servicedesk/customer/portals) is consistently maintained and goes through documentation reviews. As a result, it should be the first place to go for information.
|
|
8
8
|
|
|
9
|
-
For custom built adapters, it is a starting point to understand what you have built, provide the information for you to be able to update the adapter, and assist you with deploying the adapter into
|
|
9
|
+
For custom built adapters, it is a starting point to understand what you have built, provide the information for you to be able to update the adapter, and assist you with deploying the adapter into Itential Platform.
|
|
10
10
|
|
|
11
11
|
## Versioning
|
|
12
12
|
|
|
13
|
-
Itential Product adapters utilize SemVer for versioning. The current version of the adapter can be found in the `package.json` file or viewed in the
|
|
13
|
+
Itential Product adapters utilize SemVer for versioning. The current version of the adapter can be found in the `package.json` file or viewed in the Itential Platform GUI on the System page. For Open Source Adapters, the versions available can be found in the [Itential OpenSource Repository](https://www.npmjs.com/search?q=itentialopensource%2Fadapter).
|
|
14
14
|
|
|
15
15
|
## Release History
|
|
16
16
|
|
|
@@ -77,7 +77,7 @@ npm install
|
|
|
77
77
|
|
|
78
78
|
## Installing an Itential Product Adapter
|
|
79
79
|
|
|
80
|
-
1. Set up the name space location in your
|
|
80
|
+
1. Set up the name space location in your Itential Platform node_modules.
|
|
81
81
|
|
|
82
82
|
```json
|
|
83
83
|
cd /opt/pronghorn/current/node_modules
|
|
@@ -85,7 +85,7 @@ if the @itentialopensource directory does not exist, create it:
|
|
|
85
85
|
mkdir @itentialopensource
|
|
86
86
|
```
|
|
87
87
|
|
|
88
|
-
1. Clone the adapter into your
|
|
88
|
+
1. Clone the adapter into your Itential Platform environment.
|
|
89
89
|
|
|
90
90
|
```json
|
|
91
91
|
cd \@itentialopensource
|
|
@@ -102,7 +102,7 @@ npm install
|
|
|
102
102
|
1. Add the adapter properties for Kafka (created from Adapter Builder) to the `properties.json` file for your Itential build. You will need to change the credentials and possibly the host information below.
|
|
103
103
|
[Kafka sample properties](sampleProperties.json)
|
|
104
104
|
|
|
105
|
-
1. Restart
|
|
105
|
+
1. Restart Itential Platform
|
|
106
106
|
|
|
107
107
|
```json
|
|
108
108
|
systemctl restart pronghorn
|
|
@@ -110,7 +110,7 @@ systemctl restart pronghorn
|
|
|
110
110
|
|
|
111
111
|
## Adapter Properties and Descriptions
|
|
112
112
|
|
|
113
|
-
This section defines **all** the properties that are available for the adapter, including detailed information on what each property is for. If you are not using certain capabilities with this adapter, you do not need to define all of the properties. An example of how the properties for this adapter can be used with tests or
|
|
113
|
+
This section defines **all** the properties that are available for the adapter, including detailed information on what each property is for. If you are not using certain capabilities with this adapter, you do not need to define all of the properties. An example of how the properties for this adapter can be used with tests or Itential Platform are provided in the **Installation** section.
|
|
114
114
|
|
|
115
115
|
```json
|
|
116
116
|
{
|
|
@@ -157,6 +157,14 @@ Sample SSL and SASL properties that go under client props. In the below example
|
|
|
157
157
|
}
|
|
158
158
|
```
|
|
159
159
|
|
|
160
|
+
Sample socket properties
|
|
161
|
+
```json
|
|
162
|
+
"socketOptions": {
|
|
163
|
+
"keepAlive": true,
|
|
164
|
+
"keepAliveInterval": 45000,
|
|
165
|
+
"socketTimeout": 20000
|
|
166
|
+
}
|
|
167
|
+
```
|
|
160
168
|
|
|
161
169
|
### Topic Properties
|
|
162
170
|
|
|
@@ -311,6 +319,7 @@ The following properties are used to define the Kafka Client. These properties a
|
|
|
311
319
|
| retry -> retries | Max number of retries per call. default: 5 |
|
|
312
320
|
| ssl | Object, options to be passed to the tls broker sockets, ex. { rejectUnauthorized: false }.|
|
|
313
321
|
| sasl | Object, SASL authentication configuration (Currently, supports PLAIN, SCRAM-SHA-256, SCRAM-SHA-512), ex. { mechanism: 'plain', username: 'foo', password: 'bar' }.|
|
|
322
|
+
| socketOptions | Object, Socket Options: Configuration for socket behavior, including keep-alive, interval, and timeout settings.|
|
|
314
323
|
|
|
315
324
|
For all client config options see [Client Config](https://kafka.js.org/docs/configuration)
|
|
316
325
|
|
|
@@ -337,13 +346,13 @@ Refer to kafkajs docs for all consumer options [Consumer](https://kafka.js.org/d
|
|
|
337
346
|
|
|
338
347
|
## Parsing Properties
|
|
339
348
|
|
|
340
|
-
The `parseMessage` property allows the user to define how they want the Kafka message to be published to
|
|
349
|
+
The `parseMessage` property allows the user to define how they want the Kafka message to be published to Itential Platform's event system. If `parseMessage` is set to true or omitted, the value of the Kafka message will be parsed as either an object or string and wrapped in an outer object. The wrapper object's key can be defined with the property `wrapMessage`, or the default value `payload` can be used if omitted. If `parseMessage` is set to false, the entire kafka payload, including metadata, would be returned and the message itself would need to be transformed at a later point.
|
|
341
350
|
|
|
342
351
|
## Turning off stream if WorkflowEngine or OperationsManager is down
|
|
343
352
|
|
|
344
|
-
`check_iap_apps` (boolean) and `iap_apps_check_interval` (integer) are used when the user wants to turn off stream if WorkflowEngine or OperationsManager is down. If `check_iap_apps` is set to true, by default, the adapter will check the status of both WorkflowEngine and OperationsManager at a defined interval. If any
|
|
353
|
+
`check_iap_apps` (boolean) and `iap_apps_check_interval` (integer) are used when the user wants to turn off stream if WorkflowEngine or OperationsManager is down. If `check_iap_apps` is set to true, by default, the adapter will check the status of both WorkflowEngine and OperationsManager at a defined interval. If any Itential Platform apps are down, the consumer will be paused until the apps are active again. If the user does not want to healthcheck OperationsManager, set `check_ops_manager_status` to false. Similarly, if the user does not want to check WorkflowEngine status, set `check_wfe_status` to false.
|
|
345
354
|
|
|
346
|
-
`iap_apps_check_interval` (default 30000ms - 30s) allows the user to set the frequency in which to run
|
|
355
|
+
`iap_apps_check_interval` (default 30000ms - 30s) allows the user to set the frequency in which to run Itential Platform app healtcheck.
|
|
347
356
|
|
|
348
357
|
## Send message sample payload
|
|
349
358
|
|
|
@@ -360,9 +369,9 @@ Note that this is an array, so it can have multiple topics.
|
|
|
360
369
|
```
|
|
361
370
|
For all options of messages array please follow [kafkajs documentation](https://kafka.js.org/docs/producing#producing-messages)
|
|
362
371
|
|
|
363
|
-
## Triggering jobs in
|
|
372
|
+
## Triggering jobs in Itential Platform from a kafka message
|
|
364
373
|
|
|
365
|
-
Once you have the adapter configured and online in an
|
|
374
|
+
Once you have the adapter configured and online in an Itential Platform instance, you can create triggers in Operations Manager to kick off jobs when a kafka message is consumed. The adapter listens for messages on the subscribed topics and publishes to an event. This event is picked up by the Operations Manager. If you have created triggers for this specific topic, a job will get triggered. For a step-by-step example follow the next sub-section.
|
|
366
375
|
|
|
367
376
|
### Listen to a topic called test-topic and trigger a workflow called test
|
|
368
377
|
|
package/SUMMARY.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
## Overview
|
|
2
|
-
This adapter is used to integrate the Itential
|
|
2
|
+
This adapter is used to integrate the Itential Platform with the Kafka System. The adapter utilizes the KafkaJS client to integrate with Apache Kafka. The ReadMe file is intended to provide information on this adapter it is generated from various other Markdown files.
|
|
3
3
|
|
|
4
4
|
>**Note**: It is possible that some integrations will be supported through the KafkaV2 adapter while other integrations will not.
|
|
5
5
|
|
|
6
6
|
Itential provides information on all of its product adapters in the Customer Knowledge Base. Information in the <a href="https://itential.atlassian.net/servicedesk/customer/portals" target="_blank">Customer Knowledge Base</a> is consistently maintained and goes through documentation reviews. As a result, it should be the first place to go for information.
|
|
7
7
|
|
|
8
|
-
For opensourced and custom built adapters, the ReadMe is a starting point to understand what you have built, provide the information for you to be able to update the adapter, and assist you with deploying the adapter into
|
|
8
|
+
For opensourced and custom built adapters, the ReadMe is a starting point to understand what you have built, provide the information for you to be able to update the adapter, and assist you with deploying the adapter into Itential Platform.
|
package/SYSTEMINFO.md
CHANGED
|
@@ -16,7 +16,7 @@ Apache Kafka is an open-source distributed event streaming platform used by thou
|
|
|
16
16
|
"Store streams of data safely in a distributed, durable, fault-tolerant cluster."
|
|
17
17
|
|
|
18
18
|
## Why Integrate
|
|
19
|
-
The Kafka v2 adapter from Itential is used to integrate the Itential
|
|
19
|
+
The Kafka v2 adapter from Itential is used to integrate the Itential Platform with Kafka. With this adapter you have the ability to perform operations with Kafka on items such as:
|
|
20
20
|
|
|
21
21
|
- Produce/Publish messages onto message topics for others to consume
|
|
22
22
|
- Listen, receive and process messages placed on topics by other systems
|
package/TAB1.md
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# Overview
|
|
2
|
-
This adapter is used to integrate the Itential
|
|
2
|
+
This adapter is used to integrate the Itential Platform with the Kafka System. The adapter utilizes the KafkaJS client to integrate with Apache Kafka. The ReadMe file is intended to provide information on this adapter it is generated from various other Markdown files.
|
|
3
3
|
|
|
4
4
|
|
|
5
5
|
## Details
|
|
6
|
-
The Kafka v2 adapter from Itential is used to integrate the Itential
|
|
6
|
+
The Kafka v2 adapter from Itential is used to integrate the Itential Platform with Kafka. With this adapter you have the ability to perform operations with Kafka on items such as:
|
|
7
7
|
|
|
8
8
|
- Produce/Publish messages onto message topics for others to consume
|
|
9
9
|
- Listen, receive and process messages placed on topics by other systems
|
package/TAB2.md
CHANGED
|
@@ -11,14 +11,14 @@
|
|
|
11
11
|
## Specific Adapter Information
|
|
12
12
|
### Authentication
|
|
13
13
|
|
|
14
|
-
This document will go through the steps for authenticating the Kafka v2 adapter with. Properly configuring the properties for an adapter in
|
|
14
|
+
This document will go through the steps for authenticating the Kafka v2 adapter with. Properly configuring the properties for an adapter in Itential Platform is critical for getting the adapter online. You can read more about adapter authentication <a href="https://docs.itential.com/opensource/docs/authentication" target="_blank">HERE</a>.
|
|
15
15
|
|
|
16
16
|
#### Library Authentication
|
|
17
17
|
The Kafka v2 adapter supports SASL Authentication for Kafka v2 server. If you change authentication methods, you should change this section accordingly and merge it back into the adapter repository.
|
|
18
18
|
|
|
19
19
|
STEPS
|
|
20
20
|
1. Ensure you have access to a Kafka v2 server and that it is running
|
|
21
|
-
2. Follow the steps in the README.md to import the adapter into
|
|
21
|
+
2. Follow the steps in the README.md to import the adapter into Itential Platform if you have not already done so
|
|
22
22
|
3. Use the properties below for the ```properties.client.sasl``` field
|
|
23
23
|
4. Kafka v2 adapter supports both PLAIN and SCRAM-SHA authentication mechanism
|
|
24
24
|
|
|
@@ -44,12 +44,12 @@ For more details on sasl authentication follow the README.md
|
|
|
44
44
|
|
|
45
45
|
#### Troubleshooting
|
|
46
46
|
- Make sure you copied over the correct username and password.
|
|
47
|
-
- Turn on debug level logs for the adapter in
|
|
47
|
+
- Turn on debug level logs for the adapter in Itential Platform Admin Essentials.
|
|
48
48
|
- Investigate the logs
|
|
49
49
|
- Credentials should be ** masked ** by the adapter so make sure you verify the username and password - including that there are erroneous spaces at the front or end.
|
|
50
50
|
### Sample Properties
|
|
51
51
|
|
|
52
|
-
Sample Properties can be used to help you configure the adapter in the Itential
|
|
52
|
+
Sample Properties can be used to help you configure the adapter in the Itential Platform. You will need to update connectivity information such as the host, port, protocol and credentials.
|
|
53
53
|
|
|
54
54
|
```json
|
|
55
55
|
"properties": {
|
|
@@ -69,6 +69,16 @@ Sample Properties can be used to help you configure the adapter in the Itential
|
|
|
69
69
|
],
|
|
70
70
|
"clientId": "my-app",
|
|
71
71
|
"logLevel": "INFO",
|
|
72
|
+
"connectionTimeout": 1000,
|
|
73
|
+
"requestTimeout": 30000,
|
|
74
|
+
"enforceRequestTimeout": true,
|
|
75
|
+
"retry": {
|
|
76
|
+
"maxRetryTime": 30000,
|
|
77
|
+
"initialRetryTime": 300,
|
|
78
|
+
"factor": 0.2,
|
|
79
|
+
"multiplier": 2,
|
|
80
|
+
"retries": 5
|
|
81
|
+
},
|
|
72
82
|
"ssl": {
|
|
73
83
|
"enableTrace": true,
|
|
74
84
|
"ca": "/path/to/crt.pem",
|
|
@@ -78,6 +88,11 @@ Sample Properties can be used to help you configure the adapter in the Itential
|
|
|
78
88
|
"username": "my-user",
|
|
79
89
|
"password": "my-password",
|
|
80
90
|
"mechanism": "scram-sha-512"
|
|
91
|
+
},
|
|
92
|
+
"socketOptions": {
|
|
93
|
+
"keepAlive": true,
|
|
94
|
+
"keepAliveInterval": 45000,
|
|
95
|
+
"socketTimeout": 20000
|
|
81
96
|
}
|
|
82
97
|
},
|
|
83
98
|
"producer": {
|
package/adapter.js
CHANGED
|
@@ -17,11 +17,16 @@
|
|
|
17
17
|
const fs = require('fs-extra');
|
|
18
18
|
const path = require('path');
|
|
19
19
|
const util = require('util');
|
|
20
|
+
const net = require('net');
|
|
21
|
+
const tls = require('tls');
|
|
20
22
|
/* Fetch in the other needed components for the this Adaptor */
|
|
21
23
|
// const EventEmitterCl = require('events').EventEmitter;
|
|
22
24
|
const AdapterBaseCl = require(path.join(__dirname, 'adapterBase.js'));
|
|
23
25
|
const DBUtil = require(path.join(__dirname, 'utils', 'dbUtil.js'));
|
|
24
26
|
|
|
27
|
+
const DEFAULT_KEEP_ALIVE_INTERVAL = 60000; // 60s
|
|
28
|
+
const DEFAULT_SOCKET_TIMEOUT = 30000; // 30s
|
|
29
|
+
|
|
25
30
|
const axios = require('axios');
|
|
26
31
|
|
|
27
32
|
let needRestart = false;
|
|
@@ -216,6 +221,51 @@ function formatErrorObject(origin, type, variables, sysCode, sysRes, stack) {
|
|
|
216
221
|
return errorObject;
|
|
217
222
|
}
|
|
218
223
|
|
|
224
|
+
/**
|
|
225
|
+
* @summary Creates a socket factory for KafkaJS to handle custom socket connections.
|
|
226
|
+
*
|
|
227
|
+
* @function createSocketFactory
|
|
228
|
+
* @param {Object} socketOptions - Configuration options for the socket connection.
|
|
229
|
+
* @param {Boolean} useSSL - Determines whether SSL should be used for the connection.
|
|
230
|
+
*
|
|
231
|
+
* @return {Function} - A socket factory function that establishes a connection to Kafka brokers.
|
|
232
|
+
*/
|
|
233
|
+
function createSocketFactory(socketOptions, useSSL) {
|
|
234
|
+
return (options) => {
|
|
235
|
+
console.log('KafkaJS Socket Options:', options); // See what KafkaJS is actually passing
|
|
236
|
+
|
|
237
|
+
// Destructure properties from the options object
|
|
238
|
+
const { host, port, ssl } = options;
|
|
239
|
+
const brokerAddress = `${host}:${port}`;
|
|
240
|
+
console.log(
|
|
241
|
+
`Connecting to Kafka broker: ${brokerAddress} (${host}:${port}) ${useSSL ? 'with SSL' : 'without SSL'
|
|
242
|
+
}`
|
|
243
|
+
);
|
|
244
|
+
|
|
245
|
+
const socket = useSSL
|
|
246
|
+
? tls.connect({ host, port, rejectUnauthorized: false }) // Use TLS for SSL
|
|
247
|
+
: net.createConnection({ host, port }); // Use Net for Non-SSL
|
|
248
|
+
|
|
249
|
+
const keepAliveEnabled = socketOptions && socketOptions.keepAlive !== undefined
|
|
250
|
+
? socketOptions.keepAlive
|
|
251
|
+
: true;
|
|
252
|
+
const keepAliveInterval = socketOptions && socketOptions.keepAliveInterval !== undefined
|
|
253
|
+
? socketOptions.keepAliveInterval
|
|
254
|
+
: DEFAULT_KEEP_ALIVE_INTERVAL; // 60s default
|
|
255
|
+
const socketTimeout = socketOptions && socketOptions.socketTimeout !== undefined
|
|
256
|
+
? socketOptions.socketTimeout
|
|
257
|
+
: DEFAULT_SOCKET_TIMEOUT; // 30s default
|
|
258
|
+
|
|
259
|
+
socket.setKeepAlive(keepAliveEnabled, keepAliveInterval);
|
|
260
|
+
socket.setTimeout(socketTimeout);
|
|
261
|
+
|
|
262
|
+
socket.on('error', (err) => console.error(`Socket error with ${brokerAddress}:`, err));
|
|
263
|
+
socket.on('timeout', () => console.warn(`Socket timeout with ${brokerAddress}`));
|
|
264
|
+
|
|
265
|
+
return socket;
|
|
266
|
+
};
|
|
267
|
+
}
|
|
268
|
+
|
|
219
269
|
/**
|
|
220
270
|
* This is the adapter/interface into Kafka
|
|
221
271
|
*/
|
|
@@ -645,6 +695,14 @@ class Kafkav2 extends AdapterBaseCl {
|
|
|
645
695
|
log.info('EMITTED ONLINE ON STUB MODE');
|
|
646
696
|
return;
|
|
647
697
|
}
|
|
698
|
+
const combinedProps = this.props.client || {};
|
|
699
|
+
// Extract socket options with default values
|
|
700
|
+
const socketOptions = this.props.client.socketOptions || null;
|
|
701
|
+
const useSSL = this.props.client.ssl || false; // Determine SSL usage
|
|
702
|
+
|
|
703
|
+
if (socketOptions) {
|
|
704
|
+
combinedProps.socketFactory = createSocketFactory(socketOptions, useSSL);
|
|
705
|
+
}
|
|
648
706
|
|
|
649
707
|
if (!needRestart) {
|
|
650
708
|
try {
|
|
@@ -653,7 +711,7 @@ class Kafkav2 extends AdapterBaseCl {
|
|
|
653
711
|
this.props.client.logLevel = logLevelEnum;
|
|
654
712
|
}
|
|
655
713
|
// Create a kafka client
|
|
656
|
-
|
|
714
|
+
|
|
657
715
|
if (this.props.client.ssl && this.props.client.ssl.ca) {
|
|
658
716
|
combinedProps.ssl.ca = [fs.readFileSync(this.props.client.ssl.ca, 'utf-8')];
|
|
659
717
|
}
|
package/package.json
CHANGED
|
@@ -1,12 +1,11 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@itentialopensource/adapter-kafkav2",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.24.0",
|
|
4
4
|
"description": "Itential adapter to connect to kafka",
|
|
5
5
|
"main": "adapter.js",
|
|
6
|
-
"wizardVersion": "2.
|
|
7
|
-
"engineVersion": "1.
|
|
6
|
+
"wizardVersion": "2.44.7",
|
|
7
|
+
"engineVersion": "1.69.14",
|
|
8
8
|
"scripts": {
|
|
9
|
-
"artifactize": "npm i && node utils/packModificationScript.js",
|
|
10
9
|
"preinstall": "node utils/setup.js",
|
|
11
10
|
"deinstall": "node utils/removeHooks.js",
|
|
12
11
|
"lint": "node --max_old_space_size=4096 ./node_modules/eslint/bin/eslint.js . --ext .json --ext .js",
|
|
@@ -20,10 +19,9 @@
|
|
|
20
19
|
},
|
|
21
20
|
"keywords": [
|
|
22
21
|
"Itential",
|
|
23
|
-
"
|
|
22
|
+
"Itential Platform",
|
|
24
23
|
"Automation",
|
|
25
24
|
"Integration",
|
|
26
|
-
"App-Artifacts",
|
|
27
25
|
"Adapter",
|
|
28
26
|
"Notification",
|
|
29
27
|
"Messaging",
|
|
@@ -43,26 +41,26 @@
|
|
|
43
41
|
"author": "Itential",
|
|
44
42
|
"homepage": "https://gitlab.com/itentialopensource/adapters/adapter-kafkav2",
|
|
45
43
|
"dependencies": {
|
|
46
|
-
"ajv": "
|
|
47
|
-
"avro-schema-registry": "
|
|
48
|
-
"avsc": "
|
|
49
|
-
"axios": "
|
|
50
|
-
"fs-extra": "
|
|
51
|
-
"json-query": "
|
|
44
|
+
"ajv": "8.17.1",
|
|
45
|
+
"avro-schema-registry": "2.1.0",
|
|
46
|
+
"avsc": "5.4.21",
|
|
47
|
+
"axios": "1.9.0",
|
|
48
|
+
"fs-extra": "11.3.0",
|
|
49
|
+
"json-query": "2.2.2",
|
|
52
50
|
"kafkajs": "2.2.3",
|
|
53
|
-
"mongodb": "
|
|
54
|
-
"readline-sync": "
|
|
55
|
-
"uuid": "
|
|
51
|
+
"mongodb": "4.17.2",
|
|
52
|
+
"readline-sync": "1.4.10",
|
|
53
|
+
"uuid": "3.0.1",
|
|
54
|
+
"mocha": "10.8.2",
|
|
55
|
+
"winston": "3.17.0"
|
|
56
56
|
},
|
|
57
57
|
"devDependencies": {
|
|
58
|
-
"chai": "
|
|
59
|
-
"eslint": "
|
|
60
|
-
"eslint-config-airbnb-base": "
|
|
61
|
-
"eslint-plugin-import": "
|
|
62
|
-
"eslint-plugin-json": "
|
|
63
|
-
"
|
|
64
|
-
"testdouble": "^3.18.0",
|
|
65
|
-
"winston": "^3.13.1"
|
|
58
|
+
"chai": "4.5.0",
|
|
59
|
+
"eslint": "8.57.0",
|
|
60
|
+
"eslint-config-airbnb-base": "15.0.0",
|
|
61
|
+
"eslint-plugin-import": "2.31.0",
|
|
62
|
+
"eslint-plugin-json": "3.1.0",
|
|
63
|
+
"testdouble": "3.18.0"
|
|
66
64
|
},
|
|
67
65
|
"private": false
|
|
68
66
|
}
|
package/propertiesSchema.json
CHANGED
|
@@ -21,16 +21,18 @@
|
|
|
21
21
|
]
|
|
22
22
|
}
|
|
23
23
|
},
|
|
24
|
-
"parseMessage"
|
|
24
|
+
"parseMessage": {
|
|
25
25
|
"type": "boolean",
|
|
26
26
|
"description": "Whether or not to prase the message as JSON when consumed by IAP",
|
|
27
|
-
"default"
|
|
27
|
+
"default": true
|
|
28
28
|
},
|
|
29
|
-
"wrapMessage"
|
|
29
|
+
"wrapMessage": {
|
|
30
30
|
"type": "string",
|
|
31
31
|
"description": "A key to wrap the parsed JSON message",
|
|
32
|
-
"examples"
|
|
33
|
-
|
|
32
|
+
"examples": [
|
|
33
|
+
"message"
|
|
34
|
+
],
|
|
35
|
+
"default": "payload"
|
|
34
36
|
},
|
|
35
37
|
"interval_time": {
|
|
36
38
|
"type": "integer",
|
|
@@ -79,6 +81,26 @@
|
|
|
79
81
|
},
|
|
80
82
|
"sasl": {
|
|
81
83
|
"type": "object"
|
|
84
|
+
},
|
|
85
|
+
"socketOptions": {
|
|
86
|
+
"type": "object",
|
|
87
|
+
"properties": {
|
|
88
|
+
"keepAlive": {
|
|
89
|
+
"type": "boolean",
|
|
90
|
+
"description": "Whether to enable keep-alive on the socket."
|
|
91
|
+
},
|
|
92
|
+
"keepAliveInterval": {
|
|
93
|
+
"type": "integer",
|
|
94
|
+
"minimum": 0,
|
|
95
|
+
"description": "Interval (in milliseconds) for keep-alive packets."
|
|
96
|
+
},
|
|
97
|
+
"socketTimeout": {
|
|
98
|
+
"type": "integer",
|
|
99
|
+
"minimum": 0,
|
|
100
|
+
"description": "Timeout (in milliseconds) for socket inactivity before closing."
|
|
101
|
+
}
|
|
102
|
+
},
|
|
103
|
+
"additionalProperties": false
|
|
82
104
|
}
|
|
83
105
|
}
|
|
84
106
|
},
|
package/sampleProperties.json
CHANGED
|
@@ -14,11 +14,11 @@ const winston = require('winston');
|
|
|
14
14
|
const { expect } = require('chai');
|
|
15
15
|
const { use } = require('chai');
|
|
16
16
|
const td = require('testdouble');
|
|
17
|
+
const log = require('../../utils/logger');
|
|
17
18
|
|
|
18
19
|
const anything = td.matchers.anything();
|
|
19
20
|
|
|
20
21
|
// stub and attemptTimeout are used throughout the code so set them here
|
|
21
|
-
let logLevel = 'none';
|
|
22
22
|
const stub = true;
|
|
23
23
|
const isRapidFail = false;
|
|
24
24
|
const isSaveMockData = false;
|
|
@@ -58,43 +58,6 @@ global.pronghornProps = {
|
|
|
58
58
|
|
|
59
59
|
global.$HOME = `${__dirname}/../..`;
|
|
60
60
|
|
|
61
|
-
// set the log levels that Pronghorn uses, spam and trace are not defaulted in so without
|
|
62
|
-
// this you may error on log.trace calls.
|
|
63
|
-
const myCustomLevels = {
|
|
64
|
-
levels: {
|
|
65
|
-
spam: 6,
|
|
66
|
-
trace: 5,
|
|
67
|
-
debug: 4,
|
|
68
|
-
info: 3,
|
|
69
|
-
warn: 2,
|
|
70
|
-
error: 1,
|
|
71
|
-
none: 0
|
|
72
|
-
}
|
|
73
|
-
};
|
|
74
|
-
|
|
75
|
-
// need to see if there is a log level passed in
|
|
76
|
-
process.argv.forEach((val) => {
|
|
77
|
-
// is there a log level defined to be passed in?
|
|
78
|
-
if (val.indexOf('--LOG') === 0) {
|
|
79
|
-
// get the desired log level
|
|
80
|
-
const inputVal = val.split('=')[1];
|
|
81
|
-
|
|
82
|
-
// validate the log level is supported, if so set it
|
|
83
|
-
if (Object.hasOwnProperty.call(myCustomLevels.levels, inputVal)) {
|
|
84
|
-
logLevel = inputVal;
|
|
85
|
-
}
|
|
86
|
-
}
|
|
87
|
-
});
|
|
88
|
-
|
|
89
|
-
// need to set global logging
|
|
90
|
-
global.log = winston.createLogger({
|
|
91
|
-
level: logLevel,
|
|
92
|
-
levels: myCustomLevels.levels,
|
|
93
|
-
transports: [
|
|
94
|
-
new winston.transports.Console()
|
|
95
|
-
]
|
|
96
|
-
});
|
|
97
|
-
|
|
98
61
|
/**
|
|
99
62
|
* Runs the common asserts for test
|
|
100
63
|
*/
|
|
@@ -14,11 +14,11 @@ const execute = require('child_process').execSync;
|
|
|
14
14
|
const { expect } = require('chai');
|
|
15
15
|
const { use } = require('chai');
|
|
16
16
|
const td = require('testdouble');
|
|
17
|
+
const log = require('../../utils/logger');
|
|
17
18
|
|
|
18
19
|
const anything = td.matchers.anything();
|
|
19
20
|
|
|
20
21
|
// stub and attemptTimeout are used throughout the code so set them here
|
|
21
|
-
let logLevel = 'none';
|
|
22
22
|
const stub = true;
|
|
23
23
|
const isRapidFail = false;
|
|
24
24
|
const attemptTimeout = 120000;
|
|
@@ -53,43 +53,6 @@ global.pronghornProps = {
|
|
|
53
53
|
|
|
54
54
|
global.$HOME = `${__dirname}/../..`;
|
|
55
55
|
|
|
56
|
-
// set the log levels that Pronghorn uses, spam and trace are not defaulted in so without
|
|
57
|
-
// this you may error on log.trace calls.
|
|
58
|
-
const myCustomLevels = {
|
|
59
|
-
levels: {
|
|
60
|
-
spam: 6,
|
|
61
|
-
trace: 5,
|
|
62
|
-
debug: 4,
|
|
63
|
-
info: 3,
|
|
64
|
-
warn: 2,
|
|
65
|
-
error: 1,
|
|
66
|
-
none: 0
|
|
67
|
-
}
|
|
68
|
-
};
|
|
69
|
-
|
|
70
|
-
// need to see if there is a log level passed in
|
|
71
|
-
process.argv.forEach((val) => {
|
|
72
|
-
// is there a log level defined to be passed in?
|
|
73
|
-
if (val.indexOf('--LOG') === 0) {
|
|
74
|
-
// get the desired log level
|
|
75
|
-
const inputVal = val.split('=')[1];
|
|
76
|
-
|
|
77
|
-
// validate the log level is supported, if so set it
|
|
78
|
-
if (Object.hasOwnProperty.call(myCustomLevels.levels, inputVal)) {
|
|
79
|
-
logLevel = inputVal;
|
|
80
|
-
}
|
|
81
|
-
}
|
|
82
|
-
});
|
|
83
|
-
|
|
84
|
-
// need to set global logging
|
|
85
|
-
global.log = winston.createLogger({
|
|
86
|
-
level: logLevel,
|
|
87
|
-
levels: myCustomLevels.levels,
|
|
88
|
-
transports: [
|
|
89
|
-
new winston.transports.Console()
|
|
90
|
-
]
|
|
91
|
-
});
|
|
92
|
-
|
|
93
56
|
/**
|
|
94
57
|
* Runs the error asserts for the test
|
|
95
58
|
*/
|
|
@@ -17,13 +17,13 @@ const { expect } = require('chai');
|
|
|
17
17
|
const { use } = require('chai');
|
|
18
18
|
const td = require('testdouble');
|
|
19
19
|
const Ajv = require('ajv');
|
|
20
|
+
const log = require('../../utils/logger');
|
|
20
21
|
|
|
21
22
|
const ajv = new Ajv({ allErrors: true, unknownFormats: 'ignore' });
|
|
22
23
|
|
|
23
24
|
const anything = td.matchers.anything();
|
|
24
25
|
|
|
25
26
|
// stub and attemptTimeout are used throughout the code so set them here
|
|
26
|
-
let logLevel = 'none';
|
|
27
27
|
const stub = true;
|
|
28
28
|
const isRapidFail = false;
|
|
29
29
|
const attemptTimeout = 120000;
|
|
@@ -58,43 +58,6 @@ global.pronghornProps = {
|
|
|
58
58
|
|
|
59
59
|
global.$HOME = `${__dirname}/../..`;
|
|
60
60
|
|
|
61
|
-
// set the log levels that Pronghorn uses, spam and trace are not defaulted in so without
|
|
62
|
-
// this you may error on log.trace calls.
|
|
63
|
-
const myCustomLevels = {
|
|
64
|
-
levels: {
|
|
65
|
-
spam: 6,
|
|
66
|
-
trace: 5,
|
|
67
|
-
debug: 4,
|
|
68
|
-
info: 3,
|
|
69
|
-
warn: 2,
|
|
70
|
-
error: 1,
|
|
71
|
-
none: 0
|
|
72
|
-
}
|
|
73
|
-
};
|
|
74
|
-
|
|
75
|
-
// need to see if there is a log level passed in
|
|
76
|
-
process.argv.forEach((val) => {
|
|
77
|
-
// is there a log level defined to be passed in?
|
|
78
|
-
if (val.indexOf('--LOG') === 0) {
|
|
79
|
-
// get the desired log level
|
|
80
|
-
const inputVal = val.split('=')[1];
|
|
81
|
-
|
|
82
|
-
// validate the log level is supported, if so set it
|
|
83
|
-
if (Object.hasOwnProperty.call(myCustomLevels.levels, inputVal)) {
|
|
84
|
-
logLevel = inputVal;
|
|
85
|
-
}
|
|
86
|
-
}
|
|
87
|
-
});
|
|
88
|
-
|
|
89
|
-
// need to set global logging
|
|
90
|
-
global.log = winston.createLogger({
|
|
91
|
-
level: logLevel,
|
|
92
|
-
levels: myCustomLevels.levels,
|
|
93
|
-
transports: [
|
|
94
|
-
new winston.transports.Console()
|
|
95
|
-
]
|
|
96
|
-
});
|
|
97
|
-
|
|
98
61
|
/**
|
|
99
62
|
* Runs the error asserts for the test
|
|
100
63
|
*/
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
const customLevels = {
|
|
2
|
+
spam: 6,
|
|
3
|
+
trace: 5,
|
|
4
|
+
debug: 4,
|
|
5
|
+
info: 3,
|
|
6
|
+
warn: 2,
|
|
7
|
+
error: 1,
|
|
8
|
+
none: 0
|
|
9
|
+
};
|
|
10
|
+
|
|
11
|
+
function parseArgs(argv = process.argv) {
|
|
12
|
+
let properties = null;
|
|
13
|
+
let logLevel = 'none';
|
|
14
|
+
let maxCalls = 5;
|
|
15
|
+
let host = null;
|
|
16
|
+
|
|
17
|
+
argv.forEach((val) => {
|
|
18
|
+
if (val.startsWith('--PROPS=')) {
|
|
19
|
+
// get the properties
|
|
20
|
+
const inputVal = val.split('=')[1];
|
|
21
|
+
properties = JSON.parse(inputVal);
|
|
22
|
+
} else if (val.startsWith('--LOG=')) {
|
|
23
|
+
// get the desired log level
|
|
24
|
+
const level = val.split('=')[1];
|
|
25
|
+
// validate the log level is supported, if so set it
|
|
26
|
+
if (Object.hasOwnProperty.call(customLevels, level)) {
|
|
27
|
+
logLevel = level;
|
|
28
|
+
}
|
|
29
|
+
} else if (val.startsWith('--MAXCALLS=')) {
|
|
30
|
+
const override = parseInt(val.split('=')[1], 10);
|
|
31
|
+
if (!Number.isNaN(override) && override > 0) {
|
|
32
|
+
maxCalls = override;
|
|
33
|
+
}
|
|
34
|
+
} else if (val.startsWith('--HOST=')) {
|
|
35
|
+
[, host] = val.split('=');
|
|
36
|
+
}
|
|
37
|
+
});
|
|
38
|
+
|
|
39
|
+
return {
|
|
40
|
+
properties, logLevel, maxCalls, host
|
|
41
|
+
};
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
module.exports = { parseArgs };
|
package/utils/logger.js
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
// utils/logger.js
|
|
2
|
+
const winston = require('winston');
|
|
3
|
+
const { parseArgs } = require('./argParser');
|
|
4
|
+
|
|
5
|
+
const customLevels = {
|
|
6
|
+
spam: 6,
|
|
7
|
+
trace: 5,
|
|
8
|
+
debug: 4,
|
|
9
|
+
info: 3,
|
|
10
|
+
warn: 2,
|
|
11
|
+
error: 1,
|
|
12
|
+
none: 0
|
|
13
|
+
};
|
|
14
|
+
|
|
15
|
+
// Only set global logger if it doesn't already exist (i.e., not provided by app)
|
|
16
|
+
if (!global.log) {
|
|
17
|
+
const { logLevel = 'info' } = parseArgs();
|
|
18
|
+
|
|
19
|
+
global.log = winston.createLogger({
|
|
20
|
+
level: logLevel,
|
|
21
|
+
levels: customLevels,
|
|
22
|
+
transports: [new winston.transports.Console()]
|
|
23
|
+
});
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
module.exports = global.log;
|