@itentialopensource/adapter-kafkav2 0.22.2 → 0.22.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AUTH.md +22 -10
- package/CHANGELOG.md +8 -0
- package/SUMMARY.md +8 -0
- package/SYSTEMINFO.md +26 -0
- package/TAB1.md +11 -0
- package/TAB2.md +118 -0
- package/compliance-report.json +9 -0
- package/compliance-report.txt +5 -0
- package/metadata.json +31 -7
- package/package.json +7 -6
- package/refs?service=git-upload-pack +0 -0
- package/sampleProperties.json +44 -7
package/AUTH.md
CHANGED
|
@@ -1,26 +1,38 @@
|
|
|
1
|
-
## Authenticating Kafka Adapter
|
|
1
|
+
## Authenticating Kafka v2 Adapter
|
|
2
2
|
|
|
3
|
-
This document will go through the steps for authenticating the
|
|
3
|
+
This document will go through the steps for authenticating the Kafka v2 adapter with. Properly configuring the properties for an adapter in IAP is critical for getting the adapter online. You can read more about adapter authentication <a href="https://docs.itential.com/opensource/docs/authentication" target="_blank">HERE</a>.
|
|
4
4
|
|
|
5
5
|
### Library Authentication
|
|
6
|
-
The Kafka adapter supports SASL Authentication for Kafka server. If you change authentication methods, you should change this section accordingly and merge it back into the adapter repository.
|
|
6
|
+
The Kafka v2 adapter supports SASL Authentication for Kafka v2 server. If you change authentication methods, you should change this section accordingly and merge it back into the adapter repository.
|
|
7
7
|
|
|
8
8
|
STEPS
|
|
9
|
-
1. Ensure you have access to a Kafka server and that it is running
|
|
9
|
+
1. Ensure you have access to a Kafka v2 server and that it is running
|
|
10
10
|
2. Follow the steps in the README.md to import the adapter into IAP if you have not already done so
|
|
11
11
|
3. Use the properties below for the ```properties.client.sasl``` field
|
|
12
|
+
4. Kafka v2 adapter supports both PLAIN and SCRAM-SHA authentication mechanism
|
|
13
|
+
|
|
14
|
+
#### PLAIN
|
|
12
15
|
```json
|
|
13
16
|
"sasl": {
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
17
|
+
"mechanism": "PLAIN",
|
|
18
|
+
"username": "username",
|
|
19
|
+
"password": "password"
|
|
20
|
+
}
|
|
21
|
+
```
|
|
18
22
|
|
|
23
|
+
#### SCRAM-SHA
|
|
24
|
+
```json
|
|
25
|
+
"sasl": {
|
|
26
|
+
"mechanism": "scram-sha-512",
|
|
27
|
+
"username": "username",
|
|
28
|
+
"password": "password"
|
|
29
|
+
}
|
|
19
30
|
```
|
|
20
|
-
For more details on sasl authentication follow the
|
|
31
|
+
For more details on sasl authentication follow the README.md
|
|
21
32
|
4. Restart the adapter. If your properties were set correctly, the adapter should go online.
|
|
22
33
|
|
|
23
34
|
### Troubleshooting
|
|
24
35
|
- Make sure you copied over the correct username and password.
|
|
25
36
|
- Turn on debug level logs for the adapter in IAP Admin Essentials.
|
|
26
|
-
- Investigate the logs
|
|
37
|
+
- Investigate the logs
|
|
38
|
+
- Credentials should be ** masked ** by the adapter so make sure you verify the username and password - including that there are erroneous spaces at the front or end.
|
package/CHANGELOG.md
CHANGED
package/SUMMARY.md
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
## Overview
|
|
2
|
+
This adapter is used to integrate the Itential Automation Platform (IAP) with the Kafka System. The adapter utilizes the KafkaJS client to integrate with Apache Kafka. The ReadMe file is intended to provide information on this adapter it is generated from various other Markdown files.
|
|
3
|
+
|
|
4
|
+
>**Note**: It is possible that some integrations will be supported through the KafkaV2 adapter while other integrations will not.
|
|
5
|
+
|
|
6
|
+
Itential provides information on all of its product adapters in the Customer Knowledge Base. Information in the <a href="https://itential.atlassian.net/servicedesk/customer/portals" target="_blank">Customer Knowledge Base</a> is consistently maintained and goes through documentation reviews. As a result, it should be the first place to go for information.
|
|
7
|
+
|
|
8
|
+
For opensourced and custom built adapters, the ReadMe is a starting point to understand what you have built, provide the information for you to be able to update the adapter, and assist you with deploying the adapter into IAP.
|
package/SYSTEMINFO.md
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Kafka v2
|
|
2
|
+
|
|
3
|
+
Vendor: Apache Kafka
|
|
4
|
+
Homepage: https://kafka.apache.org/
|
|
5
|
+
|
|
6
|
+
Product: Kafka
|
|
7
|
+
Product Page: https://kafka.apache.org/
|
|
8
|
+
|
|
9
|
+
## Introduction
|
|
10
|
+
|
|
11
|
+
We classify Kafka v2 into the Notifications domain as Kafka v2 is used to send message between systems. There is an alternate Kafka adapter which uses an older Kafka library.
|
|
12
|
+
|
|
13
|
+
Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
|
|
14
|
+
|
|
15
|
+
"Deliver messages at network limited throughput using a cluster of machines with latencies as low as 2ms."
|
|
16
|
+
"Store streams of data safely in a distributed, durable, fault-tolerant cluster."
|
|
17
|
+
|
|
18
|
+
## Why Integrate
|
|
19
|
+
The Kafka v2 adapter from Itential is used to integrate the Itential Automation Platform (IAP) with Kafka. With this adapter you have the ability to perform operations with Kafka on items such as:
|
|
20
|
+
|
|
21
|
+
- Produce/Publish messages onto message topics for others to consume
|
|
22
|
+
- Listen, receive and process messages placed on topics by other systems
|
|
23
|
+
|
|
24
|
+
## Additional Product Documentation
|
|
25
|
+
The [Kafka Documentation](https://kafka.apache.org/documentation/)
|
|
26
|
+
The [Kafkajs Node Library Documentation](https://www.npmjs.com/package/kafkajs)
|
package/TAB1.md
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
# Overview
|
|
2
|
+
This adapter is used to integrate the Itential Automation Platform (IAP) with the Kafka System. The adapter utilizes the KafkaJS client to integrate with Apache Kafka. The ReadMe file is intended to provide information on this adapter it is generated from various other Markdown files.
|
|
3
|
+
|
|
4
|
+
|
|
5
|
+
## Details
|
|
6
|
+
The Kafka v2 adapter from Itential is used to integrate the Itential Automation Platform (IAP) with Kafka. With this adapter you have the ability to perform operations with Kafka on items such as:
|
|
7
|
+
|
|
8
|
+
- Produce/Publish messages onto message topics for others to consume
|
|
9
|
+
- Listen, receive and process messages placed on topics by other systems
|
|
10
|
+
|
|
11
|
+
For further technical details on how to install and use this adapter, please click the Technical Documentation tab.
|
package/TAB2.md
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
# Kafka v2
|
|
2
|
+
|
|
3
|
+
## Table of Contents
|
|
4
|
+
|
|
5
|
+
- [Specific Adapter Information](#specific-adapter-information)
|
|
6
|
+
- [Authentication](#authentication)
|
|
7
|
+
- [Sample Properties](#sample-properties)
|
|
8
|
+
- [Swagger](#swagger)
|
|
9
|
+
- [Generic Adapter Information](#generic-adapter-information)
|
|
10
|
+
|
|
11
|
+
## Specific Adapter Information
|
|
12
|
+
### Authentication
|
|
13
|
+
|
|
14
|
+
This document will go through the steps for authenticating the Kafka v2 adapter with. Properly configuring the properties for an adapter in IAP is critical for getting the adapter online. You can read more about adapter authentication <a href="https://docs.itential.com/opensource/docs/authentication" target="_blank">HERE</a>.
|
|
15
|
+
|
|
16
|
+
#### Library Authentication
|
|
17
|
+
The Kafka v2 adapter supports SASL Authentication for Kafka v2 server. If you change authentication methods, you should change this section accordingly and merge it back into the adapter repository.
|
|
18
|
+
|
|
19
|
+
STEPS
|
|
20
|
+
1. Ensure you have access to a Kafka v2 server and that it is running
|
|
21
|
+
2. Follow the steps in the README.md to import the adapter into IAP if you have not already done so
|
|
22
|
+
3. Use the properties below for the ```properties.client.sasl``` field
|
|
23
|
+
4. Kafka v2 adapter supports both PLAIN and SCRAM-SHA authentication mechanism
|
|
24
|
+
|
|
25
|
+
##### PLAIN
|
|
26
|
+
```json
|
|
27
|
+
"sasl": {
|
|
28
|
+
"mechanism": "PLAIN",
|
|
29
|
+
"username": "username",
|
|
30
|
+
"password": "password"
|
|
31
|
+
}
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
##### SCRAM-SHA
|
|
35
|
+
```json
|
|
36
|
+
"sasl": {
|
|
37
|
+
"mechanism": "scram-sha-512",
|
|
38
|
+
"username": "username",
|
|
39
|
+
"password": "password"
|
|
40
|
+
}
|
|
41
|
+
```
|
|
42
|
+
For more details on sasl authentication follow the README.md
|
|
43
|
+
4. Restart the adapter. If your properties were set correctly, the adapter should go online.
|
|
44
|
+
|
|
45
|
+
#### Troubleshooting
|
|
46
|
+
- Make sure you copied over the correct username and password.
|
|
47
|
+
- Turn on debug level logs for the adapter in IAP Admin Essentials.
|
|
48
|
+
- Investigate the logs
|
|
49
|
+
- Credentials should be ** masked ** by the adapter so make sure you verify the username and password - including that there are erroneous spaces at the front or end.
|
|
50
|
+
### Sample Properties
|
|
51
|
+
|
|
52
|
+
Sample Properties can be used to help you configure the adapter in the Itential Automation Platform. You will need to update connectivity information such as the host, port, protocol and credentials.
|
|
53
|
+
|
|
54
|
+
```json
|
|
55
|
+
"properties": {
|
|
56
|
+
"host": "localhost",
|
|
57
|
+
"port": 9092,
|
|
58
|
+
"interval_time": 5000,
|
|
59
|
+
"stub": false,
|
|
60
|
+
"parseMessage": true,
|
|
61
|
+
"wrapMessage": "myKey",
|
|
62
|
+
"check_iap_apps": true,
|
|
63
|
+
"check_wfe_status": true,
|
|
64
|
+
"check_ops_manager_status": false,
|
|
65
|
+
"fromBeginning": false,
|
|
66
|
+
"iap_apps_check_interval": 15000,
|
|
67
|
+
"client": {
|
|
68
|
+
"brokers": [
|
|
69
|
+
"broker:9093"
|
|
70
|
+
],
|
|
71
|
+
"clientId": "my-app",
|
|
72
|
+
"logLevel": "INFO"
|
|
73
|
+
},
|
|
74
|
+
"producer": {
|
|
75
|
+
"requireAcks": 1,
|
|
76
|
+
"ackTimeoutMs": 100,
|
|
77
|
+
"partitionerType": 0
|
|
78
|
+
},
|
|
79
|
+
"consumer": {
|
|
80
|
+
"groupId": "kafka-node-group"
|
|
81
|
+
},
|
|
82
|
+
"ssl": {
|
|
83
|
+
"enableTrace": true,
|
|
84
|
+
"ca": "/path/to/crt.pem",
|
|
85
|
+
"rejectUnauthorized": true
|
|
86
|
+
},
|
|
87
|
+
"sasl": {
|
|
88
|
+
"username": "my-user",
|
|
89
|
+
"password": "my-password",
|
|
90
|
+
"mechanism": "scram-sha-512"
|
|
91
|
+
},
|
|
92
|
+
"topics": [
|
|
93
|
+
{
|
|
94
|
+
"name": "t1",
|
|
95
|
+
"always": true,
|
|
96
|
+
"subscriberInfo": [
|
|
97
|
+
{
|
|
98
|
+
"subname": "default",
|
|
99
|
+
"filters": [
|
|
100
|
+
"PIXI",
|
|
101
|
+
"DRED",
|
|
102
|
+
"[abc]",
|
|
103
|
+
"F: (\\w+), L: (\\w+)",
|
|
104
|
+
"\\d{3,4}"
|
|
105
|
+
],
|
|
106
|
+
"rabbit": "kafka",
|
|
107
|
+
"throttle": {}
|
|
108
|
+
}
|
|
109
|
+
]
|
|
110
|
+
}
|
|
111
|
+
]
|
|
112
|
+
}
|
|
113
|
+
```
|
|
114
|
+
### Swagger
|
|
115
|
+
|
|
116
|
+
Note: The content for this section may be missing as its corresponding .json file is unavailable. This sections will be updated once adapter-openapi.json file is added.
|
|
117
|
+
## [Generic Adapter Information](https://gitlab.com/itentialopensource/adapters/notification-messaging/adapter-kafkav2/-/blob/master/README.md)
|
|
118
|
+
|
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
---------------------------------------------------------------------------------------------
|
|
2
|
+
**** Project Compliance Summary ****
|
|
3
|
+
0 project(s) are not valid
|
|
4
|
+
0 project(s) are valid
|
|
5
|
+
---------------------------------------------------------------------------------------------
|
package/metadata.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "adapter-kafkav2",
|
|
3
|
-
"webName": "Adapter for Integration to Kafka
|
|
3
|
+
"webName": "Adapter for Integration to Kafka using kafkajs",
|
|
4
4
|
"vendor": "Kafka",
|
|
5
5
|
"product": "Kafka v2",
|
|
6
6
|
"osVersion": [],
|
|
@@ -20,26 +20,50 @@
|
|
|
20
20
|
"Notifications & Ticket Management"
|
|
21
21
|
],
|
|
22
22
|
"useCases": [
|
|
23
|
-
"
|
|
23
|
+
"Produce a message",
|
|
24
|
+
"Consume a message"
|
|
24
25
|
],
|
|
25
26
|
"deprecated": {
|
|
26
27
|
"isDeprecated": false
|
|
27
28
|
},
|
|
28
29
|
"brokerSince": "",
|
|
30
|
+
"authMethods": [
|
|
31
|
+
{
|
|
32
|
+
"type": "SASL",
|
|
33
|
+
"subtypes": [],
|
|
34
|
+
"primary": true
|
|
35
|
+
}
|
|
36
|
+
],
|
|
29
37
|
"documentation": {
|
|
30
38
|
"storeLink": "",
|
|
31
39
|
"npmLink": "https://www.npmjs.com/package/@itentialopensource/adapter-kafkav2",
|
|
32
|
-
"repoLink": "https://gitlab.com/itentialopensource/adapters/
|
|
40
|
+
"repoLink": "https://gitlab.com/itentialopensource/adapters/adapter-kafkav2",
|
|
33
41
|
"docLink": "",
|
|
34
42
|
"demoLinks": [],
|
|
35
|
-
"trainingLinks": [
|
|
43
|
+
"trainingLinks": [
|
|
44
|
+
{
|
|
45
|
+
"title": "Itential Academy",
|
|
46
|
+
"link": "https://www.itential.com/itential-academy/"
|
|
47
|
+
}
|
|
48
|
+
],
|
|
36
49
|
"faqLink": "https://docs.itential.com/opensource/docs/troubleshooting-an-adapter",
|
|
37
50
|
"contributeLink": "https://gitlab.com/itentialopensource/adapters/contributing-guide",
|
|
38
51
|
"issueLink": "https://itential.atlassian.net/servicedesk/customer/portals",
|
|
39
52
|
"webLink": "",
|
|
40
|
-
"vendorLink": "",
|
|
41
|
-
"productLink": "",
|
|
42
|
-
"apiLinks": [
|
|
53
|
+
"vendorLink": "https://kafka.apache.org/",
|
|
54
|
+
"productLink": "https://kafka.apache.org/",
|
|
55
|
+
"apiLinks": [
|
|
56
|
+
{
|
|
57
|
+
"title": "Kafka Documentation",
|
|
58
|
+
"link": "https://kafka.apache.org/documentation/",
|
|
59
|
+
"public": true
|
|
60
|
+
},
|
|
61
|
+
{
|
|
62
|
+
"title": "Kafkajs Node Library Documentation",
|
|
63
|
+
"link": "https://www.npmjs.com/package/kafkajs",
|
|
64
|
+
"public": true
|
|
65
|
+
}
|
|
66
|
+
]
|
|
43
67
|
},
|
|
44
68
|
"assets": [],
|
|
45
69
|
"relatedItems": {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@itentialopensource/adapter-kafkav2",
|
|
3
|
-
"version": "0.22.
|
|
3
|
+
"version": "0.22.3",
|
|
4
4
|
"description": "Itential adapter to connect to kafka",
|
|
5
5
|
"main": "adapter.js",
|
|
6
6
|
"wizardVersion": "2.35.0",
|
|
@@ -33,14 +33,15 @@
|
|
|
33
33
|
],
|
|
34
34
|
"license": "Apache-2.0",
|
|
35
35
|
"engines": {
|
|
36
|
-
"node": ">=
|
|
36
|
+
"node": ">=14.0.0",
|
|
37
|
+
"npm": ">=6.0.0"
|
|
37
38
|
},
|
|
38
39
|
"repository": {
|
|
39
40
|
"type": "git",
|
|
40
|
-
"url": "git@gitlab.com:itentialopensource/adapters/
|
|
41
|
+
"url": "git@gitlab.com:itentialopensource/adapters/adapter-kafkav2.git"
|
|
41
42
|
},
|
|
42
43
|
"author": "Itential",
|
|
43
|
-
"homepage": "https://gitlab.com/itentialopensource/adapters/
|
|
44
|
+
"homepage": "https://gitlab.com/itentialopensource/adapters/adapter-kafkav2",
|
|
44
45
|
"dependencies": {
|
|
45
46
|
"ajv": "^8.12.0",
|
|
46
47
|
"avro-schema-registry": "^2.1.0",
|
|
@@ -60,9 +61,9 @@
|
|
|
60
61
|
"eslint-config-airbnb-base": "^15.0.0",
|
|
61
62
|
"eslint-plugin-import": "^2.27.5",
|
|
62
63
|
"eslint-plugin-json": "^3.1.0",
|
|
63
|
-
"mocha": "^
|
|
64
|
+
"mocha": "^10.7.0",
|
|
64
65
|
"testdouble": "^3.18.0",
|
|
65
|
-
"winston": "^3.
|
|
66
|
+
"winston": "^3.13.1"
|
|
66
67
|
},
|
|
67
68
|
"private": false
|
|
68
69
|
}
|
|
Binary file
|
package/sampleProperties.json
CHANGED
|
@@ -4,22 +4,59 @@
|
|
|
4
4
|
"properties": {
|
|
5
5
|
"host": "localhost",
|
|
6
6
|
"port": 9092,
|
|
7
|
-
"registry_url": "",
|
|
8
|
-
"topics": [],
|
|
9
7
|
"interval_time": 5000,
|
|
10
8
|
"stub": false,
|
|
9
|
+
"parseMessage": true,
|
|
10
|
+
"wrapMessage": "myKey",
|
|
11
|
+
"check_iap_apps": true,
|
|
12
|
+
"check_wfe_status": true,
|
|
13
|
+
"check_ops_manager_status": false,
|
|
14
|
+
"fromBeginning": false,
|
|
15
|
+
"iap_apps_check_interval": 15000,
|
|
11
16
|
"client": {
|
|
12
17
|
"brokers": [
|
|
13
|
-
"
|
|
18
|
+
"broker:9093"
|
|
14
19
|
],
|
|
15
|
-
"clientId": "my-app"
|
|
20
|
+
"clientId": "my-app",
|
|
21
|
+
"logLevel": "INFO"
|
|
16
22
|
},
|
|
17
23
|
"producer": {
|
|
18
24
|
"requireAcks": 1,
|
|
19
|
-
"ackTimeoutMs": 100
|
|
25
|
+
"ackTimeoutMs": 100,
|
|
26
|
+
"partitionerType": 0
|
|
20
27
|
},
|
|
21
28
|
"consumer": {
|
|
22
29
|
"groupId": "kafka-node-group"
|
|
23
|
-
}
|
|
30
|
+
},
|
|
31
|
+
"ssl": {
|
|
32
|
+
"enableTrace": true,
|
|
33
|
+
"ca": "/path/to/crt.pem",
|
|
34
|
+
"rejectUnauthorized": true
|
|
35
|
+
},
|
|
36
|
+
"sasl": {
|
|
37
|
+
"username": "my-user",
|
|
38
|
+
"password": "my-password",
|
|
39
|
+
"mechanism": "scram-sha-512"
|
|
40
|
+
},
|
|
41
|
+
"topics": [
|
|
42
|
+
{
|
|
43
|
+
"name": "t1",
|
|
44
|
+
"always": true,
|
|
45
|
+
"subscriberInfo": [
|
|
46
|
+
{
|
|
47
|
+
"subname": "default",
|
|
48
|
+
"filters": [
|
|
49
|
+
"PIXI",
|
|
50
|
+
"DRED",
|
|
51
|
+
"[abc]",
|
|
52
|
+
"F: (\\w+), L: (\\w+)",
|
|
53
|
+
"\\d{3,4}"
|
|
54
|
+
],
|
|
55
|
+
"rabbit": "kafka",
|
|
56
|
+
"throttle": {}
|
|
57
|
+
}
|
|
58
|
+
]
|
|
59
|
+
}
|
|
60
|
+
]
|
|
24
61
|
}
|
|
25
|
-
}
|
|
62
|
+
}
|