@itentialopensource/adapter-kafkav2 0.12.0 → 0.14.0

package/CHANGELOG.md

@@ -1,4 +1,20 @@
 
+## 0.14.0 [08-04-2023]
+
+* Add logic to clean topics.json from service config
+
+See merge request itentialopensource/adapters/notification-messaging/adapter-kafkav2!11
+
+---
+
+## 0.13.0 [08-03-2023]
+
+* Add seperate sasl props for consumer and producer
+
+See merge request itentialopensource/adapters/notification-messaging/adapter-kafkav2!10
+
+---
+
 ## 0.12.0 [08-03-2023]
 
 * Add seperate sasl props for consumer and producer

package/README.md

@@ -75,6 +75,39 @@ git clone git@gitlab.com:\@itentialopensource/adapters/adapter-Kafkav2
 npm install
 ```
 
+## Installing an Itential Product Adapter
+
+1. Set up the name space location in your IAP node_modules.
+
+```json
+cd /opt/pronghorn/current/node_modules
+if the @itentialopensource directory does not exist, create it:
+   mkdir @itentialopensource
+```
+
+1. Clone the adapter into your IAP environment.
+
+```json
+cd \@itentialopensource
+git clone git@gitlab.com:\@itentialopensource/adapters/adapter-Kafka
+```
+
+1. Install the dependencies for the adapter.
+
+```json
+cd adapter-Kafka
+npm install
+```
+
+1. Add the adapter properties for Kafka (created from Adapter Builder) to the `properties.json` file for your Itential build. You will need to change the credentials and possibly the host information below.
+[Kafka sample properties](sampleProperties.json)
+
+1. Restart IAP
+
+```json
+systemctl restart pronghorn
+```
+
 ## Adapter Properties and Descriptions
 
 This section defines **all** the properties that are available for the adapter, including detailed information on what each property is for. If you are not using certain capabilities with this adapter, you do not need to define all of the properties. An example of how the properties for this adapter can be used with tests or IAP are provided in the **Installation** section.
@@ -96,9 +129,7 @@ This section defines **all** the properties that are available for the adapter,
       "brokers": [
           "broker:9093"
         ],
-      "clientId": "my-app",
-      "ssl": {},
-      "sasl": {}
+      "clientId": "my-app"
     },
     "producer": {
       "requireAcks": 1,
@@ -247,10 +278,10 @@ These base properties are used to connect to Kafka upon the adapter initially co
 
 | Property | Description |
 | ------- | ------- |
-| host | Optional. A fully qualified domain name or IP address.|
-| port | Required if `host` set. Used to connect to the server.|
+| host | Optional. A fully qualified domain name or IP address. This is not needed for latest version of the adapter as the broker is set under client properties.|
+| port | Required if `host` set. Used to connect to the server. This is not needed for latest version of the adapter as the port is set under client properties|
 | interval_time | Optional. The Kafka adapter keeps information about topics and offsets in memory in order to be more efficient. In order to work across restarts the adapter must persist the data. So the data is written into the .topics.json file. This write time defines how often to write the file.|
-| stub | Optional. Slightlu different meaning than normal, this is just telling the adapter whether to run the write topics interval. This should always be false unless you are running standalone tests.|
+| stub | Optional. Slightly different meaning than normal, this is just telling the adapter to not actually send any request to kafka and mock a healthy connection. Should be set to false unless you are wanting to force the adapter to be green without an actual connection.|
 
 #### Important
 When adapter works in consumer configuration: `autoCommit:false` `fromOffset:true` current offset setting on adapter startup is loaded from .topic.json file for each (topic:partition).
@@ -301,6 +332,8 @@ The `parseMessage` property allows the user to define how they want the Kafka me
 
 ## Send message sample payload
 
+Note that this is an array, so it can have multiple topics.
+
 ```json
 [{
     "topic": "topic-name",
@@ -312,122 +345,42 @@ The `parseMessage` property allows the user to define how they want the Kafka me
 ```
 For all options of messages array please follow [kafkajs documentation](https://kafka.js.org/docs/producing#producing-messages)
 
-## Testing an Itential Product Adapter
-
-Mocha is generally used to test all Itential Product Adapters. There are unit tests as well as integration tests performed. Integration tests can generally be run as standalone using mock data and running the adapter in stub mode, or as integrated. When running integrated, every effort is made to prevent environmental failures, however there is still a possibility.
+## Triggering jobs in IAP from a kafka message
 
-### Unit Testing
+Once you have the adapter configured and online in an IAP instance, you can create triggers in Operations Manager to kick off jobs when a kafka message is consumed. The adapter listens for messages on the subscribed topics and publishes to an event. This event is picked up by the Operations Manager. If you have created triggers for this specific topic, a job will get triggered. For a step-by-step example follow the next sub-section.
 
-Unit Testing includes testing basic adapter functionality as well as error conditions that are triggered in the adapter prior to any integration. There are two ways to run unit tests. The prefered method is to use the testRunner script; however, both methods are provided here.
+### Listen to a topic called test-topic and trigger a workflow called test
 
+1. Subscribe to partition 0 of test-topic. Add this to your adapter service config. To see different options of topic properties see PROPERTIES.md
 
 ```json
-node utils/testRunner --unit
-
-npm run test:unit
-```
-
-To add new unit tests, edit the `test/unit/adapterTestUnit.js` file. The tests that are already in this file should provide guidance for adding additional tests.
-
-### Integration Testing - Standalone
-
-Standalone Integration Testing requires mock data to be provided with the entities. If this data is not provided, standalone integration testing will fail. When the adapter is set to run in stub mode (setting the stub property to true), the adapter will run through its code up to the point of making the request. It will then retrieve the mock data and return that as if it had received that data as the response from Kafka. It will then translate the data so that the adapter can return the expected response to the rest of the Itential software. Standalone is the default integration test.
-
-Similar to unit testing, there are two ways to run integration tests. Using the testRunner script is better because it prevents you from having to edit the test script; it will also resets information after testing is complete so that credentials are not saved in the file.
-
-```json
-node utils/testRunner
-  answer no at the first prompt
-
-npm run test:integration
-```
-
-To add new integration tests, edit the `test/integration/adapterTestIntegration.js` file. The tests that are already in this file should provide guidance for adding additional tests.
-
-### Integration Testing
-
-Integration Testing requires connectivity to Kafka. By using the testRunner script it prevents you from having to edit the integration test. It also resets the integration test after the test is complete so that credentials are not saved in the file.
-
-> **Note**: These tests have been written as a best effort to make them work in most environments. However, the Adapter Builder often does not have the necessary information that is required to set up valid integration tests. For example, the order of the requests can be very important and data is often required for `creates` and `updates`. Hence, integration tests may have to be enhanced before they will work (integrate) with Kafka. Even after tests have been set up properly, it is possible there are environmental constraints that could result in test failures. Some examples of possible environmental issues are customizations that have been made within Kafka which change order dependencies or required data.
-
-```json
-node utils/testRunner
-answer yes at the first prompt
-answer all other questions on connectivity and credentials
-```
-
-Test should also be written to clean up after themselves. However, it is important to understand that in some cases this may not be possible. In addition, whenever exceptions occur, test execution may be stopped, which will prevent cleanup actions from running. It is recommended that tests be utilized in dev and test labs only.
-
-> **Reminder**: Do not check in code with actual credentials to systems.
-
-## Installing an Itential Product Adapter
-
-1. Set up the name space location in your IAP node_modules.
-
-```json
-cd /opt/pronghorn/current/node_modules
-if the @itentialopensource directory does not exist, create it:
-   mkdir @itentialopensource
-```
-
-1. Clone the adapter into your IAP environment.
-
-```json
-cd \@itentialopensource
-git clone git@gitlab.com:\@itentialopensource/adapters/adapter-Kafka
-```
-
-1. Install the dependencies for the adapter.
-
-```json
-cd adapter-Kafka
-npm install
-```
-
-1. Add the adapter properties for Kafka (created from Adapter Builder) to the `properties.json` file for your Itential build. You will need to change the credentials and possibly the host information below.
-[Kafka sample properties](sampleProperties.json)
-
-1. Restart IAP
-
-```json
-systemctl restart pronghorn
-```
-
-## Installing a Custom Adapter
-
-If you built this as a custom adapter through the Adapter Builder, it is recommended you go through setting up a development environment and testing the adapter before installing it. There is often configuration and authentication work that is required before the adapter will work in IAP.
-
-1. Move the adapter into the IAP `node_modules` directory.
-
-```json
-Depending on where your code is located, this process is different.
-    Could be a tar, move, untar
-    Could be a git clone of a repository
-    Could also be a cp -R from a coding directory
-Adapter should be placed into: /opt/pronghorn/current/node_modules/\@itentialopensource
+"topics": [
+  {
+    "name": "test-topic",
+    "always": true,
+    "partition": 0,
+    "subscriberInfo": [
+      {
+        "subname": "default",
+        "filters": [],
+        "rabbit": "test-topic-rabbit",
+        "throttle": {}
+      }
+    ]
+  }
+],
 ```
 
-1. Follow Steps 3-5 (above) to install an Itential adapter to load your properties, dependencies and restart IAP.
+Note the rabbit key in the config is set to kafka. The adapter will send any message on test-topic-rabbit to a Rabbit queue named kafka. If no rabbit topic is supplied, events will be published to a topic with the same name as the Kafka topic. Also note that the filter array can be left empty to consume all messages on partion 0 of test-topic.
 
-## Using this Adapter
+2. Create an automation in Operations Manager
 
-The `adapter.js` file contains the calls the adapter makes available to the rest of the Itential Platform. The API detailed for these calls should be available through JSDOC. The following is a brief summary of the calls.
+Create a new automation, and add a trigger.
 
-### Generic Adapter Calls
-
-```json
-connect()
-The connect call is run when the Adapter is first loaded by he Itential Platform. It validates the properties have been provided correctly.
-```
-
-```json
-healthCheck(callback)
-Insures that the adapter can communicate with Kafka. The actual call that is used is defined in the adapter properties.
-```
+When selecting the Event for your trigger, make sure to select test-topic-rabbit from the drop down list. This is critical as the adpater will be sending the message on this queue. You can add filtering in the trigger by providing a Payload Schema Filter.
 
-### Specific Adapter Calls
+After saving the trigger, select the workflow you want to trigger. Save the Automation and view all jobs to see a job getting triggered as soon as a message is published on test-topic partition 0.
 
-Specific adapter calls are built based on the API of the Kafka. The Adapter Builder creates the proper method comments for generating JS-DOC for the adapter. This is the best way to get information on the calls.
 
 ## Troubleshooting the Adapter
 

package/adapter.js

@@ -349,12 +349,45 @@ class Kafkav2 extends AdapterBaseCl {
               processed: 0,
               subscribers: 99999,
               avro: useAvro,
-              subInfo
+              subInfo,
+              source: 'props'
             });
           }
         }
       }
 
+      // check if service config doesnt have a topic and the source was props, delete it from topics
+      for (let te = 0; te < this.topicsEvents.length; te += 1) {
+        let topicFound = false;
+        for (let t = 0; t < this.props.topics.length; t += 1) {
+          if (this.topicsEvents[te].topic === this.props.topics[t].name && this.topicsEvents[te].source) {
+            if (this.topicsEvents[te].subInfo) {
+              const { subInfo } = this.topicsEvents[te];
+              for (let sub = 0; sub < subInfo.length; sub += 1) {
+                let subFound = false;
+                for (let s = 0; s < this.props.topics[t].subscriberInfo.length; s += 1) {
+                  // if the subscriber is found, no need to add anything
+                  if (subInfo[sub].subname === this.props.topics[t].subscriberInfo[s].subname) {
+                    subFound = true;
+                    break;
+                  }
+                }
+                if (subFound === false) {
+                  this.topicsEvents[te].subInfo.splice(sub, 1);
+                }
+              }
+            }
+            topicFound = true;
+          }
+        }
+        if (topicFound === false && this.topicsEvents[te].source === 'props') {
+          this.topicsEvents.splice(te, 1);
+          const split = topicsFile.split('/');
+          log.debug(`Update ${split[split.length - 1]} file.`);
+          fs.writeFileSync(topicsFile, JSON.stringify(this.topicsEvents, null, 2));
+        }
+      }
+
       // if we need to restart the adapter due to changes to pronghorn.json
       if (needRestart) {
         fs.writeFileSync(pronghornFile, JSON.stringify(pjson, null, 2));

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@itentialopensource/adapter-kafkav2",
-  "version": "0.12.0",
+  "version": "0.14.0",
   "description": "Itential adapter to connect to kafka",
   "main": "adapter.js",
   "wizardVersion": "2.35.0",