@backstage/plugin-events-backend-module-kafka 0.0.0-nightly-20251213024329 → 0.0.0-nightly-20260108025012

package/CHANGELOG.md

@@ -1,14 +1,33 @@
 # @backstage/plugin-events-backend-module-kafka
 
-## 0.0.0-nightly-20251213024329
+## 0.0.0-nightly-20260108025012
+
+### Minor Changes
+
+- ef5bbd8: Add support for Kafka offset configuration (`fromBeginning`) and `autoCommit`
 
 ### Patch Changes
 
 - Updated dependencies
-  - @backstage/plugin-events-node@0.0.0-nightly-20251213024329
-  - @backstage/backend-plugin-api@0.0.0-nightly-20251213024329
+  - @backstage/backend-plugin-api@1.6.0
   - @backstage/config@1.3.6
   - @backstage/types@1.2.2
+  - @backstage/plugin-events-node@0.4.18
+
+## 0.2.0
+
+### Minor Changes
+
+- 2c74ea9: Added support for multiple named instances in `kafkaConsumingEventPublisher` configuration. The previous single configuration format is still supported for backward compatibility.
+- 2c74ea9: Added `KafkaPublishingEventConsumer` to support sending Backstage events to Kafka topics.
+
+  This addition enables Backstage to publish events to external Kafka systems, complementing the existing ability to receive events from Kafka. This allows for better integration with external systems that rely on Kafka for event streaming.
+
+### Patch Changes
+
+- Updated dependencies
+  - @backstage/plugin-events-node@0.4.18
+  - @backstage/backend-plugin-api@1.6.0
 
 ## 0.1.6-next.1
 

package/README.md

@@ -2,65 +2,182 @@
 
 Welcome to the `events-backend-module-kafka` backend module!
 
-This package is a module for the `events-backend` backend plugin and extends the events system with an `KafkaConsumingEventPublisher.`
+This package is a module for the `events-backend` backend plugin and extends the events system with a `KafkaConsumingEventPublisher` and `KafkaPublishingEventConsumer`
 
-This event publisher will allow you to receive events from an Kafka queue and will publish these to the used `EventsService` implementation.
+This module provides two-way integration with Kafka:
+
+- **KafkaConsumingEventPublisher**: Receives events from Kafka queues and publishes them to the Backstage events system
+- **KafkaPublishingEventConsumer**: Consumes events from Backstage and publishes them to Kafka queues
 
 ## Configuration
 
-To set up Kafka queues, you need to configure the following values:
+To set up Kafka integration, you need to configure one or both of the following components:
+
+### KafkaConsumingEventPublisher Configuration
+
+To receive events from Kafka queues and publish them to Backstage:
 
 ```yaml
 events:
   modules:
     kafka:
       kafkaConsumingEventPublisher:
-        clientId: your-client-id # (Required) Client ID used by Backstage to identify when connecting to the Kafka cluster.
-        brokers: # (Required) List of brokers in the Kafka cluster to connect to.
-          - broker1
-          - broker2
-        topics:
-          - topic: 'backstage.topic' # (Required) Replace with actual topic name as expected by subscribers
-            kafka:
-              topics: # (Required) The Kafka topics to subscribe to.
-                - topic1
-              groupId: your-group-id # (Required) The GroupId to be used by the topic consumers.
+        production: # Instance name, will be included in logs
+          clientId: your-client-id # (Required) Client ID used by Backstage to identify when connecting to the Kafka cluster.
+          brokers: # (Required) List of brokers in the Kafka cluster to connect to.
+            - broker1
+            - broker2
+          topics:
+            - topic: 'backstage.topic' # (Required) Replace with actual topic name as expected by subscribers
+              kafka:
+                topics: # (Required) The Kafka topics to subscribe to.
+                  - topic1
+                groupId: your-group-id # (Required) The GroupId to be used by the topic consumers.
+                # Optional offset management settings (these can be omitted to use defaults):
+                # fromBeginning: false # Start from earliest offset when no committed offset exists. Default: not set (latest)
+                # autoCommit: true # Enable auto-commit. Default: true (for backward compatibility)
+                # pauseOnError: false # Pause consumer on error. Default: false (for backward compatibility)
+```
+
+### KafkaPublishingEventConsumer Configuration
+
+To publish events from Backstage to Kafka queues, you can configure the `KafkaPublishingEventConsumer`:
+
+```yaml
+events:
+  modules:
+    kafka:
+      kafkaPublishingEventConsumer:
+        production: # Instance name, will be included in logs
+          clientId: your-client-id # (Required) Client ID used by Backstage to identify when connecting to the Kafka cluster.
+          brokers: # (Required) List of brokers in the Kafka cluster to connect to.
+            - broker1
+            - broker2
+          topics:
+            - topic: 'catalog.entity.created' # (Required) The Backstage topic to consume from
+              kafka:
+                topic: kafka-topic-name # (Required) The Kafka topic to publish to
 ```
 
 For a complete list of all available fields that can be configured, refer to the [config.d.ts file](./config.d.ts).
 
+### Offset Management
+
+The plugin supports configurable offset management to control message delivery semantics:
+
+#### Auto Commit (Default - Backward Compatible)
+
+By default (`autoCommit: true` or not specified), Kafka automatically commits offsets at regular intervals. This is the original behavior and ensures backward compatibility.
+
+#### Manual Commit (Opt-in for Reliability)
+
+When you explicitly set `autoCommit: false`, the plugin will:
+
+1. Start consuming from the last committed offset for the consumer group
+2. Process each message by publishing it to the Backstage events system
+3. Only commit the offset after successful processing
+4. If processing fails, pause the consumer and do not commit the offset
+
+**Example configuration for manual commit:**
+
+```yaml
+kafka:
+  topics:
+    - topic1
+  groupId: my-group
+  autoCommit: false # Enable manual commit
+```
+
+#### Error Handling
+
+The `pauseOnError` option controls how the consumer behaves when message processing fails:
+
+**Skip Failed Messages (Default - Backward Compatible)**
+
+By default (`pauseOnError: false` or not specified), the consumer will skip failed messages and continue processing:
+
+- The consumer logs the error but continues processing subsequent messages
+- If `autoCommit: false`, the offset is still committed to skip the failed message
+- If `autoCommit: true`, Kafka's auto-commit handles the offset
+- Recommended when occasional message failures are acceptable and should not block processing
+
+**Pause on Error (Opt-in)**
+
+When you explicitly set `pauseOnError: true`, the consumer will pause when an error occurs during message processing:
+
+- The consumer pauses and stops processing new messages
+- The failed message offset is not committed
+- The error is re-thrown and logged
+- Recommended when you want to investigate and fix issues before continuing
+
+**Example configuration to pause on error:**
+
+```yaml
+kafka:
+  topics:
+    - topic1
+  groupId: my-group
+  autoCommit: false
+  pauseOnError: true # Pause consumer when a message fails
+```
+
+**Note:** When using the default behavior (`pauseOnError: false`) with `autoCommit: false`, failed messages will have their offsets committed, meaning they will be skipped and not reprocessed. Use this configuration carefully based on your application's requirements.
+
+#### Starting Position
+
+The `fromBeginning` option controls where the consumer starts when no committed offset exists:
+
+- `fromBeginning: true` - Start from the earliest available message
+- `fromBeginning: false` (default) - Start from the latest message (only new messages)
+
+Once the consumer group has committed an offset, it will always resume from that position, regardless of the `fromBeginning` setting.
+
 ### Optional SSL Configuration
 
-If your Kafka cluster requires SSL, you can configure it as follows:
+If your Kafka cluster requires SSL, you can configure it for both `kafkaConsumingEventPublisher` and `kafkaPublishingEventConsumer` instances:
 
 ```yaml
 events:
   modules:
     kafka:
       kafkaConsumingEventPublisher:
-        ssl:
-          rejectUnauthorized: true # (Optional) If true, the server certificate is verified against the list of supplied CAs.
-          ca: [path/to/ca-cert] # (Optional) Array of trusted certificates in PEM format.
-          key: path/to/client-key # (Optional) Private key in PEM format.
-          cert: path/to/client-cert # (Optional) Public x509 certificate in PEM format.
+        production:
+          # ... other configuration ...
+          ssl:
+            rejectUnauthorized: true # (Optional) If true, the server certificate is verified against the list of supplied CAs.
+            ca: [path/to/ca-cert] # (Optional) Array of trusted certificates in PEM format.
+            key: path/to/client-key # (Optional) Private key in PEM format.
+            cert: path/to/client-cert # (Optional) Public x509 certificate in PEM format.
+      kafkaPublishingEventConsumer:
+        production:
+          # ... other configuration ...
+          ssl:
+            # Same SSL configuration options as above
 ```
 
 ### Optional SASL Authentication Configuration
 
-If your Kafka cluster requires `SASL` authentication, you can configure it as follows:
+If your Kafka cluster requires SASL authentication, you can configure it for both components:
 
 ```yaml
 events:
   modules:
     kafka:
       kafkaConsumingEventPublisher:
-        sasl:
-          mechanism: 'plain' # SASL mechanism ('plain', 'scram-sha-256' or 'scram-sha-512')
-          username: your-username # SASL username
-          password: your-password # SASL password
+        production:
+          # ... other configuration ...
+          sasl:
+            mechanism: 'plain' # SASL mechanism ('plain', 'scram-sha-256' or 'scram-sha-512')
+            username: your-username # SASL username
+            password: your-password # SASL password
+      kafkaPublishingEventConsumer:
+        production:
+          # ... other configuration ...
+          sasl:
+            # Same SASL configuration options as above
 ```
 
-This section includes optional `SSL` and `SASL` authentication configuration for enhanced security.
+These SSL and SASL configurations apply to both Kafka components and provide enhanced security for your Kafka connections.
 
 ## Installation