sentinel-kafka-manager 1.0.1 → 1.0.3

package/README.md

@@ -48,6 +48,350 @@ npm run build
 - Kafka cluster reachable from the service
 - `kafkajs` is included automatically as a dependency of this package
 
+## Full Service Example
+
+```ts
+import { KafkaManager } from 'sentinel-kafka-manager'
+
+const kafkaManager = KafkaManager.fromEnv({
+  clientId: 'claims-api',
+  mode: 'external',
+})
+
+async function bootstrap(): Promise<void> {
+  await kafkaManager.provisionTopics([
+    {
+      topic: 'claims.created',
+      numPartitions: 6,
+      replicationFactor: 3,
+    },
+  ])
+
+  await kafkaManager.publish({
+    topic: 'claims.created',
+    key: 'claim-1001',
+    payload: {
+      claimId: 'claim-1001',
+      source: 'claims-api',
+    },
+  })
+
+  await kafkaManager.runConsumer({
+    groupId: 'claims-api-group',
+    topic: 'claims.created',
+    dlqTopic: 'claims.created.dlq',
+    onMessage: async ({ message, headers }) => {
+      console.log('Received event:', {
+        traceId: headers['x-trace-id'],
+        payload: message,
+      })
+    },
+  })
+}
+
+bootstrap().catch(async (error) => {
+  console.error('Kafka bootstrap failed:', error)
+  await kafkaManager.disconnect()
+  process.exit(1)
+})
+
+process.on('SIGINT', async () => {
+  await kafkaManager.disconnect()
+  process.exit(0)
+})
+
+process.on('SIGTERM', async () => {
+  await kafkaManager.disconnect()
+  process.exit(0)
+})
+```
+
+## Create A Kafka Server With This Repo
+
+This repository already includes a full local Kafka server setup in [docker-compose.yml](/var/www/html/matrix-project/sentinel-kafka-manager/docker-compose.yml) and sample environment values in [.env.local.example](/var/www/html/matrix-project/sentinel-kafka-manager/.env.local.example).
+
+The stack includes:
+
+- 3 KRaft controllers
+- 3 Kafka brokers
+- 1 Kafka UI instance
+
+### 1. Create your `.env`
+
+Copy the example file:
+
+```bash
+cp .env.local.example .env
+```
+
+If you want a standard localhost setup, the example values are already suitable.
+
+### 2. Start the Kafka cluster
+
+```bash
+docker compose --env-file .env up -d
+```
+
+This creates:
+
+- controller quorum on port `9093` inside Docker
+- internal broker traffic on port `9092` inside Docker
+- host-accessible broker ports `29092`, `39092`, and `49092`
+- Kafka UI on `127.0.0.1:5001`
+
+### 3. Stop the Kafka cluster
+
+```bash
+docker compose --env-file .env down
+```
+
+To also remove persisted Kafka data volumes:
+
+```bash
+docker compose --env-file .env down -v
+```
+
+### 4. Check container status
+
+```bash
+docker compose --env-file .env ps
+```
+
+### 5. Current broker endpoints
+
+For applications running on your host machine:
+
+```text
+localhost:29092
+localhost:39092
+localhost:49092
+```
+
+For applications running inside Docker on the same Compose network:
+
+```text
+broker-1:9092
+broker-2:9092
+broker-3:9092
+```
+
+Important:
+
+- application clients must connect to brokers, not controllers
+- controllers are internal cluster metadata nodes only
+- `mode: 'external'` is for host-based apps
+- `mode: 'internal'` is for Dockerized apps on the same network
+
+## Kafka UI Usage
+
+The Docker stack also starts Kafka UI for cluster inspection.
+
+### Open the UI
+
+Visit:
+
+```text
+http://127.0.0.1:5001
+```
+
+### Login credentials
+
+By default, the UI uses the credentials from `.env`:
+
+```env
+KAFKA_UI_USERNAME=admin
+KAFKA_UI_PASSWORD=Admin@007
+```
+
+### What the UI connects to
+
+Kafka UI is preconfigured in `docker-compose.yml` to use all three brokers:
+
+```text
+broker-1:9092,broker-2:9092,broker-3:9092
+```
+
+The cluster name shown in the UI comes from:
+
+```env
+KAFKA_UI_CLUSTER_NAME=Project Omni Enterprise Kafka
+```
+
+### What you can do in the UI
+
+- view brokers and cluster health
+- inspect topics and partitions
+- browse messages
+- inspect consumer groups
+- check offsets and lag
+
+### Read-only mode
+
+The example `.env` sets:
+
+```env
+KAFKA_UI_READONLY=true
+```
+
+That means the UI is intended for safe inspection only.
+
+If you want to create topics or make changes from the UI, change it to:
+
+```env
+KAFKA_UI_READONLY=false
+```
+
+Then restart the stack:
+
+```bash
+docker compose --env-file .env up -d
+```
+
+### Typical UI flow
+
+1. Start the Docker stack.
+2. Open `http://127.0.0.1:5001`.
+3. Log in with `KAFKA_UI_USERNAME` and `KAFKA_UI_PASSWORD`.
+4. Open the configured cluster.
+5. Go to Topics, Consumer Groups, or Brokers as needed.
+
+## Docker Compose And `.env` Reference
+
+This project does not use a separate `Dockerfile` for Kafka. The Kafka server is created from `docker-compose.yml` and environment values from `.env`.
+
+### Core Kafka image and cluster identity
+
+```env
+KAFKA_IMAGE=confluentinc/cp-kafka:7.6.6
+KAFKA_CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk
+KAFKA_RESTART_POLICY=unless-stopped
+KAFKA_STOP_GRACE_PERIOD=60s
+KAFKA_NETWORK_NAME=kafka-network
+```
+
+- `KAFKA_IMAGE`: Kafka container image used by controllers and brokers
+- `KAFKA_CLUSTER_ID`: shared KRaft cluster id for all nodes
+- `KAFKA_NETWORK_NAME`: Docker network name used by the full stack
+
+### Controller quorum settings
+
+```env
+KAFKA_CONTROLLER_PORT=9093
+KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
+KAFKA_CONTROLLER_QUORUM_VOTERS=1@controller-1:9093,2@controller-2:9093,3@controller-3:9093
+```
+
+- controllers run only inside Docker
+- apps should never use these controller addresses as Kafka client brokers
+
+### Broker listener settings
+
+```env
+KAFKA_BROKER_BIND_ADDRESS=0.0.0.0
+KAFKA_EXTERNAL_HOST=localhost
+KAFKA_BROKER_INTERNAL_PORT=9092
+KAFKA_BROKER_EXTERNAL_CONTAINER_PORT=19092
+KAFKA_BROKER_1_EXTERNAL_PORT=29092
+KAFKA_BROKER_2_EXTERNAL_PORT=39092
+KAFKA_BROKER_3_EXTERNAL_PORT=49092
+KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
+KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
+```
+
+- `KAFKA_EXTERNAL_HOST=localhost` exposes brokers to your host machine
+- `KAFKA_BROKER_1_EXTERNAL_PORT`, `KAFKA_BROKER_2_EXTERNAL_PORT`, and `KAFKA_BROKER_3_EXTERNAL_PORT` are the ports your local apps should use
+- `KAFKA_BROKER_INTERNAL_PORT=9092` is used by containers on the Docker network
+
+### Replication and durability defaults
+
+```env
+KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=3
+KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=3
+KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=2
+KAFKA_DEFAULT_REPLICATION_FACTOR=3
+KAFKA_MIN_INSYNC_REPLICAS=2
+KAFKA_NUM_PARTITIONS=6
+```
+
+These values are designed for the included three-broker cluster.
+
+### Broker behavior
+
+```env
+KAFKA_AUTO_CREATE_TOPICS_ENABLE=false
+KAFKA_DELETE_TOPIC_ENABLE=true
+KAFKA_LOG_RETENTION_HOURS=168
+KAFKA_LOG_SEGMENT_BYTES=1073741824
+KAFKA_MESSAGE_MAX_BYTES=10485880
+KAFKA_REPLICA_FETCH_MAX_BYTES=10485880
+KAFKA_SOCKET_REQUEST_MAX_BYTES=104857600
+```
+
+- topics are not auto-created by default
+- deleting topics is allowed
+- retention is 7 days by default
+
+### Storage and safety limits
+
+```env
+KAFKA_LOG_DIRS=/var/lib/kafka/data
+KAFKA_ULIMIT_NOFILE_SOFT=65536
+KAFKA_ULIMIT_NOFILE_HARD=65536
+KAFKA_LOG_MAX_SIZE=100m
+KAFKA_LOG_MAX_FILE=5
+```
+
+### Kafka UI settings
+
+```env
+KAFKA_UI_IMAGE=provectuslabs/kafka-ui:latest
+KAFKA_UI_CONTAINER_NAME=kafka-ui
+KAFKA_UI_HOSTNAME=kafka-ui
+KAFKA_UI_RESTART_POLICY=unless-stopped
+KAFKA_UI_BIND_ADDRESS=127.0.0.1
+KAFKA_UI_PORT=5001
+KAFKA_UI_DYNAMIC_CONFIG_ENABLED=false
+KAFKA_UI_CLUSTER_NAME=Project Omni Enterprise Kafka
+KAFKA_UI_READONLY=true
+KAFKA_UI_AUTH_TYPE=LOGIN_FORM
+KAFKA_UI_USERNAME=admin
+KAFKA_UI_PASSWORD=Admin@007
+```
+
+- `KAFKA_UI_BIND_ADDRESS=127.0.0.1` keeps the UI local-only
+- `KAFKA_UI_PORT=5001` publishes the UI on your machine
+- `KAFKA_UI_READONLY=true` prevents write actions from the UI
+
+### Healthcheck settings
+
+```env
+KAFKA_HEALTHCHECK_INTERVAL=10s
+KAFKA_HEALTHCHECK_TIMEOUT=5s
+KAFKA_HEALTHCHECK_RETRIES=15
+KAFKA_HEALTHCHECK_START_PERIOD=30s
+```
+
+## Creating Topics
+
+Because `KAFKA_AUTO_CREATE_TOPICS_ENABLE=false`, topics should be created deliberately.
+
+You have two good options:
+
+- create topics from your service with `kafkaManager.provisionTopics()`
+- allow UI-based creation by setting `KAFKA_UI_READONLY=false`
+
+Example with this package:
+
+```ts
+await kafkaManager.provisionTopics([
+  {
+    topic: 'claims.created',
+    numPartitions: 6,
+    replicationFactor: 3,
+  },
+])
+```
+
 ## Enterprise Features
 
 This version now includes a stronger production baseline:
@@ -544,64 +888,6 @@ const kafkaManager = KafkaManager.fromEnv({
 
 This package is designed to work with that exact broker layout.
 
-## Full Service Example
-
-```ts
-import { KafkaManager } from 'sentinel-kafka-manager'
-
-const kafkaManager = KafkaManager.fromEnv({
-  clientId: 'claims-api',
-  mode: 'external',
-})
-
-async function bootstrap(): Promise<void> {
-  await kafkaManager.provisionTopics([
-    {
-      topic: 'claims.created',
-      numPartitions: 6,
-      replicationFactor: 3,
-    },
-  ])
-
-  await kafkaManager.publish({
-    topic: 'claims.created',
-    key: 'claim-1001',
-    payload: {
-      claimId: 'claim-1001',
-      source: 'claims-api',
-    },
-  })
-
-  await kafkaManager.runConsumer({
-    groupId: 'claims-api-group',
-    topic: 'claims.created',
-    dlqTopic: 'claims.created.dlq',
-    onMessage: async ({ message, headers }) => {
-      console.log('Received event:', {
-        traceId: headers['x-trace-id'],
-        payload: message,
-      })
-    },
-  })
-}
-
-bootstrap().catch(async (error) => {
-  console.error('Kafka bootstrap failed:', error)
-  await kafkaManager.disconnect()
-  process.exit(1)
-})
-
-process.on('SIGINT', async () => {
-  await kafkaManager.disconnect()
-  process.exit(0)
-})
-
-process.on('SIGTERM', async () => {
-  await kafkaManager.disconnect()
-  process.exit(0)
-})
-```
-
 ## API Summary
 
 ### `new KafkaManager(options)`
@@ -874,31 +1160,6 @@ For most SaaS services, the cleanest pattern is:
 5. Catch `KafkaManagerError` and branch on `error.code`.
 6. Wrap service outcomes in `OperationResponse<T>` where useful.
 
-## Publishing This Package
-
-Build:
-
-```bash
-npm run build
-```
-
-Publish:
-
-```bash
-npm publish
-```
-
-If npm rejects the publish:
-
-- `403` usually means authentication, 2FA, or token permission issues
-- `404` usually means the package name is unavailable or not publishable by your account
-
-Useful check:
-
-```bash
-npm whoami
-```
-
 ## Notes
 
 - Your current Kafka cluster works fine with PLAINTEXT, so SSL and SASL are optional unless your environment changes
@@ -907,6 +1168,11 @@ npm whoami
 
 ## Links
 
+- npm deployment guide: [NPM_DEPLOYMENT.md](/var/www/html/matrix-project/sentinel-kafka-manager/NPM_DEPLOYMENT.md)
+- Example folder: [examples/service-module-usage/README.md](/var/www/html/matrix-project/sentinel-kafka-manager/examples/service-module-usage/README.md)
+- Service module guide: [SERVICE_MODULE_USAGE.md](/var/www/html/matrix-project/sentinel-kafka-manager/SERVICE_MODULE_USAGE.md)
+- GitHub: https://github.com/dilipshaw2024/sentinel-kafka-manager
+- npm: https://www.npmjs.com/package/sentinel-kafka-manager
 - LinkedIn: https://www.linkedin.com/in/dilip-shaw-2740769/
 
 ## About Me

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sentinel-kafka-manager",
-  "version": "1.0.1",
+  "version": "1.0.3",
   "description": "Reusable Kafka manager for Node.js microservices",
   "main": "dist/index.js",
   "module": "dist/index.mjs",