sentinel-kafka-manager 1.0.1 → 1.0.2

package/README.md

@@ -48,6 +48,292 @@ npm run build
 - Kafka cluster reachable from the service
 - `kafkajs` is included automatically as a dependency of this package
 
+## Create A Kafka Server With This Repo
+
+This repository already includes a full local Kafka server setup in [docker-compose.yml](/var/www/html/matrix-project/sentinel-kafka-manager/docker-compose.yml) and sample environment values in [.env.local.example](/var/www/html/matrix-project/sentinel-kafka-manager/.env.local.example).
+
+The stack includes:
+
+- 3 KRaft controllers
+- 3 Kafka brokers
+- 1 Kafka UI instance
+
+### 1. Create your `.env`
+
+Copy the example file:
+
+```bash
+cp .env.local.example .env
+```
+
+If you want a standard localhost setup, the example values are already suitable.
+
+### 2. Start the Kafka cluster
+
+```bash
+docker compose --env-file .env up -d
+```
+
+This creates:
+
+- controller quorum on port `9093` inside Docker
+- internal broker traffic on port `9092` inside Docker
+- host-accessible broker ports `29092`, `39092`, and `49092`
+- Kafka UI on `127.0.0.1:5001`
+
+### 3. Stop the Kafka cluster
+
+```bash
+docker compose --env-file .env down
+```
+
+To also remove persisted Kafka data volumes:
+
+```bash
+docker compose --env-file .env down -v
+```
+
+### 4. Check container status
+
+```bash
+docker compose --env-file .env ps
+```
+
+### 5. Current broker endpoints
+
+For applications running on your host machine:
+
+```text
+localhost:29092
+localhost:39092
+localhost:49092
+```
+
+For applications running inside Docker on the same Compose network:
+
+```text
+broker-1:9092
+broker-2:9092
+broker-3:9092
+```
+
+Important:
+
+- application clients must connect to brokers, not controllers
+- controllers are internal cluster metadata nodes only
+- `mode: 'external'` is for host-based apps
+- `mode: 'internal'` is for Dockerized apps on the same network
+
+## Kafka UI Usage
+
+The Docker stack also starts Kafka UI for cluster inspection.
+
+### Open the UI
+
+Visit:
+
+```text
+http://127.0.0.1:5001
+```
+
+### Login credentials
+
+By default, the UI uses the credentials from `.env`:
+
+```env
+KAFKA_UI_USERNAME=admin
+KAFKA_UI_PASSWORD=Admin@007
+```
+
+### What the UI connects to
+
+Kafka UI is preconfigured in `docker-compose.yml` to use all three brokers:
+
+```text
+broker-1:9092,broker-2:9092,broker-3:9092
+```
+
+The cluster name shown in the UI comes from:
+
+```env
+KAFKA_UI_CLUSTER_NAME=Project Omni Enterprise Kafka
+```
+
+### What you can do in the UI
+
+- view brokers and cluster health
+- inspect topics and partitions
+- browse messages
+- inspect consumer groups
+- check offsets and lag
+
+### Read-only mode
+
+The example `.env` sets:
+
+```env
+KAFKA_UI_READONLY=true
+```
+
+That means the UI is intended for safe inspection only.
+
+If you want to create topics or make changes from the UI, change it to:
+
+```env
+KAFKA_UI_READONLY=false
+```
+
+Then restart the stack:
+
+```bash
+docker compose --env-file .env up -d
+```
+
+### Typical UI flow
+
+1. Start the Docker stack.
+2. Open `http://127.0.0.1:5001`.
+3. Log in with `KAFKA_UI_USERNAME` and `KAFKA_UI_PASSWORD`.
+4. Open the configured cluster.
+5. Go to Topics, Consumer Groups, or Brokers as needed.
+
+## Docker Compose And `.env` Reference
+
+This project does not use a separate `Dockerfile` for Kafka. The Kafka server is created from `docker-compose.yml` and environment values from `.env`.
+
+### Core Kafka image and cluster identity
+
+```env
+KAFKA_IMAGE=confluentinc/cp-kafka:7.6.6
+KAFKA_CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk
+KAFKA_RESTART_POLICY=unless-stopped
+KAFKA_STOP_GRACE_PERIOD=60s
+KAFKA_NETWORK_NAME=kafka-network
+```
+
+- `KAFKA_IMAGE`: Kafka container image used by controllers and brokers
+- `KAFKA_CLUSTER_ID`: shared KRaft cluster id for all nodes
+- `KAFKA_NETWORK_NAME`: Docker network name used by the full stack
+
+### Controller quorum settings
+
+```env
+KAFKA_CONTROLLER_PORT=9093
+KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
+KAFKA_CONTROLLER_QUORUM_VOTERS=1@controller-1:9093,2@controller-2:9093,3@controller-3:9093
+```
+
+- controllers run only inside Docker
+- apps should never use these controller addresses as Kafka client brokers
+
+### Broker listener settings
+
+```env
+KAFKA_BROKER_BIND_ADDRESS=0.0.0.0
+KAFKA_EXTERNAL_HOST=localhost
+KAFKA_BROKER_INTERNAL_PORT=9092
+KAFKA_BROKER_EXTERNAL_CONTAINER_PORT=19092
+KAFKA_BROKER_1_EXTERNAL_PORT=29092
+KAFKA_BROKER_2_EXTERNAL_PORT=39092
+KAFKA_BROKER_3_EXTERNAL_PORT=49092
+KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
+KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
+```
+
+- `KAFKA_EXTERNAL_HOST=localhost` exposes brokers to your host machine
+- `KAFKA_BROKER_1_EXTERNAL_PORT`, `KAFKA_BROKER_2_EXTERNAL_PORT`, and `KAFKA_BROKER_3_EXTERNAL_PORT` are the ports your local apps should use
+- `KAFKA_BROKER_INTERNAL_PORT=9092` is used by containers on the Docker network
+
+### Replication and durability defaults
+
+```env
+KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=3
+KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=3
+KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=2
+KAFKA_DEFAULT_REPLICATION_FACTOR=3
+KAFKA_MIN_INSYNC_REPLICAS=2
+KAFKA_NUM_PARTITIONS=6
+```
+
+These values are designed for the included three-broker cluster.
+
+### Broker behavior
+
+```env
+KAFKA_AUTO_CREATE_TOPICS_ENABLE=false
+KAFKA_DELETE_TOPIC_ENABLE=true
+KAFKA_LOG_RETENTION_HOURS=168
+KAFKA_LOG_SEGMENT_BYTES=1073741824
+KAFKA_MESSAGE_MAX_BYTES=10485880
+KAFKA_REPLICA_FETCH_MAX_BYTES=10485880
+KAFKA_SOCKET_REQUEST_MAX_BYTES=104857600
+```
+
+- topics are not auto-created by default
+- deleting topics is allowed
+- retention is 7 days by default
+
+### Storage and safety limits
+
+```env
+KAFKA_LOG_DIRS=/var/lib/kafka/data
+KAFKA_ULIMIT_NOFILE_SOFT=65536
+KAFKA_ULIMIT_NOFILE_HARD=65536
+KAFKA_LOG_MAX_SIZE=100m
+KAFKA_LOG_MAX_FILE=5
+```
+
+### Kafka UI settings
+
+```env
+KAFKA_UI_IMAGE=provectuslabs/kafka-ui:latest
+KAFKA_UI_CONTAINER_NAME=kafka-ui
+KAFKA_UI_HOSTNAME=kafka-ui
+KAFKA_UI_RESTART_POLICY=unless-stopped
+KAFKA_UI_BIND_ADDRESS=127.0.0.1
+KAFKA_UI_PORT=5001
+KAFKA_UI_DYNAMIC_CONFIG_ENABLED=false
+KAFKA_UI_CLUSTER_NAME=Project Omni Enterprise Kafka
+KAFKA_UI_READONLY=true
+KAFKA_UI_AUTH_TYPE=LOGIN_FORM
+KAFKA_UI_USERNAME=admin
+KAFKA_UI_PASSWORD=Admin@007
+```
+
+- `KAFKA_UI_BIND_ADDRESS=127.0.0.1` keeps the UI local-only
+- `KAFKA_UI_PORT=5001` publishes the UI on your machine
+- `KAFKA_UI_READONLY=true` prevents write actions from the UI
+
+### Healthcheck settings
+
+```env
+KAFKA_HEALTHCHECK_INTERVAL=10s
+KAFKA_HEALTHCHECK_TIMEOUT=5s
+KAFKA_HEALTHCHECK_RETRIES=15
+KAFKA_HEALTHCHECK_START_PERIOD=30s
+```
+
+## Creating Topics
+
+Because `KAFKA_AUTO_CREATE_TOPICS_ENABLE=false`, topics should be created deliberately.
+
+You have two good options:
+
+- create topics from your service with `kafkaManager.provisionTopics()`
+- allow UI-based creation by setting `KAFKA_UI_READONLY=false`
+
+Example with this package:
+
+```ts
+await kafkaManager.provisionTopics([
+  {
+    topic: 'claims.created',
+    numPartitions: 6,
+    replicationFactor: 3,
+  },
+])
+```
+
 ## Enterprise Features
 
 This version now includes a stronger production baseline:
@@ -907,6 +1193,11 @@ npm whoami
 
 ## Links
 
+- npm deployment guide: [NPM_DEPLOYMENT.md](/var/www/html/matrix-project/sentinel-kafka-manager/NPM_DEPLOYMENT.md)
+- Example folder: [examples/service-module-usage/README.md](/var/www/html/matrix-project/sentinel-kafka-manager/examples/service-module-usage/README.md)
+- Service module guide: [SERVICE_MODULE_USAGE.md](/var/www/html/matrix-project/sentinel-kafka-manager/SERVICE_MODULE_USAGE.md)
+- GitHub: https://github.com/dilipshaw2024/sentinel-kafka-manager
+- npm: https://www.npmjs.com/package/sentinel-kafka-manager
 - LinkedIn: https://www.linkedin.com/in/dilip-shaw-2740769/
 
 ## About Me

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sentinel-kafka-manager",
-  "version": "1.0.1",
+  "version": "1.0.2",
   "description": "Reusable Kafka manager for Node.js microservices",
   "main": "dist/index.js",
   "module": "dist/index.mjs",