sentinel-kafka-manager 1.0.3 → 1.0.5

package/README.md

@@ -27,6 +27,36 @@ In most services, Kafka setup gets repeated:
 
 `sentinel-kafka-manager` wraps those common tasks in one small class: `KafkaManager`.
 
+## Enterprise Features
+
+This version now includes a stronger production baseline:
+
+- config validation for client id, brokers, and broker discovery inputs
+- shared producer, admin, and consumer lifecycle
+- logger and metrics hooks
+- standard message envelope publishing
+- managed consumer runner with dead-letter topic support
+- health snapshot reporting
+- safer startup behavior for concurrent connections
+- stable error codes and structured exception details
+
+## Recommended Usage Level
+
+This package is now suitable as a shared internal Kafka module for SaaS microservices.
+
+Recommended use:
+
+- internal platform package for Node.js services
+- event publishing with standard envelopes
+- managed consumers with DLQ handling
+- consistent service-layer success and error responses
+
+Still recommended outside the package:
+
+- automated unit and integration tests in the consuming service or platform repo
+- schema validation for business payloads when required by your organization
+- organization-specific tracing, alerting, and compliance rules
+
 ## Installation
 
 Install from npm:
@@ -106,281 +136,300 @@ process.on('SIGTERM', async () => {
 })
 ```
 
-## Create A Kafka Server With This Repo
-
-This repository already includes a full local Kafka server setup in [docker-compose.yml](/var/www/html/matrix-project/sentinel-kafka-manager/docker-compose.yml) and sample environment values in [.env.local.example](/var/www/html/matrix-project/sentinel-kafka-manager/.env.local.example).
-
-The stack includes:
-
-- 3 KRaft controllers
-- 3 Kafka brokers
-- 1 Kafka UI instance
-
-### 1. Create your `.env`
+## Basic Usage
 
-Copy the example file:
+### 1. Import the package
 
-```bash
-cp .env.local.example .env
+```ts
+import { KafkaManager } from 'sentinel-kafka-manager'
 ```
 
-If you want a standard localhost setup, the example values are already suitable.
-
-### 2. Start the Kafka cluster
+If you want to catch package-specific errors explicitly:
 
-```bash
-docker compose --env-file .env up -d
+```ts
+import {
+  KafkaErrorCode,
+  KafkaManager,
+  KafkaManagerError,
+  OperationResponse,
+} from 'sentinel-kafka-manager'
 ```
 
-This creates:
-
-- controller quorum on port `9093` inside Docker
-- internal broker traffic on port `9092` inside Docker
-- host-accessible broker ports `29092`, `39092`, and `49092`
-- Kafka UI on `127.0.0.1:5001`
+### 2. Create a manager
 
-### 3. Stop the Kafka cluster
+You can create it in two ways:
 
-```bash
-docker compose --env-file .env down
-```
+1. pass brokers directly
+2. build brokers from environment variables
 
-To also remove persisted Kafka data volumes:
+### Direct broker example
 
-```bash
-docker compose --env-file .env down -v
+```ts
+const kafkaManager = new KafkaManager({
+  clientId: 'claims-service',
+  brokers: ['localhost:29092', 'localhost:39092', 'localhost:49092'],
+})
 ```
 
-### 4. Check container status
+### Environment-based example
 
-```bash
-docker compose --env-file .env ps
+```ts
+const kafkaManager = KafkaManager.fromEnv({
+  clientId: 'claims-service',
+  mode: 'external',
+})
 ```
 
-### 5. Current broker endpoints
+## Recommended Project Pattern
 
-For applications running on your host machine:
+In a real service, create one shared manager instance and reuse it.
 
-```text
-localhost:29092
-localhost:39092
-localhost:49092
-```
+Example:
 
-For applications running inside Docker on the same Compose network:
+```ts
+import { KafkaManager } from 'sentinel-kafka-manager'
 
-```text
-broker-1:9092
-broker-2:9092
-broker-3:9092
+export const kafkaManager = KafkaManager.fromEnv({
+  clientId: 'claims-service',
+  mode: 'external',
+})
 ```
 
-Important:
-
-- application clients must connect to brokers, not controllers
-- controllers are internal cluster metadata nodes only
-- `mode: 'external'` is for host-based apps
-- `mode: 'internal'` is for Dockerized apps on the same network
-
-## Kafka UI Usage
-
-The Docker stack also starts Kafka UI for cluster inspection.
+Then import that shared instance anywhere you need Kafka access.
 
-### Open the UI
+You can also attach enterprise hooks:
 
-Visit:
+```ts
+import { KafkaManager } from 'sentinel-kafka-manager'
 
-```text
-http://127.0.0.1:5001
+export const kafkaManager = KafkaManager.fromEnv({
+  clientId: 'claims-service',
+  mode: 'external',
+  logger: {
+    debug: (message, meta) => console.debug(message, meta),
+    info: (message, meta) => console.info(message, meta),
+    warn: (message, meta) => console.warn(message, meta),
+    error: (message, meta) => console.error(message, meta),
+  },
+  metrics: {
+    emit: (event) => {
+      console.log('METRIC', event)
+    },
+  },
+})
 ```
 
-### Login credentials
+## How To Use In Your Service
 
-By default, the UI uses the credentials from `.env`:
+### Publish a message
 
-```env
-KAFKA_UI_USERNAME=admin
-KAFKA_UI_PASSWORD=Admin@007
+```ts
+await kafkaManager.publish({
+  topic: 'claims.created',
+  key: 'claim-1001',
+  payload: {
+    claimId: 'claim-1001',
+    status: 'created',
+    source: 'claims-service',
+  },
+})
 ```
 
-### What the UI connects to
+`publish()` automatically:
 
-Kafka UI is preconfigured in `docker-compose.yml` to use all three brokers:
+- gets a shared producer
+- connects it once
+- serializes `payload` with `JSON.stringify()`
 
-```text
-broker-1:9092,broker-2:9092,broker-3:9092
-```
+### Publish an enterprise event envelope
 
-The cluster name shown in the UI comes from:
+Use this when you want traceable event metadata such as `eventType`, `version`, `traceId`, or `tenantId`.
 
-```env
-KAFKA_UI_CLUSTER_NAME=Project Omni Enterprise Kafka
+```ts
+await kafkaManager.publishEnvelope({
+  topic: 'claims.created',
+  key: 'claim-1001',
+  envelope: {
+    eventId: 'evt-claim-1001',
+    eventType: 'claims.created',
+    version: '1.0.0',
+    timestamp: new Date().toISOString(),
+    source: 'claims-service',
+    traceId: 'trace-123',
+    tenantId: 'tenant-abc',
+    correlationId: 'corr-001',
+    payload: {
+      claimId: 'claim-1001',
+      status: 'created',
+    },
+  },
+})
 ```
 
-### What you can do in the UI
-
-- view brokers and cluster health
-- inspect topics and partitions
-- browse messages
-- inspect consumer groups
-- check offsets and lag
+### Create or reuse a producer manually
 
-### Read-only mode
+If you need direct producer access:
 
-The example `.env` sets:
+```ts
+const producer = await kafkaManager.getProducer()
 
-```env
-KAFKA_UI_READONLY=true
+await producer.send({
+  topic: 'claims.created',
+  messages: [
+    {
+      key: 'claim-1001',
+      value: JSON.stringify({ claimId: 'claim-1001' }),
+    },
+  ],
+})
 ```
 
-That means the UI is intended for safe inspection only.
+### Create or reuse a consumer
 
-If you want to create topics or make changes from the UI, change it to:
+```ts
+const consumer = await kafkaManager.getConsumer('claims-service-group')
 
-```env
-KAFKA_UI_READONLY=false
-```
+await consumer.subscribe({
+  topic: 'claims.created',
+  fromBeginning: false,
+})
 
-Then restart the stack:
+await consumer.run({
+  eachMessage: async ({ topic, partition, message }) => {
+    const rawValue = message.value?.toString() ?? '{}'
+    const payload = JSON.parse(rawValue)
 
-```bash
-docker compose --env-file .env up -d
+    console.log({
+      topic,
+      partition,
+      key: message.key?.toString(),
+      payload,
+    })
+  },
+})
 ```
 
-### Typical UI flow
+`getConsumer(groupId)` returns one shared connected consumer per group id.
 
-1. Start the Docker stack.
-2. Open `http://127.0.0.1:5001`.
-3. Log in with `KAFKA_UI_USERNAME` and `KAFKA_UI_PASSWORD`.
-4. Open the configured cluster.
-5. Go to Topics, Consumer Groups, or Brokers as needed.
+### Run a managed consumer with DLQ support
 
-## Docker Compose And `.env` Reference
+For SaaS-style services, this is the recommended consumer pattern.
 
-This project does not use a separate `Dockerfile` for Kafka. The Kafka server is created from `docker-compose.yml` and environment values from `.env`.
-
-### Core Kafka image and cluster identity
-
-```env
-KAFKA_IMAGE=confluentinc/cp-kafka:7.6.6
-KAFKA_CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk
-KAFKA_RESTART_POLICY=unless-stopped
-KAFKA_STOP_GRACE_PERIOD=60s
-KAFKA_NETWORK_NAME=kafka-network
+```ts
+await kafkaManager.runConsumer({
+  groupId: 'claims-service-group',
+  topic: 'claims.created',
+  dlqTopic: 'claims.created.dlq',
+  onMessage: async ({ message, headers, key }) => {
+    console.log('Processing:', {
+      key,
+      traceId: headers['x-trace-id'],
+      payload: message,
+    })
+  },
+  onError: async (error, context) => {
+    console.error('Consumer error:', {
+      error: error.message,
+      topic: context.topic,
+      offset: context.offset,
+    })
+  },
+})
 ```
 
-- `KAFKA_IMAGE`: Kafka container image used by controllers and brokers
-- `KAFKA_CLUSTER_ID`: shared KRaft cluster id for all nodes
-- `KAFKA_NETWORK_NAME`: Docker network name used by the full stack
-
-### Controller quorum settings
+If processing fails:
 
-```env
-KAFKA_CONTROLLER_PORT=9093
-KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
-KAFKA_CONTROLLER_QUORUM_VOTERS=1@controller-1:9093,2@controller-2:9093,3@controller-3:9093
-```
+- the error is logged
+- a metric hook can receive the failure event
+- the message can be published to a dead-letter topic when `dlqTopic` is set
 
-- controllers run only inside Docker
-- apps should never use these controller addresses as Kafka client brokers
+### Catch structured errors
 
-### Broker listener settings
+All package-thrown operational errors now use `KafkaManagerError`.
 
-```env
-KAFKA_BROKER_BIND_ADDRESS=0.0.0.0
-KAFKA_EXTERNAL_HOST=localhost
-KAFKA_BROKER_INTERNAL_PORT=9092
-KAFKA_BROKER_EXTERNAL_CONTAINER_PORT=19092
-KAFKA_BROKER_1_EXTERNAL_PORT=29092
-KAFKA_BROKER_2_EXTERNAL_PORT=39092
-KAFKA_BROKER_3_EXTERNAL_PORT=49092
-KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
-KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
-```
+```ts
+import {
+  KafkaErrorCode,
+  KafkaManagerError,
+} from 'sentinel-kafka-manager'
 
-- `KAFKA_EXTERNAL_HOST=localhost` exposes brokers to your host machine
-- `KAFKA_BROKER_1_EXTERNAL_PORT`, `KAFKA_BROKER_2_EXTERNAL_PORT`, and `KAFKA_BROKER_3_EXTERNAL_PORT` are the ports your local apps should use
-- `KAFKA_BROKER_INTERNAL_PORT=9092` is used by containers on the Docker network
+try {
+  await kafkaManager.publish({
+    topic: 'claims.created',
+    payload: { claimId: 'claim-1001' },
+  })
+} catch (error) {
+  if (error instanceof KafkaManagerError) {
+    console.error('Kafka error', {
+      code: error.code,
+      message: error.message,
+      details: error.details,
+    })
 
-### Replication and durability defaults
+    if (error.code === KafkaErrorCode.MESSAGE_PUBLISH_FAILED) {
+      // retry, alert, or degrade gracefully
+    }
+  }
 
-```env
-KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=3
-KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=3
-KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=2
-KAFKA_DEFAULT_REPLICATION_FACTOR=3
-KAFKA_MIN_INSYNC_REPLICAS=2
-KAFKA_NUM_PARTITIONS=6
+  throw error
+}
 ```
 
-These values are designed for the included three-broker cluster.
-
-### Broker behavior
-
-```env
-KAFKA_AUTO_CREATE_TOPICS_ENABLE=false
-KAFKA_DELETE_TOPIC_ENABLE=true
-KAFKA_LOG_RETENTION_HOURS=168
-KAFKA_LOG_SEGMENT_BYTES=1073741824
-KAFKA_MESSAGE_MAX_BYTES=10485880
-KAFKA_REPLICA_FETCH_MAX_BYTES=10485880
-KAFKA_SOCKET_REQUEST_MAX_BYTES=104857600
-```
+### Use common success and error response types
 
-- topics are not auto-created by default
-- deleting topics is allowed
-- retention is 7 days by default
+If your service wraps Kafka operations in API or service-layer responses, you can use the shared response contracts from this package.
 
-### Storage and safety limits
+```ts
+import {
+  ErrorResponse,
+  OperationResponse,
+  SuccessResponse,
+} from 'sentinel-kafka-manager'
 
-```env
-KAFKA_LOG_DIRS=/var/lib/kafka/data
-KAFKA_ULIMIT_NOFILE_SOFT=65536
-KAFKA_ULIMIT_NOFILE_HARD=65536
-KAFKA_LOG_MAX_SIZE=100m
-KAFKA_LOG_MAX_FILE=5
+type PublishClaimCreatedResponse = OperationResponse<{
+  topic: string
+  key: string
+}>
 ```
 
-### Kafka UI settings
+Success example:
 
-```env
-KAFKA_UI_IMAGE=provectuslabs/kafka-ui:latest
-KAFKA_UI_CONTAINER_NAME=kafka-ui
-KAFKA_UI_HOSTNAME=kafka-ui
-KAFKA_UI_RESTART_POLICY=unless-stopped
-KAFKA_UI_BIND_ADDRESS=127.0.0.1
-KAFKA_UI_PORT=5001
-KAFKA_UI_DYNAMIC_CONFIG_ENABLED=false
-KAFKA_UI_CLUSTER_NAME=Project Omni Enterprise Kafka
-KAFKA_UI_READONLY=true
-KAFKA_UI_AUTH_TYPE=LOGIN_FORM
-KAFKA_UI_USERNAME=admin
-KAFKA_UI_PASSWORD=Admin@007
+```ts
+const response: SuccessResponse<{ topic: string; key: string }> = {
+  success: true,
+  message: 'Kafka message published successfully.',
+  data: {
+    topic: 'claims.created',
+    key: 'claim-1001',
+  },
+  meta: {
+    timestamp: new Date().toISOString(),
+    traceId: 'trace-123',
+  },
+}
 ```
 
-- `KAFKA_UI_BIND_ADDRESS=127.0.0.1` keeps the UI local-only
-- `KAFKA_UI_PORT=5001` publishes the UI on your machine
-- `KAFKA_UI_READONLY=true` prevents write actions from the UI
-
-### Healthcheck settings
+Error example:
 
-```env
-KAFKA_HEALTHCHECK_INTERVAL=10s
-KAFKA_HEALTHCHECK_TIMEOUT=5s
-KAFKA_HEALTHCHECK_RETRIES=15
-KAFKA_HEALTHCHECK_START_PERIOD=30s
+```ts
+const response: ErrorResponse = {
+  success: false,
+  message: 'Failed to publish Kafka message.',
+  error: {
+    code: KafkaErrorCode.MESSAGE_PUBLISH_FAILED,
+    details: {
+      topic: 'claims.created',
+      key: 'claim-1001',
+    },
+  },
+  meta: {
+    timestamp: new Date().toISOString(),
+    retryable: true,
+  },
+}
 ```
 
-## Creating Topics
-
-Because `KAFKA_AUTO_CREATE_TOPICS_ENABLE=false`, topics should be created deliberately.
-
-You have two good options:
-
-- create topics from your service with `kafkaManager.provisionTopics()`
-- allow UI-based creation by setting `KAFKA_UI_READONLY=false`
-
-Example with this package:
+### Provision topics
 
 ```ts
 await kafkaManager.provisionTopics([
@@ -389,505 +438,458 @@ await kafkaManager.provisionTopics([
     numPartitions: 6,
     replicationFactor: 3,
   },
+  {
+    topic: 'claims.updated',
+    numPartitions: 6,
+    replicationFactor: 3,
+  },
 ])
 ```
 
-## Enterprise Features
+This is useful during service startup when your topics should exist before producers or consumers begin work.
 
-This version now includes a stronger production baseline:
+### Disconnect on shutdown
 
-- config validation for client id, brokers, and broker discovery inputs
-- shared producer, admin, and consumer lifecycle
-- logger and metrics hooks
-- standard message envelope publishing
-- managed consumer runner with dead-letter topic support
-- health snapshot reporting
-- safer startup behavior for concurrent connections
-- stable error codes and structured exception details
+Always close Kafka clients during application shutdown.
 
-## Recommended Usage Level
+```ts
+process.on('SIGINT', async () => {
+  await kafkaManager.disconnect()
+  process.exit(0)
+})
 
-This package is now suitable as a shared internal Kafka module for SaaS microservices.
+process.on('SIGTERM', async () => {
+  await kafkaManager.disconnect()
+  process.exit(0)
+})
+```
 
-Recommended use:
+### Inspect health
 
-- internal platform package for Node.js services
-- event publishing with standard envelopes
-- managed consumers with DLQ handling
-- consistent service-layer success and error responses
+```ts
+const health = kafkaManager.getHealthSnapshot()
 
-Still recommended outside the package:
+console.log(health)
+```
 
-- automated unit and integration tests in the consuming service or platform repo
-- schema validation for business payloads when required by your organization
-- organization-specific tracing, alerting, and compliance rules
+## Environment Setup
 
-## Basic Usage
+This package supports two environment styles:
 
-### 1. Import the package
+1. one explicit broker list with `KAFKA_BROKERS`
+2. derived brokers from your existing Kafka runtime variables
 
-```ts
-import { KafkaManager } from 'sentinel-kafka-manager'
+### Option 1: Use `KAFKA_BROKERS`
+
+This is the simplest option.
+
+### `.env`
+
+```env
+KAFKA_BROKERS=localhost:29092,localhost:39092,localhost:49092
 ```
 
-If you want to catch package-specific errors explicitly:
+### Code
 
 ```ts
-import {
-  KafkaErrorCode,
-  KafkaManager,
-  KafkaManagerError,
-  OperationResponse,
-} from 'sentinel-kafka-manager'
+const kafkaManager = KafkaManager.fromEnv({
+  clientId: 'claims-service',
+})
 ```
 
-### 2. Create a manager
+### Option 2: Use your current Kafka cluster variables
 
-You can create it in two ways:
+This matches the Kafka server setup you shared.
 
-1. pass brokers directly
-2. build brokers from environment variables
+### For host machine apps
 
-### Direct broker example
+Use this when your Node.js service runs on the host machine.
+
+### `.env`
+
+```env
+KAFKA_EXTERNAL_HOST=localhost
+KAFKA_BROKER_1_EXTERNAL_PORT=29092
+KAFKA_BROKER_2_EXTERNAL_PORT=39092
+KAFKA_BROKER_3_EXTERNAL_PORT=49092
+KAFKA_BROKER_INTERNAL_PORT=9092
+```
+
+### Code
 
 ```ts
-const kafkaManager = new KafkaManager({
+const kafkaManager = KafkaManager.fromEnv({
   clientId: 'claims-service',
-  brokers: ['localhost:29092', 'localhost:39092', 'localhost:49092'],
+  mode: 'external',
 })
 ```
 
-### Environment-based example
+This resolves to:
+
+```text
+localhost:29092
+localhost:39092
+localhost:49092
+```
+
+### For Dockerized apps on the Kafka network
+
+Use this when your Node.js service runs inside Docker on the same Kafka network.
+
+### `.env`
+
+```env
+KAFKA_BROKER_INTERNAL_PORT=9092
+```
+
+### Code
 
 ```ts
 const kafkaManager = KafkaManager.fromEnv({
   clientId: 'claims-service',
-  mode: 'external',
+  mode: 'internal',
 })
 ```
 
-## Recommended Project Pattern
+This resolves to:
+
+```text
+broker-1:9092
+broker-2:9092
+broker-3:9092
+```
+
+### Which Mode Should You Use?
 
-In a real service, create one shared manager instance and reuse it.
+- Use `mode: 'external'` when your service uses `localhost` broker ports like `29092`, `39092`, and `49092`
+- Use `mode: 'internal'` when your service is inside Docker and should reach brokers like `broker-1:9092`
+- Use `KAFKA_BROKERS` when you want the simplest and most explicit configuration
 
-Example:
+### Your Current Running Kafka Cluster
 
-```ts
-import { KafkaManager } from 'sentinel-kafka-manager'
+Based on your running Docker containers, your Kafka cluster is exposed like this:
 
-export const kafkaManager = KafkaManager.fromEnv({
-  clientId: 'claims-service',
-  mode: 'external',
-})
-```
+- `broker-1` -> host port `29092`
+- `broker-2` -> host port `39092`
+- `broker-3` -> host port `49092`
+- `controller-1`, `controller-2`, and `controller-3` are controller-only nodes and should not be used by application clients
 
-Then import that shared instance anywhere you need Kafka access.
+If your Node.js service runs on the host machine, use:
 
-You can also attach enterprise hooks:
+```env
+KAFKA_EXTERNAL_HOST=localhost
+KAFKA_BROKER_1_EXTERNAL_PORT=29092
+KAFKA_BROKER_2_EXTERNAL_PORT=39092
+KAFKA_BROKER_3_EXTERNAL_PORT=49092
+KAFKA_BROKER_INTERNAL_PORT=9092
+```
 
 ```ts
-import { KafkaManager } from 'sentinel-kafka-manager'
-
-export const kafkaManager = KafkaManager.fromEnv({
+const kafkaManager = KafkaManager.fromEnv({
   clientId: 'claims-service',
   mode: 'external',
-  logger: {
-    debug: (message, meta) => console.debug(message, meta),
-    info: (message, meta) => console.info(message, meta),
-    warn: (message, meta) => console.warn(message, meta),
-    error: (message, meta) => console.error(message, meta),
-  },
-  metrics: {
-    emit: (event) => {
-      console.log('METRIC', event)
-    },
-  },
 })
 ```
 
-## How To Use In Your Service
-
-### Publish a message
+If your Node.js service runs inside Docker on the same Kafka network, use:
 
 ```ts
-await kafkaManager.publish({
-  topic: 'claims.created',
-  key: 'claim-1001',
-  payload: {
-    claimId: 'claim-1001',
-    status: 'created',
-    source: 'claims-service',
-  },
+const kafkaManager = KafkaManager.fromEnv({
+  clientId: 'claims-service',
+  mode: 'internal',
 })
 ```
 
-`publish()` automatically:
+This package is designed to work with that exact broker layout.
 
-- gets a shared producer
-- connects it once
-- serializes `payload` with `JSON.stringify()`
+## Local Infrastructure And Operations
 
-### Publish an enterprise event envelope
+### Create A Kafka Server With This Repo
 
-Use this when you want traceable event metadata such as `eventType`, `version`, `traceId`, or `tenantId`.
+This repository already includes a full local Kafka server setup in [docker-compose.yml](/var/www/html/matrix-project/sentinel-kafka-manager/docker-compose.yml) and sample environment values in [.env.local.example](/var/www/html/matrix-project/sentinel-kafka-manager/.env.local.example).
 
-```ts
-await kafkaManager.publishEnvelope({
-  topic: 'claims.created',
-  key: 'claim-1001',
-  envelope: {
-    eventId: 'evt-claim-1001',
-    eventType: 'claims.created',
-    version: '1.0.0',
-    timestamp: new Date().toISOString(),
-    source: 'claims-service',
-    traceId: 'trace-123',
-    tenantId: 'tenant-abc',
-    correlationId: 'corr-001',
-    payload: {
-      claimId: 'claim-1001',
-      status: 'created',
-    },
-  },
-})
-```
+The stack includes:
 
-### Create or reuse a producer manually
+- 3 KRaft controllers
+- 3 Kafka brokers
+- 1 Kafka UI instance
 
-If you need direct producer access:
+### 1. Create your `.env`
 
-```ts
-const producer = await kafkaManager.getProducer()
+Copy the example file:
 
-await producer.send({
-  topic: 'claims.created',
-  messages: [
-    {
-      key: 'claim-1001',
-      value: JSON.stringify({ claimId: 'claim-1001' }),
-    },
-  ],
-})
+```bash
+cp .env.local.example .env
 ```
 
-### Create or reuse a consumer
+If you want a standard localhost setup, the example values are already suitable.
 
-```ts
-const consumer = await kafkaManager.getConsumer('claims-service-group')
+### 2. Start the Kafka cluster
 
-await consumer.subscribe({
-  topic: 'claims.created',
-  fromBeginning: false,
-})
+```bash
+docker compose --env-file .env up -d
+```
 
-await consumer.run({
-  eachMessage: async ({ topic, partition, message }) => {
-    const rawValue = message.value?.toString() ?? '{}'
-    const payload = JSON.parse(rawValue)
+This creates:
 
-    console.log({
-      topic,
-      partition,
-      key: message.key?.toString(),
-      payload,
-    })
-  },
-})
+- controller quorum on port `9093` inside Docker
+- internal broker traffic on port `9092` inside Docker
+- host-accessible broker ports `29092`, `39092`, and `49092`
+- Kafka UI on `127.0.0.1:5001`
+
+### 3. Stop the Kafka cluster
+
+```bash
+docker compose --env-file .env down
 ```
 
-`getConsumer(groupId)` returns one shared connected consumer per group id.
+To also remove persisted Kafka data volumes:
 
-### Run a managed consumer with DLQ support
+```bash
+docker compose --env-file .env down -v
+```
 
-For SaaS-style services, this is the recommended consumer pattern.
+### 4. Check container status
 
-```ts
-await kafkaManager.runConsumer({
-  groupId: 'claims-service-group',
-  topic: 'claims.created',
-  dlqTopic: 'claims.created.dlq',
-  onMessage: async ({ message, headers, key }) => {
-    console.log('Processing:', {
-      key,
-      traceId: headers['x-trace-id'],
-      payload: message,
-    })
-  },
-  onError: async (error, context) => {
-    console.error('Consumer error:', {
-      error: error.message,
-      topic: context.topic,
-      offset: context.offset,
-    })
-  },
-})
+```bash
+docker compose --env-file .env ps
 ```
 
-If processing fails:
+### 5. Current broker endpoints
 
-- the error is logged
-- a metric hook can receive the failure event
-- the message can be published to a dead-letter topic when `dlqTopic` is set
+For applications running on your host machine:
 
-### Catch structured errors
+```text
+localhost:29092
+localhost:39092
+localhost:49092
+```
 
-All package-thrown operational errors now use `KafkaManagerError`.
+For applications running inside Docker on the same Compose network:
 
-```ts
-import {
-  KafkaErrorCode,
-  KafkaManagerError,
-} from 'sentinel-kafka-manager'
+```text
+broker-1:9092
+broker-2:9092
+broker-3:9092
+```
 
-try {
-  await kafkaManager.publish({
-    topic: 'claims.created',
-    payload: { claimId: 'claim-1001' },
-  })
-} catch (error) {
-  if (error instanceof KafkaManagerError) {
-    console.error('Kafka error', {
-      code: error.code,
-      message: error.message,
-      details: error.details,
-    })
+Important:
 
-    if (error.code === KafkaErrorCode.MESSAGE_PUBLISH_FAILED) {
-      // retry, alert, or degrade gracefully
-    }
-  }
+- application clients must connect to brokers, not controllers
+- controllers are internal cluster metadata nodes only
+- `mode: 'external'` is for host-based apps
+- `mode: 'internal'` is for Dockerized apps on the same network
 
-  throw error
-}
-```
+### Kafka UI Usage
 
-### Use common success and error response types
+The Docker stack also starts Kafka UI for cluster inspection.
 
-If your service wraps Kafka operations in API or service-layer responses, you can use the shared response contracts from this package.
+### Open the UI
 
-```ts
-import {
-  ErrorResponse,
-  OperationResponse,
-  SuccessResponse,
-} from 'sentinel-kafka-manager'
+Visit:
 
-type PublishClaimCreatedResponse = OperationResponse<{
-  topic: string
-  key: string
-}>
+```text
+http://127.0.0.1:5001
 ```
 
-Success example:
+### Login credentials
 
-```ts
-const response: SuccessResponse<{ topic: string; key: string }> = {
-  success: true,
-  message: 'Kafka message published successfully.',
-  data: {
-    topic: 'claims.created',
-    key: 'claim-1001',
-  },
-  meta: {
-    timestamp: new Date().toISOString(),
-    traceId: 'trace-123',
-  },
-}
+By default, the UI uses the credentials from `.env`:
+
+```env
+KAFKA_UI_USERNAME=admin
+KAFKA_UI_PASSWORD=Admin@007
 ```
 
-Error example:
+### What the UI connects to
 
-```ts
-const response: ErrorResponse = {
-  success: false,
-  message: 'Failed to publish Kafka message.',
-  error: {
-    code: KafkaErrorCode.MESSAGE_PUBLISH_FAILED,
-    details: {
-      topic: 'claims.created',
-      key: 'claim-1001',
-    },
-  },
-  meta: {
-    timestamp: new Date().toISOString(),
-    retryable: true,
-  },
-}
+Kafka UI is preconfigured in `docker-compose.yml` to use all three brokers:
+
+```text
+broker-1:9092,broker-2:9092,broker-3:9092
 ```
 
-### Provision topics
+The cluster name shown in the UI comes from:
 
-```ts
-await kafkaManager.provisionTopics([
-  {
-    topic: 'claims.created',
-    numPartitions: 6,
-    replicationFactor: 3,
-  },
-  {
-    topic: 'claims.updated',
-    numPartitions: 6,
-    replicationFactor: 3,
-  },
-])
+```env
+KAFKA_UI_CLUSTER_NAME=Project Omni Enterprise Kafka
 ```
 
-This is useful during service startup when your topics should exist before producers or consumers begin work.
+### What you can do in the UI
 
-### Disconnect on shutdown
+- view brokers and cluster health
+- inspect topics and partitions
+- browse messages
+- inspect consumer groups
+- check offsets and lag
 
-Always close Kafka clients during application shutdown.
+### Read-only mode
 
-```ts
-process.on('SIGINT', async () => {
-  await kafkaManager.disconnect()
-  process.exit(0)
-})
+The example `.env` sets:
 
-process.on('SIGTERM', async () => {
-  await kafkaManager.disconnect()
-  process.exit(0)
-})
+```env
+KAFKA_UI_READONLY=true
 ```
 
-### Inspect health
+That means the UI is intended for safe inspection only.
 
-```ts
-const health = kafkaManager.getHealthSnapshot()
+If you want to create topics or make changes from the UI, change it to:
 
-console.log(health)
+```env
+KAFKA_UI_READONLY=false
 ```
 
-## Environment Setup
-
-This package supports two environment styles:
+Then restart the stack:
 
-1. one explicit broker list with `KAFKA_BROKERS`
-2. derived brokers from your existing Kafka runtime variables
+```bash
+docker compose --env-file .env up -d
+```
 
-## Option 1: Use `KAFKA_BROKERS`
+### Typical UI flow
 
-This is the simplest option.
+1. Start the Docker stack.
+2. Open `http://127.0.0.1:5001`.
+3. Log in with `KAFKA_UI_USERNAME` and `KAFKA_UI_PASSWORD`.
+4. Open the configured cluster.
+5. Go to Topics, Consumer Groups, or Brokers as needed.
 
-### `.env`
+### Docker Compose And `.env` Reference
 
-```env
-KAFKA_BROKERS=localhost:29092,localhost:39092,localhost:49092
-```
+This project does not use a separate `Dockerfile` for Kafka. The Kafka server is created from `docker-compose.yml` and environment values from `.env`.
 
-### Code
+### Core Kafka image and cluster identity
 
-```ts
-const kafkaManager = KafkaManager.fromEnv({
-  clientId: 'claims-service',
-})
+```env
+KAFKA_IMAGE=confluentinc/cp-kafka:7.6.6
+KAFKA_CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk
+KAFKA_RESTART_POLICY=unless-stopped
+KAFKA_STOP_GRACE_PERIOD=60s
+KAFKA_NETWORK_NAME=kafka-network
 ```
 
-## Option 2: Use your current Kafka cluster variables
+- `KAFKA_IMAGE`: Kafka container image used by controllers and brokers
+- `KAFKA_CLUSTER_ID`: shared KRaft cluster id for all nodes
+- `KAFKA_NETWORK_NAME`: Docker network name used by the full stack
 
-This matches the Kafka server setup you shared.
+### Controller quorum settings
 
-### For host machine apps
+```env
+KAFKA_CONTROLLER_PORT=9093
+KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
+KAFKA_CONTROLLER_QUORUM_VOTERS=1@controller-1:9093,2@controller-2:9093,3@controller-3:9093
+```
 
-Use this when your Node.js service runs on the host machine.
+- controllers run only inside Docker
+- apps should never use these controller addresses as Kafka client brokers
 
-### `.env`
+### Broker listener settings
 
 ```env
+KAFKA_BROKER_BIND_ADDRESS=0.0.0.0
 KAFKA_EXTERNAL_HOST=localhost
+KAFKA_BROKER_INTERNAL_PORT=9092
+KAFKA_BROKER_EXTERNAL_CONTAINER_PORT=19092
 KAFKA_BROKER_1_EXTERNAL_PORT=29092
 KAFKA_BROKER_2_EXTERNAL_PORT=39092
 KAFKA_BROKER_3_EXTERNAL_PORT=49092
-KAFKA_BROKER_INTERNAL_PORT=9092
+KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
+KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
 ```
 
-### Code
-
-```ts
-const kafkaManager = KafkaManager.fromEnv({
-  clientId: 'claims-service',
-  mode: 'external',
-})
-```
+- `KAFKA_EXTERNAL_HOST=localhost` exposes brokers to your host machine
+- `KAFKA_BROKER_1_EXTERNAL_PORT`, `KAFKA_BROKER_2_EXTERNAL_PORT`, and `KAFKA_BROKER_3_EXTERNAL_PORT` are the ports your local apps should use
+- `KAFKA_BROKER_INTERNAL_PORT=9092` is used by containers on the Docker network
 
-This resolves to:
+### Replication and durability defaults
 
-```text
-localhost:29092
-localhost:39092
-localhost:49092
+```env
+KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=3
+KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=3
+KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=2
+KAFKA_DEFAULT_REPLICATION_FACTOR=3
+KAFKA_MIN_INSYNC_REPLICAS=2
+KAFKA_NUM_PARTITIONS=6
 ```
 
-### For Dockerized apps on the Kafka network
-
-Use this when your Node.js service runs inside Docker on the same Kafka network.
+These values are designed for the included three-broker cluster.
 
-### `.env`
+### Broker behavior
 
 ```env
-KAFKA_BROKER_INTERNAL_PORT=9092
+KAFKA_AUTO_CREATE_TOPICS_ENABLE=false
+KAFKA_DELETE_TOPIC_ENABLE=true
+KAFKA_LOG_RETENTION_HOURS=168
+KAFKA_LOG_SEGMENT_BYTES=1073741824
+KAFKA_MESSAGE_MAX_BYTES=10485880
+KAFKA_REPLICA_FETCH_MAX_BYTES=10485880
+KAFKA_SOCKET_REQUEST_MAX_BYTES=104857600
 ```
 
-### Code
+- topics are not auto-created by default
+- deleting topics is allowed
+- retention is 7 days by default
 
-```ts
-const kafkaManager = KafkaManager.fromEnv({
-  clientId: 'claims-service',
-  mode: 'internal',
-})
+### Storage and safety limits
+
+```env
+KAFKA_LOG_DIRS=/var/lib/kafka/data
+KAFKA_ULIMIT_NOFILE_SOFT=65536
+KAFKA_ULIMIT_NOFILE_HARD=65536
+KAFKA_LOG_MAX_SIZE=100m
+KAFKA_LOG_MAX_FILE=5
 ```
 
-This resolves to:
+### Kafka UI settings
 
-```text
-broker-1:9092
-broker-2:9092
-broker-3:9092
+```env
+KAFKA_UI_IMAGE=provectuslabs/kafka-ui:latest
+KAFKA_UI_CONTAINER_NAME=kafka-ui
+KAFKA_UI_HOSTNAME=kafka-ui
+KAFKA_UI_RESTART_POLICY=unless-stopped
+KAFKA_UI_BIND_ADDRESS=127.0.0.1
+KAFKA_UI_PORT=5001
+KAFKA_UI_DYNAMIC_CONFIG_ENABLED=false
+KAFKA_UI_CLUSTER_NAME=Project Omni Enterprise Kafka
+KAFKA_UI_READONLY=true
+KAFKA_UI_AUTH_TYPE=LOGIN_FORM
+KAFKA_UI_USERNAME=admin
+KAFKA_UI_PASSWORD=Admin@007
 ```
 
-## Which Mode Should You Use?
-
-- Use `mode: 'external'` when your service uses `localhost` broker ports like `29092`, `39092`, and `49092`
-- Use `mode: 'internal'` when your service is inside Docker and should reach brokers like `broker-1:9092`
-- Use `KAFKA_BROKERS` when you want the simplest and most explicit configuration
+- `KAFKA_UI_BIND_ADDRESS=127.0.0.1` keeps the UI local-only
+- `KAFKA_UI_PORT=5001` publishes the UI on your machine
+- `KAFKA_UI_READONLY=true` prevents write actions from the UI
 
-## Your Current Running Kafka Cluster
+### Healthcheck settings
 
-Based on your running Docker containers, your Kafka cluster is exposed like this:
+```env
+KAFKA_HEALTHCHECK_INTERVAL=10s
+KAFKA_HEALTHCHECK_TIMEOUT=5s
+KAFKA_HEALTHCHECK_RETRIES=15
+KAFKA_HEALTHCHECK_START_PERIOD=30s
+```
 
-- `broker-1` -> host port `29092`
-- `broker-2` -> host port `39092`
-- `broker-3` -> host port `49092`
-- `controller-1`, `controller-2`, and `controller-3` are controller-only nodes and should not be used by application clients
+### Creating Topics
 
-If your Node.js service runs on the host machine, use:
+Because `KAFKA_AUTO_CREATE_TOPICS_ENABLE=false`, topics should be created deliberately.
 
-```env
-KAFKA_EXTERNAL_HOST=localhost
-KAFKA_BROKER_1_EXTERNAL_PORT=29092
-KAFKA_BROKER_2_EXTERNAL_PORT=39092
-KAFKA_BROKER_3_EXTERNAL_PORT=49092
-KAFKA_BROKER_INTERNAL_PORT=9092
-```
+You have two good options:
 
-```ts
-const kafkaManager = KafkaManager.fromEnv({
-  clientId: 'claims-service',
-  mode: 'external',
-})
-```
+- create topics from your service with `kafkaManager.provisionTopics()`
+- allow UI-based creation by setting `KAFKA_UI_READONLY=false`
 
-If your Node.js service runs inside Docker on the same Kafka network, use:
+Example with this package:
 
 ```ts
-const kafkaManager = KafkaManager.fromEnv({
-  clientId: 'claims-service',
-  mode: 'internal',
-})
+await kafkaManager.provisionTopics([
+  {
+    topic: 'claims.created',
+    numPartitions: 6,
+    replicationFactor: 3,
+  },
+])
 ```
 
-This package is designed to work with that exact broker layout.
-
 ## API Summary
 
 ### `new KafkaManager(options)`
@@ -1138,7 +1140,9 @@ const kafkaManager = new KafkaManager({
 })
 ```
 
-## Common Mistakes
+## Operational Guidance
+
+### Common Mistakes
 
 - Do not connect application clients to KRaft controller nodes. Connect only to brokers.
 - Do not use `mode: 'external'` inside Docker unless you truly want host-published ports.
@@ -1149,7 +1153,7 @@ const kafkaManager = new KafkaManager({
 - Do not publish business-critical events without metadata. Prefer `publishEnvelope()` for traceability.
 - Do not depend on raw driver error strings in application logic. Use `KafkaManagerError.code`.
 
-## Recommended SaaS Pattern
+### Recommended SaaS Pattern
 
 For most SaaS services, the cleanest pattern is:
 
@@ -1160,13 +1164,13 @@ For most SaaS services, the cleanest pattern is:
 5. Catch `KafkaManagerError` and branch on `error.code`.
 6. Wrap service outcomes in `OperationResponse<T>` where useful.
 
-## Notes
+### Notes
 
 - Your current Kafka cluster works fine with PLAINTEXT, so SSL and SASL are optional unless your environment changes
 - This package is designed for both host-based services and Dockerized services
 - If you want the least surprise, use `KAFKA_BROKERS`
 
-## Links
+## Additional Documentation
 
 - npm deployment guide: [NPM_DEPLOYMENT.md](/var/www/html/matrix-project/sentinel-kafka-manager/NPM_DEPLOYMENT.md)
 - Example folder: [examples/service-module-usage/README.md](/var/www/html/matrix-project/sentinel-kafka-manager/examples/service-module-usage/README.md)