sentinel-kafka-manager 1.0.0 → 1.0.2

package/README.md

@@ -2,56 +2,454 @@
 
 Reusable Kafka manager for Node.js microservices built on top of `kafkajs`.
 
-This package helps services:
+This package gives you one shared way to:
 
-- connect to Kafka with a small shared API
-- create and reuse producers, consumers, and admin clients safely
+- connect to Kafka
+- reuse a single producer safely
+- reuse consumers by `groupId`
+- provision topics if they do not exist
 - publish JSON messages
-- provision topics if they do not already exist
-- bootstrap from environment variables for both Docker-internal and host-external Kafka access
+- build broker lists from environment variables
+- attach logging and metrics hooks
+- publish standard event envelopes
+- run consumers with DLQ support
+- inspect connection health
+
+## What This Package Solves
+
+In most services, Kafka setup gets repeated:
+
+- build broker arrays
+- create producers
+- create consumers
+- manage connect and disconnect
+- handle Docker-internal vs host-external broker addresses
+
+`sentinel-kafka-manager` wraps those common tasks in one small class: `KafkaManager`.
 
 ## Installation
 
+Install from npm:
+
 ```bash
 npm install sentinel-kafka-manager
 ```
 
-For local development in this package:
+For local package development:
 
 ```bash
 npm install
 npm run build
 ```
 
-## Features
+## Requirements
+
+- Node.js service
+- Kafka cluster reachable from the service
+- `kafkajs` is included automatically as a dependency of this package
+
+## Create A Kafka Server With This Repo
+
+This repository already includes a full local Kafka server setup in [docker-compose.yml](/var/www/html/matrix-project/sentinel-kafka-manager/docker-compose.yml) and sample environment values in [.env.local.example](/var/www/html/matrix-project/sentinel-kafka-manager/.env.local.example).
+
+The stack includes:
+
+- 3 KRaft controllers
+- 3 Kafka brokers
+- 1 Kafka UI instance
+
+### 1. Create your `.env`
+
+Copy the example file:
+
+```bash
+cp .env.local.example .env
+```
+
+If you want a standard localhost setup, the example values are already suitable.
+
+### 2. Start the Kafka cluster
+
+```bash
+docker compose --env-file .env up -d
+```
+
+This creates:
+
+- controller quorum on port `9093` inside Docker
+- internal broker traffic on port `9092` inside Docker
+- host-accessible broker ports `29092`, `39092`, and `49092`
+- Kafka UI on `127.0.0.1:5001`
+
+### 3. Stop the Kafka cluster
+
+```bash
+docker compose --env-file .env down
+```
+
+To also remove persisted Kafka data volumes:
+
+```bash
+docker compose --env-file .env down -v
+```
+
+### 4. Check container status
+
+```bash
+docker compose --env-file .env ps
+```
+
+### 5. Current broker endpoints
+
+For applications running on your host machine:
+
+```text
+localhost:29092
+localhost:39092
+localhost:49092
+```
+
+For applications running inside Docker on the same Compose network:
+
+```text
+broker-1:9092
+broker-2:9092
+broker-3:9092
+```
+
+Important:
+
+- application clients must connect to brokers, not controllers
+- controllers are internal cluster metadata nodes only
+- `mode: 'external'` is for host-based apps
+- `mode: 'internal'` is for Dockerized apps on the same network
+
+## Kafka UI Usage
+
+The Docker stack also starts Kafka UI for cluster inspection.
+
+### Open the UI
+
+Visit:
+
+```text
+http://127.0.0.1:5001
+```
+
+### Login credentials
+
+By default, the UI uses the credentials from `.env`:
+
+```env
+KAFKA_UI_USERNAME=admin
+KAFKA_UI_PASSWORD=Admin@007
+```
+
+### What the UI connects to
+
+Kafka UI is preconfigured in `docker-compose.yml` to use all three brokers:
+
+```text
+broker-1:9092,broker-2:9092,broker-3:9092
+```
+
+The cluster name shown in the UI comes from:
+
+```env
+KAFKA_UI_CLUSTER_NAME=Project Omni Enterprise Kafka
+```
+
+### What you can do in the UI
+
+- view brokers and cluster health
+- inspect topics and partitions
+- browse messages
+- inspect consumer groups
+- check offsets and lag
+
+### Read-only mode
+
+The example `.env` sets:
+
+```env
+KAFKA_UI_READONLY=true
+```
+
+That means the UI is intended for safe inspection only.
+
+If you want to create topics or make changes from the UI, change it to:
+
+```env
+KAFKA_UI_READONLY=false
+```
+
+Then restart the stack:
+
+```bash
+docker compose --env-file .env up -d
+```
+
+### Typical UI flow
+
+1. Start the Docker stack.
+2. Open `http://127.0.0.1:5001`.
+3. Log in with `KAFKA_UI_USERNAME` and `KAFKA_UI_PASSWORD`.
+4. Open the configured cluster.
+5. Go to Topics, Consumer Groups, or Brokers as needed.
+
+## Docker Compose And `.env` Reference
+
+This project does not use a separate `Dockerfile` for Kafka. The Kafka server is created from `docker-compose.yml` and environment values from `.env`.
+
+### Core Kafka image and cluster identity
+
+```env
+KAFKA_IMAGE=confluentinc/cp-kafka:7.6.6
+KAFKA_CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk
+KAFKA_RESTART_POLICY=unless-stopped
+KAFKA_STOP_GRACE_PERIOD=60s
+KAFKA_NETWORK_NAME=kafka-network
+```
+
+- `KAFKA_IMAGE`: Kafka container image used by controllers and brokers
+- `KAFKA_CLUSTER_ID`: shared KRaft cluster id for all nodes
+- `KAFKA_NETWORK_NAME`: Docker network name used by the full stack
+
+### Controller quorum settings
+
+```env
+KAFKA_CONTROLLER_PORT=9093
+KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
+KAFKA_CONTROLLER_QUORUM_VOTERS=1@controller-1:9093,2@controller-2:9093,3@controller-3:9093
+```
+
+- controllers run only inside Docker
+- apps should never use these controller addresses as Kafka client brokers
+
+### Broker listener settings
+
+```env
+KAFKA_BROKER_BIND_ADDRESS=0.0.0.0
+KAFKA_EXTERNAL_HOST=localhost
+KAFKA_BROKER_INTERNAL_PORT=9092
+KAFKA_BROKER_EXTERNAL_CONTAINER_PORT=19092
+KAFKA_BROKER_1_EXTERNAL_PORT=29092
+KAFKA_BROKER_2_EXTERNAL_PORT=39092
+KAFKA_BROKER_3_EXTERNAL_PORT=49092
+KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
+KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
+```
+
+- `KAFKA_EXTERNAL_HOST=localhost` exposes brokers to your host machine
+- `KAFKA_BROKER_1_EXTERNAL_PORT`, `KAFKA_BROKER_2_EXTERNAL_PORT`, and `KAFKA_BROKER_3_EXTERNAL_PORT` are the ports your local apps should use
+- `KAFKA_BROKER_INTERNAL_PORT=9092` is used by containers on the Docker network
+
+### Replication and durability defaults
+
+```env
+KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=3
+KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=3
+KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=2
+KAFKA_DEFAULT_REPLICATION_FACTOR=3
+KAFKA_MIN_INSYNC_REPLICAS=2
+KAFKA_NUM_PARTITIONS=6
+```
+
+These values are designed for the included three-broker cluster.
+
+### Broker behavior
+
+```env
+KAFKA_AUTO_CREATE_TOPICS_ENABLE=false
+KAFKA_DELETE_TOPIC_ENABLE=true
+KAFKA_LOG_RETENTION_HOURS=168
+KAFKA_LOG_SEGMENT_BYTES=1073741824
+KAFKA_MESSAGE_MAX_BYTES=10485880
+KAFKA_REPLICA_FETCH_MAX_BYTES=10485880
+KAFKA_SOCKET_REQUEST_MAX_BYTES=104857600
+```
 
-- built on `kafkajs`
-- supports direct broker lists or env-driven broker discovery
-- supports internal Docker-network broker routing
-- supports external localhost or host-based broker routing
-- supports optional SSL, SASL, retry, and timeout settings
-- avoids duplicate producer, consumer, and admin connections during concurrent startup
+- topics are not auto-created by default
+- deleting topics is allowed
+- retention is 7 days by default
 
-## Exports
+### Storage and safety limits
+
+```env
+KAFKA_LOG_DIRS=/var/lib/kafka/data
+KAFKA_ULIMIT_NOFILE_SOFT=65536
+KAFKA_ULIMIT_NOFILE_HARD=65536
+KAFKA_LOG_MAX_SIZE=100m
+KAFKA_LOG_MAX_FILE=5
+```
+
+### Kafka UI settings
+
+```env
+KAFKA_UI_IMAGE=provectuslabs/kafka-ui:latest
+KAFKA_UI_CONTAINER_NAME=kafka-ui
+KAFKA_UI_HOSTNAME=kafka-ui
+KAFKA_UI_RESTART_POLICY=unless-stopped
+KAFKA_UI_BIND_ADDRESS=127.0.0.1
+KAFKA_UI_PORT=5001
+KAFKA_UI_DYNAMIC_CONFIG_ENABLED=false
+KAFKA_UI_CLUSTER_NAME=Project Omni Enterprise Kafka
+KAFKA_UI_READONLY=true
+KAFKA_UI_AUTH_TYPE=LOGIN_FORM
+KAFKA_UI_USERNAME=admin
+KAFKA_UI_PASSWORD=Admin@007
+```
+
+- `KAFKA_UI_BIND_ADDRESS=127.0.0.1` keeps the UI local-only
+- `KAFKA_UI_PORT=5001` publishes the UI on your machine
+- `KAFKA_UI_READONLY=true` prevents write actions from the UI
+
+### Healthcheck settings
+
+```env
+KAFKA_HEALTHCHECK_INTERVAL=10s
+KAFKA_HEALTHCHECK_TIMEOUT=5s
+KAFKA_HEALTHCHECK_RETRIES=15
+KAFKA_HEALTHCHECK_START_PERIOD=30s
+```
+
+## Creating Topics
+
+Because `KAFKA_AUTO_CREATE_TOPICS_ENABLE=false`, topics should be created deliberately.
+
+You have two good options:
+
+- create topics from your service with `kafkaManager.provisionTopics()`
+- allow UI-based creation by setting `KAFKA_UI_READONLY=false`
+
+Example with this package:
 
 ```ts
-import { KafkaManager } from 'sentinel-kafka-manager'
+await kafkaManager.provisionTopics([
+  {
+    topic: 'claims.created',
+    numPartitions: 6,
+    replicationFactor: 3,
+  },
+])
 ```
 
-## Quick Start
+## Enterprise Features
+
+This version now includes a stronger production baseline:
+
+- config validation for client id, brokers, and broker discovery inputs
+- shared producer, admin, and consumer lifecycle
+- logger and metrics hooks
+- standard message envelope publishing
+- managed consumer runner with dead-letter topic support
+- health snapshot reporting
+- safer startup behavior for concurrent connections
+- stable error codes and structured exception details
+
+## Recommended Usage Level
 
-### 1. Create a manager with explicit brokers
+This package is now suitable as a shared internal Kafka module for SaaS microservices.
+
+Recommended use:
+
+- internal platform package for Node.js services
+- event publishing with standard envelopes
+- managed consumers with DLQ handling
+- consistent service-layer success and error responses
+
+Still recommended outside the package:
+
+- automated unit and integration tests in the consuming service or platform repo
+- schema validation for business payloads when required by your organization
+- organization-specific tracing, alerting, and compliance rules
+
+## Basic Usage
+
+### 1. Import the package
 
 ```ts
 import { KafkaManager } from 'sentinel-kafka-manager'
+```
 
+If you want to catch package-specific errors explicitly:
+
+```ts
+import {
+  KafkaErrorCode,
+  KafkaManager,
+  KafkaManagerError,
+  OperationResponse,
+} from 'sentinel-kafka-manager'
+```
+
+### 2. Create a manager
+
+You can create it in two ways:
+
+1. pass brokers directly
+2. build brokers from environment variables
+
+### Direct broker example
+
+```ts
 const kafkaManager = new KafkaManager({
   clientId: 'claims-service',
   brokers: ['localhost:29092', 'localhost:39092', 'localhost:49092'],
 })
 ```
 
-### 2. Publish a message
+### Environment-based example
+
+```ts
+const kafkaManager = KafkaManager.fromEnv({
+  clientId: 'claims-service',
+  mode: 'external',
+})
+```
+
+## Recommended Project Pattern
+
+In a real service, create one shared manager instance and reuse it.
+
+Example:
+
+```ts
+import { KafkaManager } from 'sentinel-kafka-manager'
+
+export const kafkaManager = KafkaManager.fromEnv({
+  clientId: 'claims-service',
+  mode: 'external',
+})
+```
+
+Then import that shared instance anywhere you need Kafka access.
+
+You can also attach enterprise hooks:
+
+```ts
+import { KafkaManager } from 'sentinel-kafka-manager'
+
+export const kafkaManager = KafkaManager.fromEnv({
+  clientId: 'claims-service',
+  mode: 'external',
+  logger: {
+    debug: (message, meta) => console.debug(message, meta),
+    info: (message, meta) => console.info(message, meta),
+    warn: (message, meta) => console.warn(message, meta),
+    error: (message, meta) => console.error(message, meta),
+  },
+  metrics: {
+    emit: (event) => {
+      console.log('METRIC', event)
+    },
+  },
+})
+```
+
+## How To Use In Your Service
+
+### Publish a message
 
 ```ts
 await kafkaManager.publish({
@@ -60,11 +458,206 @@ await kafkaManager.publish({
   payload: {
     claimId: 'claim-1001',
     status: 'created',
+    source: 'claims-service',
+  },
+})
+```
+
+`publish()` automatically:
+
+- gets a shared producer
+- connects it once
+- serializes `payload` with `JSON.stringify()`
+
+### Publish an enterprise event envelope
+
+Use this when you want traceable event metadata such as `eventType`, `version`, `traceId`, or `tenantId`.
+
+```ts
+await kafkaManager.publishEnvelope({
+  topic: 'claims.created',
+  key: 'claim-1001',
+  envelope: {
+    eventId: 'evt-claim-1001',
+    eventType: 'claims.created',
+    version: '1.0.0',
+    timestamp: new Date().toISOString(),
+    source: 'claims-service',
+    traceId: 'trace-123',
+    tenantId: 'tenant-abc',
+    correlationId: 'corr-001',
+    payload: {
+      claimId: 'claim-1001',
+      status: 'created',
+    },
   },
 })
 ```
 
-### 3. Provision topics
+### Create or reuse a producer manually
+
+If you need direct producer access:
+
+```ts
+const producer = await kafkaManager.getProducer()
+
+await producer.send({
+  topic: 'claims.created',
+  messages: [
+    {
+      key: 'claim-1001',
+      value: JSON.stringify({ claimId: 'claim-1001' }),
+    },
+  ],
+})
+```
+
+### Create or reuse a consumer
+
+```ts
+const consumer = await kafkaManager.getConsumer('claims-service-group')
+
+await consumer.subscribe({
+  topic: 'claims.created',
+  fromBeginning: false,
+})
+
+await consumer.run({
+  eachMessage: async ({ topic, partition, message }) => {
+    const rawValue = message.value?.toString() ?? '{}'
+    const payload = JSON.parse(rawValue)
+
+    console.log({
+      topic,
+      partition,
+      key: message.key?.toString(),
+      payload,
+    })
+  },
+})
+```
+
+`getConsumer(groupId)` returns one shared connected consumer per group id.
+
+### Run a managed consumer with DLQ support
+
+For SaaS-style services, this is the recommended consumer pattern.
+
+```ts
+await kafkaManager.runConsumer({
+  groupId: 'claims-service-group',
+  topic: 'claims.created',
+  dlqTopic: 'claims.created.dlq',
+  onMessage: async ({ message, headers, key }) => {
+    console.log('Processing:', {
+      key,
+      traceId: headers['x-trace-id'],
+      payload: message,
+    })
+  },
+  onError: async (error, context) => {
+    console.error('Consumer error:', {
+      error: error.message,
+      topic: context.topic,
+      offset: context.offset,
+    })
+  },
+})
+```
+
+If processing fails:
+
+- the error is logged
+- a metric hook can receive the failure event
+- the message can be published to a dead-letter topic when `dlqTopic` is set
+
+### Catch structured errors
+
+All package-thrown operational errors now use `KafkaManagerError`.
+
+```ts
+import {
+  KafkaErrorCode,
+  KafkaManagerError,
+} from 'sentinel-kafka-manager'
+
+try {
+  await kafkaManager.publish({
+    topic: 'claims.created',
+    payload: { claimId: 'claim-1001' },
+  })
+} catch (error) {
+  if (error instanceof KafkaManagerError) {
+    console.error('Kafka error', {
+      code: error.code,
+      message: error.message,
+      details: error.details,
+    })
+
+    if (error.code === KafkaErrorCode.MESSAGE_PUBLISH_FAILED) {
+      // retry, alert, or degrade gracefully
+    }
+  }
+
+  throw error
+}
+```
+
+### Use common success and error response types
+
+If your service wraps Kafka operations in API or service-layer responses, you can use the shared response contracts from this package.
+
+```ts
+import {
+  ErrorResponse,
+  OperationResponse,
+  SuccessResponse,
+} from 'sentinel-kafka-manager'
+
+type PublishClaimCreatedResponse = OperationResponse<{
+  topic: string
+  key: string
+}>
+```
+
+Success example:
+
+```ts
+const response: SuccessResponse<{ topic: string; key: string }> = {
+  success: true,
+  message: 'Kafka message published successfully.',
+  data: {
+    topic: 'claims.created',
+    key: 'claim-1001',
+  },
+  meta: {
+    timestamp: new Date().toISOString(),
+    traceId: 'trace-123',
+  },
+}
+```
+
+Error example:
+
+```ts
+const response: ErrorResponse = {
+  success: false,
+  message: 'Failed to publish Kafka message.',
+  error: {
+    code: KafkaErrorCode.MESSAGE_PUBLISH_FAILED,
+    details: {
+      topic: 'claims.created',
+      key: 'claim-1001',
+    },
+  },
+  meta: {
+    timestamp: new Date().toISOString(),
+    retryable: true,
+  },
+}
+```
+
+### Provision topics
 
 ```ts
 await kafkaManager.provisionTopics([
@@ -73,54 +666,74 @@ await kafkaManager.provisionTopics([
     numPartitions: 6,
     replicationFactor: 3,
   },
+  {
+    topic: 'claims.updated',
+    numPartitions: 6,
+    replicationFactor: 3,
+  },
 ])
 ```
 
-### 4. Reuse a connected consumer
+This is useful during service startup when your topics should exist before producers or consumers begin work.
+
+### Disconnect on shutdown
+
+Always close Kafka clients during application shutdown.
 
 ```ts
-const consumer = await kafkaManager.getConsumer('claims-service-group')
+process.on('SIGINT', async () => {
+  await kafkaManager.disconnect()
+  process.exit(0)
+})
 
-await consumer.subscribe({
-  topic: 'claims.created',
-  fromBeginning: false,
+process.on('SIGTERM', async () => {
+  await kafkaManager.disconnect()
+  process.exit(0)
 })
 ```
 
-### 5. Disconnect on shutdown
+### Inspect health
 
 ```ts
-await kafkaManager.disconnect()
+const health = kafkaManager.getHealthSnapshot()
+
+console.log(health)
 ```
 
-## Environment-Based Setup
+## Environment Setup
 
-`KafkaManager.fromEnv()` supports two styles:
+This package supports two environment styles:
 
-1. explicit broker list with `KAFKA_BROKERS`
-2. derived broker list from your Kafka runtime env naming
+1. one explicit broker list with `KAFKA_BROKERS`
+2. derived brokers from your existing Kafka runtime variables
 
-### Option A: Explicit broker list
+## Option 1: Use `KAFKA_BROKERS`
+
+This is the simplest option.
+
+### `.env`
 
 ```env
 KAFKA_BROKERS=localhost:29092,localhost:39092,localhost:49092
 ```
 
-```ts
-import { KafkaManager } from 'sentinel-kafka-manager'
+### Code
 
+```ts
 const kafkaManager = KafkaManager.fromEnv({
   clientId: 'claims-service',
 })
 ```
 
-### Option B: Derived from your current Kafka cluster env
+## Option 2: Use your current Kafka cluster variables
+
+This matches the Kafka server setup you shared.
 
-This package supports the env structure you shared for your KRaft cluster.
+### For host machine apps
 
-#### External mode
+Use this when your Node.js service runs on the host machine.
 
-Use this when your app runs on the host machine or outside the Kafka Docker network.
+### `.env`
 
 ```env
 KAFKA_EXTERNAL_HOST=localhost
@@ -130,6 +743,8 @@ KAFKA_BROKER_3_EXTERNAL_PORT=49092
 KAFKA_BROKER_INTERNAL_PORT=9092
 ```
 
+### Code
+
 ```ts
 const kafkaManager = KafkaManager.fromEnv({
   clientId: 'claims-service',
@@ -145,14 +760,18 @@ localhost:39092
 localhost:49092
 ```
 
-#### Internal mode
+### For Dockerized apps on the Kafka network
 
-Use this when your app runs inside Docker on the same Kafka network.
+Use this when your Node.js service runs inside Docker on the same Kafka network.
+
+### `.env`
 
 ```env
 KAFKA_BROKER_INTERNAL_PORT=9092
 ```
 
+### Code
+
 ```ts
 const kafkaManager = KafkaManager.fromEnv({
   clientId: 'claims-service',
@@ -168,97 +787,144 @@ broker-2:9092
 broker-3:9092
 ```
 
-## Supported Environment Variables
+## Which Mode Should You Use?
+
+- Use `mode: 'external'` when your service uses `localhost` broker ports like `29092`, `39092`, and `49092`
+- Use `mode: 'internal'` when your service is inside Docker and should reach brokers like `broker-1:9092`
+- Use `KAFKA_BROKERS` when you want the simplest and most explicit configuration
+
+## Your Current Running Kafka Cluster
 
-### Minimal variables
+Based on your running Docker containers, your Kafka cluster is exposed like this:
 
-- `KAFKA_BROKERS`
-- `KAFKA_EXTERNAL_HOST`
-- `KAFKA_BROKER_1_EXTERNAL_PORT`
-- `KAFKA_BROKER_2_EXTERNAL_PORT`
-- `KAFKA_BROKER_3_EXTERNAL_PORT`
-- `KAFKA_BROKER_INTERNAL_PORT`
+- `broker-1` -> host port `29092`
+- `broker-2` -> host port `39092`
+- `broker-3` -> host port `49092`
+- `controller-1`, `controller-2`, and `controller-3` are controller-only nodes and should not be used by application clients
 
-### `fromEnv()` options
+If your Node.js service runs on the host machine, use:
 
-You can customize the env lookup behavior:
+```env
+KAFKA_EXTERNAL_HOST=localhost
+KAFKA_BROKER_1_EXTERNAL_PORT=29092
+KAFKA_BROKER_2_EXTERNAL_PORT=39092
+KAFKA_BROKER_3_EXTERNAL_PORT=49092
+KAFKA_BROKER_INTERNAL_PORT=9092
+```
 
 ```ts
 const kafkaManager = KafkaManager.fromEnv({
   clientId: 'claims-service',
   mode: 'external',
-  brokerCount: 3,
-  envKey: 'KAFKA_BROKERS',
-  externalHostEnvKey: 'KAFKA_EXTERNAL_HOST',
-  internalPortEnvKey: 'KAFKA_BROKER_INTERNAL_PORT',
-  externalPortEnvKeyPrefix: 'KAFKA_BROKER_',
-  externalPortEnvKeySuffix: '_EXTERNAL_PORT',
-  internalBrokerHostPrefix: 'broker-',
 })
 ```
 
-## Advanced Connection Options
-
-### SSL
+If your Node.js service runs inside Docker on the same Kafka network, use:
 
 ```ts
-const kafkaManager = new KafkaManager({
+const kafkaManager = KafkaManager.fromEnv({
   clientId: 'claims-service',
-  brokers: ['localhost:29092'],
-  ssl: {
-    rejectUnauthorized: true,
-  },
+  mode: 'internal',
 })
 ```
 
-### SASL
+This package is designed to work with that exact broker layout.
+
+## Full Service Example
 
 ```ts
-const kafkaManager = new KafkaManager({
-  clientId: 'claims-service',
-  brokers: ['localhost:29092'],
-  sasl: {
-    mechanism: 'plain',
-    username: 'kafka-user',
-    password: 'secret',
-  },
+import { KafkaManager } from 'sentinel-kafka-manager'
+
+const kafkaManager = KafkaManager.fromEnv({
+  clientId: 'claims-api',
+  mode: 'external',
+})
+
+async function bootstrap(): Promise<void> {
+  await kafkaManager.provisionTopics([
+    {
+      topic: 'claims.created',
+      numPartitions: 6,
+      replicationFactor: 3,
+    },
+  ])
+
+  await kafkaManager.publish({
+    topic: 'claims.created',
+    key: 'claim-1001',
+    payload: {
+      claimId: 'claim-1001',
+      source: 'claims-api',
+    },
+  })
+
+  await kafkaManager.runConsumer({
+    groupId: 'claims-api-group',
+    topic: 'claims.created',
+    dlqTopic: 'claims.created.dlq',
+    onMessage: async ({ message, headers }) => {
+      console.log('Received event:', {
+        traceId: headers['x-trace-id'],
+        payload: message,
+      })
+    },
+  })
+}
+
+bootstrap().catch(async (error) => {
+  console.error('Kafka bootstrap failed:', error)
+  await kafkaManager.disconnect()
+  process.exit(1)
+})
+
+process.on('SIGINT', async () => {
+  await kafkaManager.disconnect()
+  process.exit(0)
+})
+
+process.on('SIGTERM', async () => {
+  await kafkaManager.disconnect()
+  process.exit(0)
 })
 ```
 
-### Retry and timeout tuning
+## API Summary
+
+### `new KafkaManager(options)`
+
+Use this when you already know your brokers.
 
 ```ts
 const kafkaManager = new KafkaManager({
   clientId: 'claims-service',
   brokers: ['localhost:29092'],
-  connectionTimeout: 5000,
-  requestTimeout: 30000,
-  retry: {
-    initialRetryTime: 300,
-    retries: 10,
-  },
 })
 ```
 
-## API
-
-### `new KafkaManager(options)`
-
-Creates a manager from an explicit broker list.
-
 Options:
 
 - `clientId: string`
 - `brokers: string[]`
 - `ssl?: boolean | { rejectUnauthorized?: boolean }`
-- `sasl?: { mechanism, username, password }`
+- `sasl?: SASLOptions`
 - `connectionTimeout?: number`
 - `requestTimeout?: number`
 - `retry?: { initialRetryTime?: number; retries?: number }`
+- `logger?: KafkaLogger`
+- `metrics?: KafkaMetrics`
+- `producerConfig?: ProducerConfig`
+- `consumerDefaults?: Omit<ConsumerConfig, 'groupId'>`
 
 ### `KafkaManager.fromEnv(options)`
 
-Creates a manager by reading broker information from environment variables.
+Use this when you want brokers to come from environment variables.
+
+```ts
+const kafkaManager = KafkaManager.fromEnv({
+  clientId: 'claims-service',
+  mode: 'external',
+})
+```
 
 Options:
 
@@ -272,10 +938,14 @@ Options:
 - `externalPortEnvKeySuffix?: string`
 - `internalBrokerHostPrefix?: string`
 - `ssl?: boolean | { rejectUnauthorized?: boolean }`
-- `sasl?: { mechanism, username, password }`
+- `sasl?: SASLOptions`
 - `connectionTimeout?: number`
 - `requestTimeout?: number`
 - `retry?: { initialRetryTime?: number; retries?: number }`
+- `logger?: KafkaLogger`
+- `metrics?: KafkaMetrics`
+- `producerConfig?: ProducerConfig`
+- `consumerDefaults?: Omit<ConsumerConfig, 'groupId'>`
 
 ### `getProducer()`
 
@@ -283,109 +953,257 @@ Returns a shared connected Kafka producer.
 
 ### `getConsumer(groupId)`
 
-Returns a shared connected Kafka consumer for the provided group id.
-
-### `provisionTopics(topics)`
+Returns a shared connected Kafka consumer for that `groupId`.
 
-Creates missing topics using the Kafka admin client.
+### `publish(options)`
 
-Topic shape:
+Publishes one JSON message.
 
 ```ts
-{
-  topic: string
-  numPartitions?: number
-  replicationFactor?: number
-}
+await kafkaManager.publish({
+  topic: 'claims.created',
+  key: 'claim-1001',
+  payload: { claimId: 'claim-1001' },
+})
 ```
 
-### `publish(options)`
+### `publishEnvelope(options)`
 
-Publishes one JSON-encoded message.
+Publishes a standard metadata envelope for traceable event-driven systems.
 
-Message shape:
+### `provisionTopics(topics)`
 
-```ts
-{
-  topic: string
-  payload: T
-  key?: string
-  headers?: IHeaders
-}
-```
+Creates missing topics.
+
+### `runConsumer(options)`
+
+Runs a managed consumer with:
+
+- a typed message parser
+- application callback handling
+- optional DLQ publishing
+- optional error callback
+
+### `getHealthSnapshot()`
+
+Returns a lightweight connection-health snapshot.
 
 ### `disconnect()`
 
-Disconnects the managed producer, admin client, and any connected consumers.
+Disconnects the producer, admin client, and any connected consumers.
+
+## Error Handling
+
+This package now throws structured errors with stable codes for production handling.
+
+### Error class
+
+- `KafkaManagerError`
+
+### Common response types
+
+- `SuccessResponse<T>`
+- `ErrorResponse`
+- `OperationResponse<T>`
+
+### Error fields
 
-## Example With Your Kafka Cluster
+- `code`: stable machine-readable error code
+- `message`: human-readable explanation
+- `details`: structured metadata for logs or monitoring
+- `cause`: original thrown error when available
 
-### Host-based app
+### Common error codes
+
+- `INVALID_CLIENT_ID`
+- `INVALID_BROKERS`
+- `INVALID_GROUP_ID`
+- `INVALID_TOPIC`
+- `INVALID_ENVELOPE`
+- `INVALID_BROKER_COUNT`
+- `MISSING_ENVIRONMENT_VARIABLE`
+- `PRODUCER_CONNECTION_FAILED`
+- `CONSUMER_CONNECTION_FAILED`
+- `ADMIN_CONNECTION_FAILED`
+- `MESSAGE_PUBLISH_FAILED`
+- `TOPIC_PROVISION_FAILED`
+- `CONSUMER_RUN_FAILED`
+- `CONSUMER_HANDLER_FAILED`
+- `DLQ_PUBLISH_FAILED`
+- `MESSAGE_PARSE_FAILED`
+- `DISCONNECT_FAILED`
+
+### Example
 
 ```ts
-import { KafkaManager } from 'sentinel-kafka-manager'
+import { KafkaManagerError } from 'sentinel-kafka-manager'
+
+try {
+  await kafkaManager.runConsumer({
+    groupId: 'claims-service-group',
+    topic: 'claims.created',
+    onMessage: async ({ message }) => {
+      console.log(message)
+    },
+  })
+} catch (error) {
+  if (error instanceof KafkaManagerError) {
+    console.error(error.code, error.message, error.details)
+  }
+
+  throw error
+}
+```
 
+## Advanced Examples
+
+### Logger and metrics example
+
+```ts
 const kafkaManager = KafkaManager.fromEnv({
-  clientId: 'claims-api',
+  clientId: 'claims-service',
   mode: 'external',
+  logger: {
+    debug: (message, meta) => console.debug(message, meta),
+    info: (message, meta) => console.info(message, meta),
+    warn: (message, meta) => console.warn(message, meta),
+    error: (message, meta) => console.error(message, meta),
+  },
+  metrics: {
+    emit: (event) => {
+      console.log('METRIC', event.name, event.tags, event.meta)
+    },
+  },
 })
+```
 
-await kafkaManager.publish({
+### SSL example
+
+```ts
+const kafkaManager = new KafkaManager({
+  clientId: 'claims-service',
+  brokers: ['localhost:29092'],
+  ssl: {
+    rejectUnauthorized: true,
+  },
+})
+```
+
+### Managed consumer with custom parser
+
+```ts
+await kafkaManager.runConsumer({
+  groupId: 'claims-service-group',
   topic: 'claims.created',
-  key: 'claim-1001',
-  payload: {
-    claimId: 'claim-1001',
-    source: 'claims-api',
+  dlqTopic: 'claims.created.dlq',
+  parser: (payload) => {
+    const rawValue = payload.message.value?.toString() ?? '{}'
+    return JSON.parse(rawValue) as {
+      claimId: string
+      status: string
+    }
+  },
+  onMessage: async ({ message }) => {
+    console.log(message.claimId, message.status)
   },
 })
 ```
 
-### Dockerized app on the same network
+### SASL example
 
 ```ts
-import { KafkaManager } from 'sentinel-kafka-manager'
+const kafkaManager = new KafkaManager({
+  clientId: 'claims-service',
+  brokers: ['localhost:29092'],
+  sasl: {
+    mechanism: 'plain',
+    username: 'kafka-user',
+    password: 'secret',
+  },
+})
+```
 
-const kafkaManager = KafkaManager.fromEnv({
-  clientId: 'claims-worker',
-  mode: 'internal',
+### Retry and timeout example
+
+```ts
+const kafkaManager = new KafkaManager({
+  clientId: 'claims-service',
+  brokers: ['localhost:29092'],
+  connectionTimeout: 5000,
+  requestTimeout: 30000,
+  retry: {
+    initialRetryTime: 300,
+    retries: 10,
+  },
 })
 ```
 
-## Publishing
+## Common Mistakes
+
+- Do not connect application clients to KRaft controller nodes. Connect only to brokers.
+- Do not use `mode: 'external'` inside Docker unless you truly want host-published ports.
+- Do not create a new `KafkaManager` for every request. Reuse one shared instance.
+- Do not forget `disconnect()` during shutdown.
+- Do not assume topics will exist unless your platform creates them or your service provisions them.
+- Do not ignore failed consumer messages in production. Use `runConsumer()` with a `dlqTopic`.
+- Do not publish business-critical events without metadata. Prefer `publishEnvelope()` for traceability.
+- Do not depend on raw driver error strings in application logic. Use `KafkaManagerError.code`.
+
+## Recommended SaaS Pattern
+
+For most SaaS services, the cleanest pattern is:
 
-Build before publishing:
+1. Create one shared `KafkaManager` during service bootstrap.
+2. Use `publishEnvelope()` for business events.
+3. Use `runConsumer()` instead of raw `consumer.run()` for managed handling.
+4. Configure a `dlqTopic` for important consumers.
+5. Catch `KafkaManagerError` and branch on `error.code`.
+6. Wrap service outcomes in `OperationResponse<T>` where useful.
+
+## Publishing This Package
+
+Build:
 
 ```bash
 npm run build
 ```
 
-Publish the package:
+Publish:
 
 ```bash
 npm publish
 ```
 
-This package already sets `"publishConfig": { "access": "public" }`, so you do not need to pass `--access public` every time.
-
-### Publish Troubleshooting
+If npm rejects the publish:
 
-#### `403` with 2FA or token message
+- `403` usually means authentication, 2FA, or token permission issues
+- `404` usually means the package name is unavailable or not publishable by your account
 
-If npm says two-factor authentication or a granular token with bypass 2FA is required, authenticate with an account that can publish or use a publish-capable npm token.
-
-Useful checks:
+Useful check:
 
 ```bash
 npm whoami
 ```
 
-#### `404` or package name unavailable
+## Notes
+
+- Your current Kafka cluster works fine with PLAINTEXT, so SSL and SASL are optional unless your environment changes
+- This package is designed for both host-based services and Dockerized services
+- If you want the least surprise, use `KAFKA_BROKERS`
 
-If npm says the package could not be published, confirm that the package name is available to your npm account and not already owned by someone else.
+## Links
 
-## Notes
+- npm deployment guide: [NPM_DEPLOYMENT.md](/var/www/html/matrix-project/sentinel-kafka-manager/NPM_DEPLOYMENT.md)
+- Example folder: [examples/service-module-usage/README.md](/var/www/html/matrix-project/sentinel-kafka-manager/examples/service-module-usage/README.md)
+- Service module guide: [SERVICE_MODULE_USAGE.md](/var/www/html/matrix-project/sentinel-kafka-manager/SERVICE_MODULE_USAGE.md)
+- GitHub: https://github.com/dilipshaw2024/sentinel-kafka-manager
+- npm: https://www.npmjs.com/package/sentinel-kafka-manager
+- LinkedIn: https://www.linkedin.com/in/dilip-shaw-2740769/
+
+## About Me
+
+I'm a full stack developer. Experienced developer with over 20 years of expertise in crafting scalable web applications. Proficient in frontend technologies such as Angular and React, alongside extensive experience in WordPress, Drupal, and backend frameworks like Node.js. With a proven track record of delivering high-quality solutions to meet client needs and drive business objectives, I bring a versatile skill set and a commitment to excellence to every project.
+
+## Feedback
 
-- connect clients only to brokers, not KRaft controller nodes
-- for your current cluster, PLAINTEXT is enough and SSL/SASL is optional
-- if your services use `localhost`, use `mode: 'external'`
-- if your services run in Docker on the Kafka network, use `mode: 'internal'`
+If you have any feedback, please reach out to us at `dilipabc@gmail.com`