@autofleet/kafka 0.0.5 → 0.1.1

package/README.md

@@ -1,6 +1,6 @@
 # @autofleet/kafka
 
-Internal wrapper for Apache Kafka producer using [@platformatic/kafka](https://www.npmjs.com/package/@platformatic/kafka), providing a production-ready interface for publishing messages to Kafka topics.
+Internal wrapper for Apache Kafka producer using [@platformatic/kafka](https://www.npmjs.com/package/@platformatic/kafka), providing a production-ready interface for managing multiple Kafka producers.
 
 ## Table of Contents
 
@@ -8,9 +8,11 @@ Internal wrapper for Apache Kafka producer using [@platformatic/kafka](https://w
 - [Features](#features)
 - [Quick Start](#quick-start)
 - [Usage](#usage)
+  - [Setup with Multiple Producers](#setup-with-multiple-producers)
   - [Publishing Messages](#publishing-messages)
-  - [Batch Publishing](#batch-publishing)
-  - [Managing Connection](#managing-connection)
+  - [Mock Mode (Disable Kafka)](#mock-mode-disable-kafka)
+  - [Health Checks & Readiness Probes](#health-checks--readiness-probes)
+  - [Migration from getKafka() Pattern](#migration-from-getkafka-pattern)
 - [Configuration](#configuration)
 - [Advanced Features](#advanced-features)
   - [Message Keys for Partitioning](#message-keys-for-partitioning)
@@ -28,172 +30,314 @@ pnpm add @autofleet/kafka
 
 ## Features
 
-- **Simple Producer API** - Easy-to-use interface for publishing messages
+- **Multi-Producer Management** - Manage multiple named producers with different broker configurations
+- **Type-Safe Producer Names** - Producer names are typed based on your configuration with full autocomplete
+- **Built-in Mock Mode** - Disable Kafka entirely with automatic mock implementations
+- **Direct API Access** - No need for `getKafka()` wrappers, access producers directly
+- **Centralized Health Checks** - Single readiness check for all producers
 - **Automatic Connection Management** - Handles connection lifecycle automatically
 - **Batch Publishing** - Efficient batch message sending
 - **Message Partitioning** - Control message distribution across partitions
 - **Custom Headers** - Attach metadata to messages
 - **Graceful Shutdown** - Automatic cleanup on process termination
-- **Type Safety** - Full TypeScript support
-- **ESM & CommonJS Compatible** - Works in both ESM and CommonJS projects via dynamic imports
+- **Full TypeScript Support** - Type-safe APIs with IntelliSense autocomplete
+- **ESM & CommonJS Compatible** - Works in both ESM and CommonJS projects
 - **Production Ready** - Built with reliability and performance in mind
 
 ## Quick Start
 
 ```typescript
-import AfKafka from '@autofleet/kafka';
-
-// Initialize the producer (async factory)
-const kafka = await AfKafka.create({
-  brokers: ['localhost:9092'],
-  clientId: 'my-service',
+import { KafkaManager } from '@autofleet/kafka';
+import logger from './logger';
+
+// Initialize with multiple named producers (synchronous!)
+const kafka = KafkaManager.create({
+  enabled: process.env.ENABLE_KAFKA === 'true', // Built-in mock mode
+  logger,
+  producers: {
+    main: {
+      brokers: ['kafka-main.svc.cluster.local:9092'],
+      clientId: 'my-service-main',
+    },
+    analytics: {
+      brokers: ['kafka-analytics.svc.cluster.local:9092'],
+      clientId: 'my-service-analytics',
+    },
+  },
 });
 
-// Publish a message
-await kafka.publish('user-events', {
+// Publish directly - no getKafka() needed!
+// Producers initialize automatically on first publish
+// TypeScript knows 'main' and 'analytics' are the only valid producer names!
+await kafka.publish('main', 'user-events', {
   userId: '123',
   action: 'user.created',
   timestamp: Date.now(),
 });
 
-// Publish multiple messages in a batch
-await kafka.publishBatch({
-  topic: 'user-events',
-  messages: [
-    { value: { userId: '123', action: 'login' } },
-    { value: { userId: '456', action: 'logout' } },
-  ],
+// Publish to different producer - autocomplete suggests 'main' | 'analytics'
+await kafka.publish('analytics', 'metrics', {
+  metric: 'user.signup',
+  value: 1,
 });
 
-// Disconnect when done
-await kafka.disconnect();
+// TypeScript error: Producer 'invalid' doesn't exist!
+// await kafka.publish('invalid', 'topic', {}); // ❌ Type error
+
+// Use in health checks
+app.get('/health/ready', async (req, res) => {
+  const ready = await kafka.isReady();
+  res.status(ready ? 200 : 503).json({ ready });
+});
 ```
 
 ## Usage
 
-### Publishing Messages
+### Setup with Multiple Producers
 
-#### Basic Publishing
+Create a `kafka.ts` file in your service:
 
 ```typescript
-const kafka = await AfKafka.create({
-  brokers: ['localhost:9092', 'localhost:9093'],
-  clientId: 'user-service',
-});
-
-// Publish a simple message
-await kafka.publish('user-events', {
-  userId: '123',
-  event: 'profile_updated',
-  data: {
-    name: 'John Doe',
-    email: 'john@example.com',
+import { KafkaManager } from '@autofleet/kafka';
+import logger from './logger';
+
+const ENABLE_KAFKA = process.env.ENABLE_KAFKA === 'true';
+
+// No await needed - initialization is synchronous!
+// Producers connect lazily on first publish
+export const kafka = KafkaManager.create({
+  enabled: ENABLE_KAFKA,
+  logger,
+  producers: {
+    // Primary producer for main events
+    main: {
+      brokers: ['kafka-main.svc.cluster.local:9092'],
+      clientId: 'driver-ms-main',
+      autoCreateTopics: true,
+    },
+    // Secondary producer for analytics
+    analytics: {
+      brokers: ['kafka-analytics.svc.cluster.local:9092'],
+      clientId: 'driver-ms-analytics',
+    },
   },
 });
+
+// Define your topics
+export const TOPICS = {
+  DRIVER_CONSENT_V1: 'backend.driver.consent.v1',
+  DRIVER_ASSIGNED_V1: 'backend.driver.vehicle.assigned.v1',
+  DRIVER_UNASSIGNED_V1: 'backend.driver.vehicle.unassigned.v1',
+} as const;
 ```
 
-#### Publishing with Message Key
+### Publishing Messages
 
-Message keys determine which partition a message goes to, ensuring messages with the same key are always sent to the same partition (maintaining order).
+Now you can use it directly throughout your service with full type safety:
 
 ```typescript
-await kafka.publish(
-  'user-events',
-  { userId: '123', action: 'update' },
-  {
-    key: 'user-123', // All messages with this key go to the same partition
-  }
-);
-```
-
-#### Publishing with Custom Headers
+import { kafka, TOPICS } from './kafka';
+
+// Publish directly - no getKafka() overhead!
+// TypeScript validates producer names: 'main' | 'analytics'
+await kafka.publish('main', TOPICS.DRIVER_CONSENT_V1, {
+  state: 'accepted',
+  driverId: '123',
+  fleetId: '456',
+});
 
-```typescript
+// With options - autocomplete works!
 await kafka.publish(
-  'user-events',
-  { userId: '123', action: 'login' },
+  'main', // TypeScript autocompletes 'main' | 'analytics'
+  TOPICS.DRIVER_ASSIGNED_V1,
+  {
+    driverId: '123',
+    vehicleId: '789',
+  },
   {
+    key: `driver-123`, // Ensures ordering per driver
     headers: {
       'correlation-id': requestId,
-      'source': 'api-gateway',
-      'version': '1.0.0',
     },
   }
 );
+
+// TypeScript will error if you use a non-existent producer
+// await kafka.publish('wrong', TOPICS.DRIVER_CONSENT_V1, {}); // ❌ Error!
 ```
 
-#### Publishing to Specific Partition
+### Batch Publishing
 
 ```typescript
-await kafka.publish(
-  'user-events',
-  { userId: '123', action: 'update' },
-  {
-    partition: 2, // Send directly to partition 2
-  }
-);
+await kafka.publishBatch('main', {
+  topic: TOPICS.DRIVER_CONSENT_V1,
+  messages: [
+    { value: { driverId: '123', state: 'accepted' } },
+    { value: { driverId: '456', state: 'rejected' } },
+    { value: { driverId: '789', state: 'pending' } },
+  ],
+});
 ```
 
-### Batch Publishing
+### Mock Mode (Disable Kafka)
 
-Batch publishing is more efficient for sending multiple messages:
+When `enabled: false`, all producers become mocks automatically:
 
 ```typescript
-await kafka.publishBatch({
-  topic: 'user-events',
-  messages: [
-    {
-      value: { userId: '123', action: 'login' },
-      key: 'user-123',
-    },
-    {
-      value: { userId: '456', action: 'logout' },
-      key: 'user-456',
+const kafka = KafkaManager.create({
+  enabled: false, // or process.env.ENABLE_KAFKA !== 'true'
+  logger,
+  producers: {
+    main: {
+      brokers: ['kafka:9092'],
+      clientId: 'my-service',
     },
-    {
-      value: { userId: '789', action: 'update' },
-      key: 'user-789',
-      headers: { priority: 'high' },
-    },
-  ],
+  },
 });
+
+// This will log a debug message but not actually publish
+await kafka.publish('main', 'topic', { data: 'test' });
+// Output: "Kafka: [main] Mock mode - skipping publish to topic: topic"
 ```
 
-### Managing Connection
+### Health Checks & Readiness Probes
+
+Use the built-in health check for Kubernetes readiness probes:
 
 ```typescript
-// The producer connects automatically on first publish
-await kafka.publish('my-topic', { data: 'value' });
+import { kafka } from './kafka';
+
+// Express/Fastify/etc
+app.get('/health/ready', async (req, res) => {
+  const ready = await kafka.isReady();
+  res.status(ready ? 200 : 503).json({
+    ready,
+    kafka: kafka.getConnectionStatus(),
+  });
+});
 
-// Disconnect when you're done
-await kafka.disconnect();
+// Or manually check each producer
+app.get('/health/detailed', async (req, res) => {
+  const status = kafka.getConnectionStatus();
+  const allConnected = Object.values(status).every(s => s);
 
-// Graceful shutdown is automatic on SIGTERM/SIGINT
-// unless dontGracefulShutdown is set to true
+  res.status(allConnected ? 200 : 503).json({
+    producers: status,
+    enabled: kafka.isEnabled,
+  });
+});
+```
+
+### Migration from getKafka() Pattern
+
+**Before (problematic pattern):**
+
+```typescript
+// kafka.ts
+let kafkaInstance: AfKafka | null = null;
+const ENABLE_KAFKA = process.env.ENABLE_KAFKA === 'true';
+
+const disabledKafkaMock: Partial<AfKafka> = {
+  ping: async () => { logger.info('Kafka disabled'); },
+  publish: async (topic, message) => {
+    logger.debug('Skipping publish', { topic, message });
+    return [];
+  },
+};
+
+async function getKafka(): Promise<AfKafka> {
+  if (!ENABLE_KAFKA) {
+    return disabledKafkaMock as AfKafka;
+  }
+
+  if (!kafkaInstance) {
+    const { default: Kafka } = await import('@autofleet/kafka');
+    kafkaInstance = await Kafka.create({
+      brokers: ['kafka:9092'],
+      clientId: 'my-service',
+    });
+  }
+  return kafkaInstance;
+}
+
+// Usage - awkward!
+async function publishEvent() {
+  const kafka = await getKafka(); // Overhead on every call
+  await kafka.publish('topic', data);
+}
+```
+
+**After (clean pattern):**
+
+```typescript
+// kafka.ts
+import { KafkaManager } from '@autofleet/kafka';
+import logger from './logger';
+
+// Synchronous initialization - no await!
+export const kafka = KafkaManager.create({
+  enabled: process.env.ENABLE_KAFKA === 'true', // Built-in mock mode!
+  logger,
+  producers: {
+    main: {
+      brokers: ['kafka:9092'],
+      clientId: 'my-service',
+    },
+  },
+});
+
+export const TOPICS = {
+  MY_TOPIC: 'my.topic',
+} as const;
+
+// Usage - clean and direct!
+async function publishEvent() {
+  await kafka.publish('main', TOPICS.MY_TOPIC, data); // Direct access!
+}
 ```
 
 ## Configuration
 
-### KafkaOptions
+### KafkaManagerOptions
+
+```typescript
+interface KafkaManagerOptions {
+  // Enable/disable Kafka - when false, returns mock implementations
+  enabled?: boolean;
+
+  // Custom logger instance
+  logger?: LoggerInstanceManager;
+
+  // Skip automatic graceful shutdown (default: false)
+  dontGracefulShutdown?: boolean;
+
+  // Named producers configuration
+  producers: Record<string, ProducerConfig>;
+}
+```
+
+### ProducerConfig
 
 ```typescript
-interface KafkaOptions {
+interface ProducerConfig {
   // Array of Kafka broker addresses (required)
   brokers: string[];
 
   // Client ID for this producer (optional)
   clientId?: string;
 
-  // Custom logger instance (optional)
-  logger?: LoggerInstanceManager;
+  // SASL authentication options (optional)
+  sasl?: {
+    mechanism: 'plain' | 'scram-sha-256' | 'scram-sha-512';
+    username: string;
+    password: string;
+  };
 
-  // Skip automatic graceful shutdown (default: false)
-  dontGracefulShutdown?: boolean;
+  // Automatically create topics if they don't exist (default: false)
+  autoCreateTopics?: boolean;
 }
 ```
 
-The package uses [@platformatic/kafka](https://www.npmjs.com/package/@platformatic/kafka) with `stringSerializers` for automatic serialization. The ESM-only `@platformatic/kafka` library is loaded via dynamic imports, ensuring compatibility with both ESM and CommonJS environments.
-
 ## Advanced Features
 
 ### Message Keys for Partitioning
@@ -202,12 +346,12 @@ Use message keys to control partitioning and maintain order:
 
 ```typescript
 // All messages for the same user go to the same partition
-await kafka.publish('user-events', event, {
+await kafka.publish('main', 'user-events', event, {
   key: `user-${userId}`,
 });
 
 // All messages for the same order go to the same partition
-await kafka.publish('order-events', event, {
+await kafka.publish('main', 'order-events', event, {
   key: `order-${orderId}`,
 });
 ```
@@ -222,7 +366,7 @@ await kafka.publish('order-events', event, {
 Headers are useful for metadata and message routing:
 
 ```typescript
-await kafka.publish('events', data, {
+await kafka.publish('main', 'events', data, {
   headers: {
     // Correlation ID for distributed tracing
     'correlation-id': correlationId,
@@ -233,9 +377,6 @@ await kafka.publish('events', data, {
     // Message schema version
     'schema-version': '2.0',
 
-    // Content encoding
-    'encoding': 'json',
-
     // Custom application headers
     'tenant-id': 'tenant-123',
   },
@@ -248,41 +389,60 @@ Direct partition control for advanced use cases:
 
 ```typescript
 // Send to specific partition
-await kafka.publish('events', data, {
+await kafka.publish('main', 'events', data, {
   partition: 0, // Always send to partition 0
 });
 
 // Balance across partitions using keys
 const partitionKey = `${customerId % 10}`;
-await kafka.publish('events', data, {
+await kafka.publish('main', 'events', data, {
   key: partitionKey,
 });
 ```
 
 ## API Reference
 
-### `AfKafka.create(options): Promise<AfKafka>`
+### `KafkaManager.create(options): KafkaManager<ProducerNames>`
 
-Creates a new Kafka producer instance using an async factory pattern. This method dynamically imports the ESM-only `@platformatic/kafka` library, making it compatible with both ESM and CommonJS environments.
+Creates a new KafkaManager instance with multiple named producers. Producers are initialized lazily on first use. **Producer names are type-safe** - TypeScript will autocomplete and validate them based on your configuration.
 
 **Parameters:**
 - `options` - Configuration options (see [Configuration](#configuration))
 
-**Returns:** A Promise that resolves to an `AfKafka` instance
+**Returns:** A `KafkaManager<ProducerNames>` instance where `ProducerNames` are the keys from your producers config
 
 **Example:**
 ```typescript
-const kafka = await AfKafka.create({
-  brokers: ['localhost:9092'],
-  clientId: 'my-service',
+const kafka = KafkaManager.create({
+  enabled: true,
+  logger,
+  producers: {
+    main: {
+      brokers: ['kafka:9092'],
+      clientId: 'my-service',
+    },
+    analytics: {
+      brokers: ['kafka-analytics:9092'],
+      clientId: 'my-service-analytics',
+    },
+  },
 });
+
+// Type: KafkaManager<'main' | 'analytics'>
+// TypeScript knows the valid producer names!
+
+// Producers connect automatically on first publish
+await kafka.publish('main', 'my-topic', { data: 'test' }); // ✅ Valid
+await kafka.publish('analytics', 'metrics', { value: 1 }); // ✅ Valid
+// await kafka.publish('wrong', 'topic', {}); // ❌ TypeScript error!
 ```
 
-### `publish<T>(topic, value, options?): Promise<RecordMetadata[]>`
+### `publish<T>(producerName, topic, value, options?): Promise<RecordMetadata[]>`
 
-Publishes a single message to a topic.
+Publishes a single message to a topic using a named producer. Producer name is **type-safe** - must match one of the configured producers.
 
 **Parameters:**
+- `producerName` - Name of the producer to use (type-safe: only accepts configured producer names)
 - `topic` - Topic name
 - `value` - Message value (will be JSON stringified)
 - `options` (optional) - Publish options
@@ -291,19 +451,22 @@ Publishes a single message to a topic.
 
 **Example:**
 ```typescript
-const metadata = await kafka.publish('events', { id: 1, data: 'test' }, {
+// TypeScript autocompletes and validates producer names
+const metadata = await kafka.publish('main', 'events', { id: 1, data: 'test' }, {
   key: 'key-1',
   headers: { type: 'create' },
 });
 
-console.log(`Published to partition ${metadata[0].partition} at offset ${metadata[0].offset}`);
+// Type error if using non-existent producer
+// await kafka.publish('nonexistent', 'topic', {}); // ❌ Error!
 ```
 
-### `publishBatch(options): Promise<RecordMetadata[]>`
+### `publishBatch(producerName, options): Promise<RecordMetadata[]>`
 
-Publishes multiple messages in a batch to a topic.
+Publishes multiple messages in a batch using a named producer. Producer name is **type-safe**.
 
 **Parameters:**
+- `producerName` - Name of the producer to use (type-safe: only accepts configured producer names)
 - `options.topic` - Topic name
 - `options.messages` - Array of messages with values, keys, headers, etc.
 
@@ -311,7 +474,8 @@ Publishes multiple messages in a batch to a topic.
 
 **Example:**
 ```typescript
-const metadata = await kafka.publishBatch({
+// TypeScript validates 'main' is a configured producer
+const metadata = await kafka.publishBatch('main', {
   topic: 'events',
   messages: [
     { value: { id: 1 }, key: 'key-1' },
@@ -320,62 +484,113 @@ const metadata = await kafka.publishBatch({
 });
 ```
 
+### `isReady(): Promise<boolean>`
+
+Check if all producers are connected and ready. Useful for health checks.
+
+**Example:**
+```typescript
+const ready = await kafka.isReady();
+if (!ready) {
+  throw new Error('Kafka not ready');
+}
+```
+
+### `getConnectionStatus(): Record<ProducerNames, boolean>`
+
+Get connection status for all producers. Return type is **type-safe** based on your configuration.
+
+**Example:**
+```typescript
+const status = kafka.getConnectionStatus();
+// Type: { main: boolean; analytics: boolean }
+// { main: true, analytics: false }
+```
+
+### `ping(): Promise<void>`
+
+Ping all producers to verify connectivity.
+
+**Example:**
+```typescript
+await kafka.ping(); // Throws if any producer fails to connect
+```
+
+### `pingProducer(name): Promise<void>`
+
+Ping a specific producer to verify connectivity. Producer name is **type-safe**.
+
+**Example:**
+```typescript
+await kafka.pingProducer('main'); // ✅ Valid
+// await kafka.pingProducer('invalid'); // ❌ Type error!
+```
+
 ### `disconnect(): Promise<void>`
 
-Disconnects the producer and cleans up resources.
+Disconnects all producers and cleans up resources.
 
 **Example:**
 ```typescript
 await kafka.disconnect();
 ```
 
+### `disconnectProducer(name): Promise<void>`
+
+Disconnects a specific producer and cleans up resources. Producer name is **type-safe**.
+
+**Example:**
+```typescript
+await kafka.disconnectProducer('main'); // ✅ Valid
+// await kafka.disconnectProducer('invalid'); // ❌ Type error!
+```
+
 ## Best Practices
 
-### 1. Use Centralized Topic Definitions
+### 1. Use Centralized Kafka Configuration
 
-Define your topics in `src/topics.ts`:
+Define your Kafka instance and topics in one place:
 
 ```typescript
+// src/kafka.ts
+import { KafkaManager } from '@autofleet/kafka';
+import logger from './logger';
+
+export const kafka = KafkaManager.create({
+  enabled: process.env.ENABLE_KAFKA === 'true',
+  logger,
+  producers: {
+    main: {
+      brokers: process.env.KAFKA_BROKERS?.split(',') || ['localhost:9092'],
+      clientId: 'my-service',
+      autoCreateTopics: process.env.NODE_ENV === 'development',
+    },
+  },
+});
+
 export const TOPICS = {
   USER_EVENTS: 'user-events',
   ORDER_EVENTS: 'order-events',
-  PAYMENT_EVENTS: 'payment-events',
 } as const;
-
-export type TopicName = typeof TOPICS[keyof typeof TOPICS];
-```
-
-Usage:
-
-```typescript
-import { TOPICS } from './topics';
-
-await kafka.publish(TOPICS.USER_EVENTS, data);
 ```
 
 ### 2. Use Message Keys for Ordering
 
 ```typescript
 // Ensure all events for a user are processed in order
-await kafka.publish('user-events', event, {
+await kafka.publish('main', TOPICS.USER_EVENTS, event, {
   key: `user-${userId}`,
 });
-
-// Ensure all events for a session are processed in order
-await kafka.publish('session-events', event, {
-  key: `session-${sessionId}`,
-});
 ```
 
 ### 3. Add Metadata with Headers
 
 ```typescript
-await kafka.publish('events', data, {
+await kafka.publish('main', 'events', data, {
   headers: {
     'correlation-id': requestId,
-    'source-service': serviceName,
+    'source-service': 'user-service',
     'event-type': eventType,
-    'schema-version': '1.0',
   },
 });
 ```
@@ -385,51 +600,61 @@ await kafka.publish('events', data, {
 ```typescript
 // Instead of multiple publish calls
 for (const event of events) {
-  await kafka.publish('events', event); // ❌ Slow
+  await kafka.publish('main', 'events', event); // ❌ Slow
 }
 
 // Use batch publishing
-await kafka.publishBatch({
+await kafka.publishBatch('main', {
   topic: 'events',
   messages: events.map(e => ({ value: e })), // ✅ Fast
 });
 ```
 
-### 5. Handle Errors Gracefully
+### 5. Integrate with Health Checks
+
+```typescript
+app.get('/health/ready', async (req, res) => {
+  const ready = await kafka.isReady();
+  res.status(ready ? 200 : 503).json({ ready });
+});
+```
+
+### 6. Handle Errors Gracefully
 
 ```typescript
 try {
-  await kafka.publish('events', data);
+  await kafka.publish('main', 'events', data);
 } catch (error) {
   logger.error('Failed to publish to Kafka', { error, data });
-  // Consider:
-  // - Retry logic
-  // - Fallback to queue
-  // - Alert monitoring
+  // Consider: retry logic, fallback queue, monitoring alerts
 }
 ```
 
-### 6. Use Multiple Brokers for High Availability
+### 7. Use Multiple Producers for Different Clusters
 
 ```typescript
-const kafka = await AfKafka.create({
-  brokers: ['kafka1:9092', 'kafka2:9092', 'kafka3:9092'],
-  clientId: 'my-service',
+const kafka = KafkaManager.create({
+  enabled: true,
+  logger,
+  producers: {
+    // Production events
+    main: {
+      brokers: ['kafka-prod.svc.cluster.local:9092'],
+      clientId: 'my-service-main',
+    },
+    // Analytics (separate cluster)
+    analytics: {
+      brokers: ['kafka-analytics.svc.cluster.local:9092'],
+      clientId: 'my-service-analytics',
+    },
+  },
 });
-```
 
-### 7. Monitor Production Metrics
+// Critical events go to main cluster
+await kafka.publish('main', 'orders', orderData);
 
-```typescript
-const metadata = await kafka.publish('events', data);
-
-// Log metrics
-logger.info('Message published', {
-  topic: 'events',
-  partition: metadata[0].partition,
-  offset: metadata[0].offset,
-  latency: Date.now() - startTime,
-});
+// Analytics go to analytics cluster
+await kafka.publish('analytics', 'metrics', metricsData);
 ```
 
 ## Testing
@@ -450,27 +675,36 @@ pnpm run coverage
 
 ```typescript
 import { describe, it, expect, beforeEach } from 'vitest';
-import AfKafka from '@autofleet/kafka';
-
-describe('MyKafkaService', () => {
-  let kafka: AfKafka;
-
-  beforeEach(async () => {
-    kafka = await AfKafka.create({
-      brokers: ['localhost:9092'],
-      clientId: 'test-client',
+import { KafkaManager } from '@autofleet/kafka';
+
+describe('KafkaService', () => {
+  let kafka: KafkaManager;
+
+  beforeEach(() => {
+    kafka = KafkaManager.create({
+      enabled: false, // Use mock mode for tests
+      producers: {
+        main: {
+          brokers: ['localhost:9092'],
+          clientId: 'test-client',
+        },
+      },
     });
   });
 
-  it('should publish event successfully', async () => {
-    const result = await kafka.publish('test-topic', {
+  it('should publish event successfully in mock mode', async () => {
+    const result = await kafka.publish('main', 'test-topic', {
       userId: '123',
       action: 'test',
     });
 
     expect(result).toBeDefined();
-    expect(result[0]).toHaveProperty('partition');
-    expect(result[0]).toHaveProperty('offset');
+    expect(result[0]).toHaveProperty('topic', 'test-topic');
+  });
+
+  it('should report ready status', async () => {
+    const ready = await kafka.isReady();
+    expect(ready).toBe(true);
   });
 });
 ```
@@ -480,99 +714,30 @@ describe('MyKafkaService', () => {
 Common environment variable patterns:
 
 ```bash
-# Kafka brokers
+# Enable/disable Kafka
+ENABLE_KAFKA=true
+
+# Kafka brokers (comma-separated)
 KAFKA_BROKERS=kafka1:9092,kafka2:9092,kafka3:9092
 
 # Client configuration
 KAFKA_CLIENT_ID=my-service
-KAFKA_SASL_USERNAME=user
-KAFKA_SASL_PASSWORD=pass
 ```
 
 Usage:
 
 ```typescript
-const kafka = await AfKafka.create({
-  brokers: process.env.KAFKA_BROKERS?.split(',') || ['localhost:9092'],
-  clientId: process.env.KAFKA_CLIENT_ID || 'default-client',
-});
-```
-
-## Common Patterns
-
-### Event Sourcing
-
-```typescript
-await kafka.publish('user-events', {
-  eventType: 'UserCreated',
-  aggregateId: userId,
-  timestamp: Date.now(),
-  data: userData,
-}, {
-  key: userId,
-  headers: {
-    'event-type': 'UserCreated',
-    'aggregate-type': 'User',
-  },
-});
-```
-
-### Change Data Capture (CDC)
-
-```typescript
-await kafka.publish('database-changes', {
-  operation: 'INSERT',
-  table: 'users',
-  before: null,
-  after: newUserRecord,
-}, {
-  key: newUserRecord.id,
-  headers: {
-    'schema': 'public',
-    'table': 'users',
+const kafka = KafkaManager.create({
+  enabled: process.env.ENABLE_KAFKA === 'true',
+  producers: {
+    main: {
+      brokers: process.env.KAFKA_BROKERS?.split(',') || ['localhost:9092'],
+      clientId: process.env.KAFKA_CLIENT_ID || 'default-client',
+    },
   },
 });
 ```
 
-### Dead Letter Queue Pattern
-
-```typescript
-try {
-  await processMessage(message);
-} catch (error) {
-  // Send to DLQ for manual review
-  await kafka.publish('events-dlq', {
-    originalTopic: 'events',
-    error: error.message,
-    message: message,
-  });
-}
-```
-
-## Troubleshooting
-
-### Connection Issues
-
-```typescript
-// Use multiple brokers for redundancy
-const kafka = await AfKafka.create({
-  brokers: ['kafka1:9092', 'kafka2:9092', 'kafka3:9092'],
-  clientId: 'my-service',
-});
-```
-
-### Performance Tuning
-
-For high-throughput scenarios, use batch publishing:
-
-```typescript
-// Batch multiple messages together
-await kafka.publishBatch({
-  topic: 'events',
-  messages: events.map(e => ({ value: e })),
-});
-```
-
 ## License
 
 Proprietary - Autofleet