@venizia/ignis-docs 0.0.7-1 → 0.0.7

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@venizia/ignis-docs",
-  "version": "0.0.7-1",
+  "version": "0.0.7",
   "description": "Interactive documentation site and MCP (Model Context Protocol) server for the Ignis Framework. Includes a VitePress-powered documentation site with guides, API references, and best practices. Ships an MCP server (CLI: ignis-docs-mcp) with 11 tools for AI assistants to search docs, browse source code, verify dependencies, and access real-time framework knowledge. Built with Mastra MCP SDK and Fuse.js fuzzy search.",
   "keywords": [
     "ai",

package/wiki/references/helpers/kafka/admin.md

@@ -0,0 +1,178 @@
+# Admin
+
+The `KafkaAdminHelper` wraps `@platformatic/kafka`'s `Admin` with health tracking, graceful shutdown, and broker event callbacks. Use `getAdmin()` to access the full Admin API directly.
+
+```typescript
+class KafkaAdminHelper extends BaseKafkaHelper<Admin>
+```
+
+> [!NOTE]
+> `KafkaAdminHelper` has **no generic type parameters** — the Admin client does not deal with serialized messages.
+
+## Helper API
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `newInstance(opts)` | `static newInstance(opts): KafkaAdminHelper` | Factory method |
+| `getAdmin()` | `(): Admin` | Access the underlying `Admin` |
+| `isHealthy()` | `(): boolean` | `true` when broker connected |
+| `isReady()` | `(): boolean` | Same as `isHealthy()` |
+| `getHealthStatus()` | `(): TKafkaHealthStatus` | `'connected'` \| `'disconnected'` \| `'unknown'` |
+| `close(opts?)` | `(opts?: { isForce?: boolean }): Promise<void>` | Close the admin connection (default: graceful) |
+
+## IKafkaAdminOptions
+
+```typescript
+interface IKafkaAdminOptions extends IKafkaConnectionOptions {
+  identifier?: string;       // Default: 'kafka-admin'
+  shutdownTimeout?: number;  // Default: 30000ms
+  onBrokerConnect?: TKafkaBrokerEventCallback;
+  onBrokerDisconnect?: TKafkaBrokerEventCallback;
+}
+```
+
+Plus all [Connection Options](./#connection-options).
+
+## Basic Example
+
+```typescript
+import { KafkaAdminHelper } from '@venizia/ignis-helpers/kafka';
+
+const helper = KafkaAdminHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-admin',
+  onBrokerConnect: ({ broker }) => console.log(`Connected to ${broker.host}:${broker.port}`),
+  onBrokerDisconnect: ({ broker }) => console.log(`Disconnected from ${broker.host}`),
+});
+
+const admin = helper.getAdmin();
+
+// Create a topic
+await admin.createTopics({ topics: ['orders'], partitions: 3, replicas: 2 });
+
+// List all topics
+const topics = await admin.listTopics();
+
+// Health check
+helper.isHealthy(); // true when connected
+
+// Graceful close
+await helper.close();
+
+// Or force close
+await helper.close({ isForce: true });
+```
+
+## API Reference (`@platformatic/kafka`)
+
+After calling `helper.getAdmin()`, you have full access to the `Admin` class.
+
+### Topic Management
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `createTopics(opts)` | `(opts: { topics: string[], partitions?: number, replicas?: number, configs?: Config[] }): Promise<CreatedTopic[]>` | Create topics |
+| `deleteTopics(opts)` | `(opts: { topics: string[] }): Promise<void>` | Delete topics |
+| `listTopics(opts?)` | `(opts?: { includeInternals?: boolean }): Promise<string[]>` | List all topics |
+| `createPartitions(opts)` | `(opts: { topics: CreatePartitionsRequestTopic[], validateOnly?: boolean }): Promise<void>` | Add partitions to existing topics |
+| `deleteRecords(opts)` | `(opts: { topics: { name, partitions: { partition, offset }[] }[] }): Promise<DeletedRecordsTopic[]>` | Delete records up to offset |
+
+```typescript
+// Create with custom configuration
+await admin.createTopics({
+  topics: ['orders'],
+  partitions: 6,
+  replicas: 3,
+  configs: [
+    { name: 'retention.ms', value: '604800000' },   // 7 days
+    { name: 'cleanup.policy', value: 'compact' },
+    { name: 'compression.type', value: 'zstd' },
+  ],
+});
+
+// Add partitions (can only increase, never decrease)
+await admin.createPartitions({
+  topics: [{ name: 'orders', count: 12 }],
+});
+
+// Delete records before offset 1000 on partition 0
+await admin.deleteRecords({
+  topics: [{ name: 'orders', partitions: [{ partition: 0, offset: 1000n }] }],
+});
+```
+
+### Consumer Group Management
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `listGroups(opts?)` | `(opts?: { states?: string[], types?: string[] }): Promise<Map<string, GroupBase>>` | List consumer groups |
+| `describeGroups(opts)` | `(opts: { groups: string[] }): Promise<Map<string, Group>>` | Describe consumer groups (members, assignments) |
+| `deleteGroups(opts)` | `(opts: { groups: string[] }): Promise<void>` | Delete consumer groups |
+| `removeMembersFromConsumerGroup(opts)` | `(opts: { groupId, members? }): Promise<void>` | Remove specific members |
+
+```typescript
+// List all active groups
+const groups = await admin.listGroups({ states: ['STABLE'] });
+
+// Describe group members and partition assignments
+const details = await admin.describeGroups({ groups: ['order-processing'] });
+for (const [groupId, group] of details) {
+  console.log(`Group: ${groupId}, State: ${group.state}`);
+  for (const [memberId, member] of group.members) {
+    console.log(`  Member: ${member.clientId} (${member.clientHost})`);
+  }
+}
+```
+
+### Offset Management
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `listOffsets(opts)` | `(opts): Promise<ListedOffsetsTopic[]>` | List partition offsets at timestamps |
+| `listConsumerGroupOffsets(opts)` | `(opts: { groups }): Promise<ListConsumerGroupOffsetsGroup[]>` | List committed offsets for groups |
+| `alterConsumerGroupOffsets(opts)` | `(opts: { groupId, topics }): Promise<void>` | Reset/alter committed offsets |
+| `deleteConsumerGroupOffsets(opts)` | `(opts: { groupId, topics }): Promise<...>` | Delete committed offsets |
+
+```typescript
+// Reset consumer group offsets to earliest
+await admin.alterConsumerGroupOffsets({
+  groupId: 'order-processing',
+  topics: [{
+    name: 'orders',
+    partitionOffsets: [
+      { partition: 0, offset: 0n },
+      { partition: 1, offset: 0n },
+      { partition: 2, offset: 0n },
+    ],
+  }],
+});
+```
+
+### Configuration Management
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `describeConfigs(opts)` | `(opts: { resources, includeSynonyms?, includeDocumentation? }): Promise<ConfigDescription[]>` | Describe broker/topic configurations |
+| `alterConfigs(opts)` | `(opts: { resources, validateOnly? }): Promise<void>` | Replace topic/broker configs |
+| `incrementalAlterConfigs(opts)` | `(opts: { resources, validateOnly? }): Promise<void>` | Incrementally modify configs |
+
+### ACL Management
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `createAcls(opts)` | `(opts: { creations: Acl[] }): Promise<void>` | Create access control lists |
+| `describeAcls(opts)` | `(opts: { filter: AclFilter }): Promise<DescribeAclsResponseResource[]>` | Describe ACLs |
+| `deleteAcls(opts)` | `(opts: { filters: AclFilter[] }): Promise<Acl[]>` | Delete ACLs |
+
+### Quota Management
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `describeClientQuotas(opts)` | `(opts): Promise<DescribeClientQuotasResponseEntry[]>` | Describe client quotas |
+| `alterClientQuotas(opts)` | `(opts): Promise<AlterClientQuotasResponseEntries[]>` | Alter client quotas |
+
+### Log Management
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `describeLogDirs(opts)` | `(opts: { topics }): Promise<BrokerLogDirDescription[]>` | Describe broker log directories |

package/wiki/references/helpers/kafka/consumer.md

@@ -0,0 +1,384 @@
+# Consumer
+
+The `KafkaConsumerHelper` wraps `@platformatic/kafka`'s `Consumer` with health tracking, graceful shutdown, message callbacks, consumer group event callbacks, and lag monitoring.
+
+```typescript
+class KafkaConsumerHelper<
+  KeyType = string,
+  ValueType = string,
+  HeaderKeyType = string,
+  HeaderValueType = string,
+> extends BaseKafkaHelper<Consumer<KeyType, ValueType, HeaderKeyType, HeaderValueType>>
+```
+
+## Helper API
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `newInstance(opts)` | `static newInstance<K,V,HK,HV>(opts): KafkaConsumerHelper<K,V,HK,HV>` | Factory method |
+| `getConsumer()` | `(): Consumer<K,V,HK,HV>` | Access the underlying `Consumer` |
+| `getStream()` | `(): MessagesStream \| null` | Get the active stream (after `start()`) |
+| `start(opts)` | `(opts: IKafkaConsumeStartOptions): Promise<void>` | Start consuming (creates stream, wires callbacks) |
+| `startLagMonitoring(opts)` | `(opts: { topics: string[]; interval?: number }): void` | Start periodic lag monitoring |
+| `stopLagMonitoring()` | `(): void` | Stop lag monitoring |
+| `isHealthy()` | `(): boolean` | `true` when broker connected |
+| `isReady()` | `(): boolean` | `isHealthy()` **and** `consumer.isActive()` |
+| `getHealthStatus()` | `(): TKafkaHealthStatus` | `'connected'` \| `'disconnected'` \| `'unknown'` |
+| `close(opts?)` | `(opts?: { isForce?: boolean }): Promise<void>` | Stop lag, close stream, close consumer |
+
+## IKafkaConsumerOptions
+
+```typescript
+interface IKafkaConsumerOptions<KeyType, ValueType, HeaderKeyType, HeaderValueType>
+  extends IKafkaConnectionOptions
+```
+
+### Consumer Configuration
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `groupId` | `string` | — | Consumer group ID. **Required** |
+| `identifier` | `string` | `'kafka-consumer'` | Scoped logging identifier |
+| `deserializers` | `Partial<Deserializers<K,V,HK,HV>>` | — | Key/value/header deserializers |
+| `autocommit` | `boolean \| number` | `false` | Auto-commit offsets. `true` = default interval, `number` = custom ms |
+| `sessionTimeout` | `number` | `30000` | Session timeout — consumer removed from group if no heartbeat |
+| `heartbeatInterval` | `number` | `3000` | Heartbeat interval — must be less than `sessionTimeout` |
+| `rebalanceTimeout` | `number` | `sessionTimeout` | Max time for rebalance |
+| `highWaterMark` | `number` | `1024` | Stream buffer size (messages) |
+| `minBytes` | `number` | `1` | Min bytes per fetch response |
+| `maxBytes` | `number` | — | Max bytes per fetch response per partition |
+| `maxWaitTime` | `number` | — | Max time (ms) broker waits for `minBytes` |
+| `metadataMaxAge` | `number` | `300000` | Metadata cache TTL (ms) |
+| `groupProtocol` | `'classic' \| 'consumer'` | `'classic'` | Consumer group protocol. `'consumer'` = KIP-848 (Kafka 3.7+) |
+| `groupInstanceId` | `string` | — | Static group membership ID — prevents rebalance on restart |
+| `shutdownTimeout` | `number` | `30000` | Graceful shutdown timeout in ms |
+| `registry` | `SchemaRegistry` | — | Schema registry for auto deser |
+
+### Lifecycle Callbacks
+
+| Option | Type | Description |
+|--------|------|-------------|
+| `onBrokerConnect` | `TKafkaBrokerEventCallback` | Called when broker connects |
+| `onBrokerDisconnect` | `TKafkaBrokerEventCallback` | Called when broker disconnects |
+
+### Message Callbacks
+
+| Option | Type | Description |
+|--------|------|-------------|
+| `onMessage` | `TKafkaMessageCallback<K,V,HK,HV>` | Called for each message. Receives `{ message }` |
+| `onMessageDone` | `TKafkaMessageDoneCallback<K,V,HK,HV>` | Called after `onMessage` succeeds. Receives `{ message }` |
+| `onMessageError` | `TKafkaMessageErrorCallback<K,V,HK,HV>` | Called on processing error. Receives `{ error, message? }` |
+
+### Consumer Group Callbacks
+
+| Option | Type | Description |
+|--------|------|-------------|
+| `onGroupJoin` | `TKafkaGroupJoinCallback` | Receives `{ groupId, memberId, generationId? }` |
+| `onGroupLeave` | `TKafkaGroupLeaveCallback` | Receives `{ groupId, memberId }` |
+| `onGroupRebalance` | `TKafkaGroupRebalanceCallback` | Receives `{ groupId }` |
+| `onHeartbeatError` | `TKafkaHeartbeatErrorCallback` | Receives `{ error, groupId?, memberId? }` |
+
+### Lag Monitoring Callbacks
+
+| Option | Type | Description |
+|--------|------|-------------|
+| `onLag` | `TKafkaLagCallback` | Receives `{ lag }` (Offsets map) |
+| `onLagError` | `TKafkaLagErrorCallback` | Receives `{ error }` |
+
+Plus all [Connection Options](./#connection-options).
+
+## Basic Example
+
+```typescript
+import { KafkaConsumerHelper } from '@venizia/ignis-helpers/kafka';
+import { stringDeserializers } from '@platformatic/kafka';
+
+const helper = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'order-consumer',
+  groupId: 'order-processing',
+  deserializers: stringDeserializers,
+
+  // Message lifecycle
+  onMessage: async ({ message }) => {
+    console.log(`${message.topic}[${message.partition}] @${message.offset}: ${message.key} → ${message.value}`);
+    await message.commit();
+  },
+  onMessageDone: ({ message }) => {
+    console.log(`Done processing: ${message.key}`);
+  },
+  onMessageError: ({ error, message }) => {
+    console.error('Processing failed:', error.message, message?.key);
+  },
+
+  // Consumer group events
+  onGroupJoin: ({ groupId, memberId }) => console.log(`Joined ${groupId} as ${memberId}`),
+  onGroupLeave: ({ groupId }) => console.log(`Left ${groupId}`),
+  onGroupRebalance: ({ groupId }) => console.log(`Rebalance in ${groupId}`),
+  onHeartbeatError: ({ error }) => console.error('Heartbeat failed:', error),
+
+  // Broker events
+  onBrokerConnect: ({ broker }) => console.log(`Connected to ${broker.host}:${broker.port}`),
+  onBrokerDisconnect: ({ broker }) => console.log(`Disconnected from ${broker.host}`),
+
+  // Lag monitoring
+  onLag: ({ lag }) => {
+    for (const [topic, partitionLags] of lag) {
+      partitionLags.forEach((lagValue, partition) => {
+        if (lagValue > 1000n) {
+          console.warn(`High lag on ${topic}[${partition}]: ${lagValue}`);
+        }
+      });
+    }
+  },
+  onLagError: ({ error }) => console.error('Lag monitoring error:', error),
+});
+
+// Start consuming
+await helper.start({ topics: ['orders'] });
+
+// Start lag monitoring (optional)
+helper.startLagMonitoring({ topics: ['orders'], interval: 10_000 });
+
+// Health check
+helper.isHealthy(); // true when broker connected
+helper.isReady();   // true when broker connected AND consumer is active
+
+// Shutdown
+await helper.close();
+```
+
+## Message Callback Flow
+
+When `start()` is called, the helper creates a `MessagesStream` and wires the callbacks:
+
+```
+Stream 'data' event
+  → onMessage({ message })
+    ├── success → onMessageDone({ message })
+    └── error   → onMessageError({ error, message })
+
+Stream 'error' event
+  → onMessageError({ error })  (no message available)
+```
+
+- `onMessage` is the main processing callback — do your business logic here
+- `onMessageDone` fires only after `onMessage` resolves successfully — use for logging, metrics, etc.
+- `onMessageError` fires if `onMessage` OR `onMessageDone` throws — use for error tracking
+- The stream `'error'` event also calls `onMessageError` (without `message` since it's a stream-level error)
+
+## start()
+
+`start()` creates the consume stream and wires all message callbacks. It must be called explicitly after construction.
+
+```typescript
+interface IKafkaConsumeStartOptions {
+  topics: string[];
+  mode?: MessagesStreamModeValue;         // Default: 'committed'
+  fallbackMode?: MessagesStreamFallbackModeValue; // Default: 'latest'
+}
+```
+
+| Mode | Description |
+|------|-------------|
+| `'committed'` | Resume from last committed offset. **Recommended for production** |
+| `'latest'` | Start from the latest offset (skip existing messages) |
+| `'earliest'` | Start from the beginning of the topic |
+| `'manual'` | Start from explicitly provided offsets |
+
+| Fallback | Description |
+|----------|-------------|
+| `'latest'` | Start from latest (default) — ignore historical messages |
+| `'earliest'` | Start from beginning — process all historical messages |
+| `'fail'` | Throw an error |
+
+```typescript
+// Production pattern
+await helper.start({ topics: ['orders'] });
+
+// Replay all historical messages
+await helper.start({ topics: ['orders'], mode: 'earliest' });
+
+// Custom mode
+await helper.start({
+  topics: ['orders'],
+  mode: 'committed',
+  fallbackMode: 'earliest',
+});
+```
+
+Guards against duplicate starts — calling `start()` twice logs a warning and returns immediately.
+
+## Lag Monitoring
+
+```typescript
+// Start monitoring (polls every interval)
+helper.startLagMonitoring({ topics: ['orders'], interval: 10_000 });
+
+// Stop monitoring
+helper.stopLagMonitoring();
+```
+
+Lag data is delivered via the `onLag` callback. Errors via `onLagError`.
+
+Guards against duplicate starts — calling `startLagMonitoring()` twice logs a warning.
+
+For one-time lag checks, use the underlying consumer directly:
+
+```typescript
+const lag = await helper.getConsumer().getLag({ topics: ['orders'] });
+```
+
+## Graceful Shutdown
+
+`close()` implements an ordered shutdown:
+
+1. Stop lag monitoring
+2. Close the stream (leave consumer group)
+3. Close the consumer client (graceful with timeout, or force)
+4. Set health status to `'disconnected'`
+
+```typescript
+// Graceful (recommended)
+await helper.close();
+
+// Force
+await helper.close({ isForce: true });
+```
+
+## Direct Stream Access
+
+If you don't use the callback pattern, you can access the stream directly after `start()`:
+
+```typescript
+// After start()
+const stream = helper.getStream();
+
+// Or use the consumer directly (bypass helper's start())
+const consumer = helper.getConsumer();
+const stream = await consumer.consume({
+  topics: ['orders'],
+  mode: 'committed',
+  fallbackMode: 'latest',
+});
+
+for await (const message of stream) {
+  await processMessage(message);
+  await message.commit();
+}
+```
+
+## API Reference (`@platformatic/kafka`)
+
+### Message Object
+
+```typescript
+interface Message<Key, Value, HeaderKey, HeaderValue> {
+  topic: string;
+  key: Key;
+  value: Value;
+  partition: number;
+  offset: bigint;
+  timestamp: bigint;
+  headers: Map<HeaderKey, HeaderValue>;
+  metadata: Record<string, unknown>;
+  commit(callback?: (error?: Error) => void): void | Promise<void>;
+  toJSON(): MessageJSON<Key, Value, HeaderKey, HeaderValue>;
+}
+```
+
+> [!WARNING]
+> `message.offset` and `message.timestamp` are `bigint`. When using `JSON.stringify`, provide a custom replacer:
+> ```typescript
+> JSON.stringify(data, (_key, value) => (typeof value === 'bigint' ? value.toString() : value))
+> ```
+
+### MessagesStream
+
+`MessagesStream` extends Node.js `Readable`. Three consumption patterns:
+
+**Async Iterator** (sequential, backpressure):
+```typescript
+for await (const message of stream) {
+  await processMessage(message);
+  await message.commit();
+}
+```
+
+**Event-Based** (high-throughput):
+```typescript
+stream.on('data', (message) => {
+  processMessage(message);
+  message.commit();
+});
+```
+
+**Pause/Resume** (manual flow control):
+```typescript
+stream.on('data', async (message) => {
+  stream.pause();
+  await heavyProcessing(message);
+  message.commit();
+  stream.resume();
+});
+```
+
+### Offset Management
+
+```typescript
+// Manual commit (when autocommit: false)
+for await (const message of stream) {
+  await processMessage(message);
+  await message.commit();
+}
+
+// Bulk commit
+await consumer.commit({
+  offsets: [
+    { topic: 'orders', partition: 0, offset: 150n, leaderEpoch: 0 },
+    { topic: 'orders', partition: 1, offset: 300n, leaderEpoch: 0 },
+  ],
+});
+
+// List offsets
+const offsets = await consumer.listOffsets({ topics: ['orders'] });
+const committed = await consumer.listCommittedOffsets({
+  topics: [{ topic: 'orders', partitions: [0, 1, 2] }],
+});
+```
+
+### Consumer Group Management
+
+```typescript
+consumer.groupId;        // string
+consumer.memberId;       // string | null
+consumer.generationId;   // number
+consumer.assignments;    // GroupAssignment[] | null
+consumer.isActive();     // boolean
+
+// Static membership — prevents rebalance on restart
+const helper = KafkaConsumerHelper.newInstance({
+  ...
+  groupInstanceId: 'worker-1',
+  sessionTimeout: 60_000,
+});
+```
+
+### Consumer Group Partitioning
+
+When multiple consumers share the same `groupId`, Kafka distributes topic partitions across group members:
+
+```
+Topic "orders" (3 partitions)
+├── Partition 0 → Consumer A
+├── Partition 1 → Consumer B
+└── Partition 2 → Consumer C
+```
+
+- Each partition is assigned to **exactly one** consumer in the group
+- If a consumer leaves/crashes, its partitions are redistributed (**rebalance**)
+- If consumers > partitions, excess consumers sit idle
+- Messages within a partition are processed **in order**
+
+> [!TIP]
+> Create topics with enough partitions for your expected parallelism. You can increase partitions later with `admin.createPartitions()`, but you cannot decrease them.