specweave 0.30.13 → 0.30.16

package/plugins/specweave-kafka/examples/README.md

@@ -0,0 +1,191 @@
+# Kafka Examples
+
+**Working code examples for SpecWeave Kafka plugins**
+
+This directory contains 5 complete, runnable examples demonstrating Kafka best practices.
+
+## Quick Start
+
+Each example is self-contained with its own README, dependencies, and Docker Compose file.
+
+```bash
+cd simple-producer-consumer
+npm install
+npm start
+```
+
+## Examples
+
+### 1. [Simple Producer-Consumer](./simple-producer-consumer/)
+
+**Basic Kafka operations**
+
+- ✅ Connect to Kafka
+- ✅ Produce JSON messages
+- ✅ Consume with consumer groups
+- ✅ Local Kafka cluster (Docker Compose)
+
+**Difficulty**: Beginner
+**Time**: 15 minutes
+
+### 2. [Avro Schema Registry](./avro-schema-registry/)
+
+**Schema-based serialization**
+
+- ✅ Register Avro schemas
+- ✅ Schema evolution (backward compatible)
+- ✅ Binary serialization (smaller than JSON)
+- ✅ Message validation
+
+**Difficulty**: Intermediate
+**Time**: 20 minutes
+
+### 3. [Exactly-Once Semantics](./exactly-once-semantics/)
+
+**Zero message duplication or loss**
+
+- ✅ Transactional producers
+- ✅ Read-committed consumers
+- ✅ Atomic offset commits
+- ✅ End-to-end exactly-once pipeline
+
+**Difficulty**: Advanced
+**Time**: 30 minutes
+
+### 4. [Kafka Streams Application](./kafka-streams-app/)
+
+**Real-time stream processing**
+
+- ✅ Windowed aggregations (tumbling windows)
+- ✅ Stateful processing
+- ✅ Event-time processing
+- ✅ RocksDB state store
+
+**Difficulty**: Advanced
+**Time**: 45 minutes
+
+### 5. [n8n Workflow Integration](./n8n-workflow/)
+
+**No-code event-driven automation**
+
+- ✅ Kafka trigger workflows
+- ✅ Event filtering and routing
+- ✅ Multi-sink integration (Slack, DB, email)
+- ✅ Visual workflow builder
+
+**Difficulty**: Beginner
+**Time**: 20 minutes
+
+## Learning Path
+
+### Beginner
+1. Start with **simple-producer-consumer** to understand basics
+2. Try **n8n-workflow** for no-code integration
+
+### Intermediate
+3. Learn schema management with **avro-schema-registry**
+4. Explore windowed aggregations in **kafka-streams-app**
+
+### Advanced
+5. Master reliability with **exactly-once-semantics**
+
+## Prerequisites
+
+All examples require:
+- **Node.js 18+** (`node --version`)
+- **Docker Desktop** (for local Kafka cluster)
+- **npm** (`npm --version`)
+
+Optional (for specific examples):
+- **Schema Registry** (avro-schema-registry example)
+- **n8n** (n8n-workflow example)
+
+## Quick Start Local Kafka
+
+Each example includes a `docker-compose.yml` for local Kafka:
+
+```bash
+# Start Kafka
+docker-compose up -d
+
+# Wait for cluster (~30 seconds)
+docker logs kafka-broker -f
+
+# Stop Kafka
+docker-compose down
+```
+
+## Common Issues
+
+### Port 9092 Already in Use
+
+```bash
+# Find process using port 9092
+lsof -i :9092
+
+# Kill the process
+kill -9 <PID>
+```
+
+### Kafka Container Won't Start
+
+```bash
+# Check Docker resources
+docker system df
+
+# Increase Docker memory to 8GB
+# Docker Desktop → Preferences → Resources → Memory
+```
+
+### Consumer Not Receiving Messages
+
+```bash
+# Check topic exists
+kafka-topics --bootstrap-server localhost:9092 --list
+
+# Reset consumer group offsets
+kafka-consumer-groups --bootstrap-server localhost:9092 \
+  --group my-group \
+  --topic my-topic \
+  --reset-offsets --to-earliest \
+  --execute
+```
+
+## Running Tests
+
+Some examples include tests:
+
+```bash
+cd simple-producer-consumer
+npm test
+```
+
+## Production Deployment
+
+These examples are for learning. For production:
+
+1. **Security**: Enable SASL/SSL authentication
+2. **Monitoring**: Add Prometheus + Grafana
+3. **Reliability**: Use 3+ brokers with replication
+4. **Performance**: Tune batching, compression, partitions
+5. **Deployment**: Use Terraform modules (see `../terraform/`)
+
+## Documentation
+
+- **Getting Started**: [kafka-getting-started.md](../../../.specweave/docs/public/guides/kafka-getting-started.md)
+- **Advanced Usage**: [kafka-advanced-usage.md](../../../.specweave/docs/public/guides/kafka-advanced-usage.md)
+- **Terraform Guide**: [kafka-terraform.md](../../../.specweave/docs/public/guides/kafka-terraform.md)
+- **Troubleshooting**: [kafka-troubleshooting.md](../../../.specweave/docs/public/guides/kafka-troubleshooting.md)
+
+## Contributing
+
+Found a bug or want to add an example? See [CONTRIBUTING.md](../../../CONTRIBUTING.md).
+
+## Support
+
+- **GitHub Issues**: [Report a problem](https://github.com/anton-abyzov/specweave/issues)
+- **Discussions**: [Ask questions](https://github.com/anton-abyzov/specweave/discussions)
+
+---
+
+**Last Updated**: 2025-11-15

package/plugins/specweave-kafka/examples/avro-schema-registry/.env.example

@@ -0,0 +1,8 @@
+KAFKA_BROKERS=localhost:9092
+SCHEMA_REGISTRY_URL=http://localhost:8081
+
+# For Confluent Cloud:
+# KAFKA_BROKERS=pkc-xxxxx.us-east-1.aws.confluent.cloud:9092
+# SCHEMA_REGISTRY_URL=https://psrc-xxxxx.us-east-1.aws.confluent.cloud
+# SCHEMA_REGISTRY_API_KEY=your-api-key
+# SCHEMA_REGISTRY_API_SECRET=your-secret

package/plugins/specweave-kafka/examples/avro-schema-registry/README.md

@@ -0,0 +1,69 @@
+# Avro Schema Registry Example
+
+**Schema-based serialization with Confluent Schema Registry**
+
+Demonstrates Avro schema registration, evolution, and message validation.
+
+## Prerequisites
+
+- Node.js 18+
+- Kafka cluster with Schema Registry
+- Schemas can evolve while maintaining backward compatibility
+
+## Setup
+
+```bash
+npm install
+cp .env.example .env
+# Configure KAFKA_BROKERS and SCHEMA_REGISTRY_URL
+```
+
+## Run
+
+```bash
+# Register schema and produce
+npm run producer
+
+# Consume with schema validation
+npm run consumer
+```
+
+## Features
+
+- **Schema Registration** - Auto-register Avro schemas
+- **Schema Evolution** - Add optional fields safely
+- **Validation** - Messages must match schema
+- **Serialization** - Binary Avro encoding (smaller than JSON)
+
+## Schema Evolution Example
+
+```javascript
+// V1 - Initial schema
+const userSchemaV1 = {
+  type: 'record',
+  name: 'User',
+  fields: [
+    { name: 'id', type: 'long' },
+    { name: 'name', type: 'string' }
+  ]
+};
+
+// V2 - BACKWARD compatible (add optional field)
+const userSchemaV2 = {
+  type: 'record',
+  name: 'User',
+  fields: [
+    { name: 'id', type: 'long' },
+    { name: 'name', type: 'string' },
+    { name: 'email', type: ['null', 'string'], default: null } // NEW
+  ]
+};
+
+// Old consumers can still read V2 messages (ignore email)
+// New producers can send V2 messages with email field
+```
+
+## Documentation
+
+- [Avro Specification](https://avro.apache.org/docs/current/spec.html)
+- [Confluent Schema Registry](https://docs.confluent.io/platform/current/schema-registry/index.html)

package/plugins/specweave-kafka/examples/avro-schema-registry/consumer.js

@@ -0,0 +1,37 @@
+require('dotenv').config();
+const { Kafka } = require('kafkajs');
+const { SchemaRegistry } = require('@kafkajs/confluent-schema-registry');
+
+const kafka = new Kafka({
+  brokers: (process.env.KAFKA_BROKERS || 'localhost:9092').split(',')
+});
+
+const registry = new SchemaRegistry({
+  host: process.env.SCHEMA_REGISTRY_URL || 'http://localhost:8081'
+});
+
+async function run() {
+  const consumer = kafka.consumer({ groupId: 'avro-consumer-group' });
+
+  try {
+    await consumer.connect();
+    await consumer.subscribe({ topic: 'users', fromBeginning: true });
+    console.log('✓ Consumer ready');
+
+    await consumer.run({
+      eachMessage: async ({ message }) => {
+        // Decode automatically (schema ID in message)
+        const user = await registry.decode(message.value);
+        console.log('📥 Received user:', user);
+        // Output: { id: 1, name: 'Alice', email: 'alice@example.com' }
+      }
+    });
+
+  } catch (error) {
+    console.error('❌ Error:', error);
+  }
+}
+
+process.on('SIGINT', async () => process.exit(0));
+
+run().catch(console.error);

package/plugins/specweave-kafka/examples/avro-schema-registry/package.json

@@ -0,0 +1,14 @@
+{
+  "name": "kafka-avro-schema-registry",
+  "version": "1.0.0",
+  "description": "Avro Schema Registry integration example",
+  "scripts": {
+    "producer": "node producer.js",
+    "consumer": "node consumer.js"
+  },
+  "dependencies": {
+    "kafkajs": "^2.2.4",
+    "@kafkajs/confluent-schema-registry": "^3.3.0",
+    "dotenv": "^16.3.1"
+  }
+}

package/plugins/specweave-kafka/examples/avro-schema-registry/producer.js

@@ -0,0 +1,57 @@
+require('dotenv').config();
+const { Kafka } = require('kafkajs');
+const { SchemaRegistry, SchemaType } = require('@kafkajs/confluent-schema-registry');
+
+const kafka = new Kafka({
+  brokers: (process.env.KAFKA_BROKERS || 'localhost:9092').split(',')
+});
+
+const registry = new SchemaRegistry({
+  host: process.env.SCHEMA_REGISTRY_URL || 'http://localhost:8081'
+});
+
+// Avro schema definition
+const schema = {
+  type: 'record',
+  name: 'User',
+  fields: [
+    { name: 'id', type: 'long' },
+    { name: 'name', type: 'string' },
+    { name: 'email', type: ['null', 'string'], default: null }
+  ]
+};
+
+async function run() {
+  const producer = kafka.producer();
+
+  try {
+    await producer.connect();
+    console.log('✓ Connected to Kafka');
+
+    // Register schema
+    const { id } = await registry.register({
+      type: SchemaType.AVRO,
+      schema: JSON.stringify(schema)
+    });
+    console.log(`✓ Schema registered with ID: ${id}`);
+
+    // Encode message with schema
+    const message = { id: 1, name: 'Alice', email: 'alice@example.com' };
+    const encodedValue = await registry.encode(id, message);
+
+    // Send message
+    await producer.send({
+      topic: 'users',
+      messages: [{ key: '1', value: encodedValue }]
+    });
+
+    console.log('✅ Message sent with schema validation');
+
+  } catch (error) {
+    console.error('❌ Error:', error);
+  } finally {
+    await producer.disconnect();
+  }
+}
+
+run().catch(console.error);

package/plugins/specweave-kafka/examples/exactly-once-semantics/.env.example

@@ -0,0 +1,5 @@
+KAFKA_BROKERS=localhost:9092
+
+# Important: Ensure broker configuration supports transactions:
+# transaction.state.log.replication.factor=3
+# transaction.state.log.min.isr=2

package/plugins/specweave-kafka/examples/exactly-once-semantics/README.md

@@ -0,0 +1,30 @@
+# Exactly-Once Semantics (EOS) Example
+
+**Zero message duplication or loss with transactional producers**
+
+Demonstrates end-to-end exactly-once processing for financial transactions.
+
+## Features
+
+- **Transactional Producer** - Atomic writes to multiple topics
+- **Read-Committed Consumer** - Only reads committed messages
+- **Offset Management** - Commits offsets within transactions
+- **Guaranteed Delivery** - Zero duplicates, zero message loss
+
+## Run
+
+```bash
+npm install
+npm run eos-pipeline
+```
+
+## Use Cases
+
+- Financial transactions
+- Order processing
+- Billing systems
+- Any scenario where duplicates are unacceptable
+
+## Documentation
+
+See [Advanced Usage Guide](../../.specweave/docs/public/guides/kafka-advanced-usage.md#exactly-once-semantics-eos)

package/plugins/specweave-kafka/examples/exactly-once-semantics/eos-pipeline.js

@@ -0,0 +1,79 @@
+require('dotenv').config();
+const { Kafka } = require('kafkajs');
+
+const kafka = new Kafka({
+  brokers: (process.env.KAFKA_BROKERS || 'localhost:9092').split(',')
+});
+
+const consumer = kafka.consumer({
+  groupId: 'eos-pipeline',
+  isolation: 'read_committed' // Only read committed transactions
+});
+
+const producer = kafka.producer({
+  transactional: true,
+  transactionalId: 'eos-pipeline-tx-001', // Unique per instance
+  idempotent: true, // Prevent duplicates
+  maxInFlightRequests: 1,
+  acks: -1
+});
+
+async function run() {
+  await consumer.connect();
+  await producer.connect();
+
+  await consumer.subscribe({ topic: 'input-events', fromBeginning: false });
+
+  console.log('✓ EOS Pipeline started');
+  console.log('⏳ Processing messages with exactly-once semantics...\n');
+
+  await consumer.run({
+    eachMessage: async ({ topic, partition, message }) => {
+      const transaction = await producer.transaction();
+
+      try {
+        // Parse input
+        const input = JSON.parse(message.value.toString());
+        console.log('📥 Input:', input);
+
+        // Transform
+        const output = {
+          ...input,
+          processed: true,
+          timestamp: new Date().toISOString()
+        };
+
+        // Send to output topic
+        await transaction.send({
+          topic: 'output-events',
+          messages: [{ key: message.key, value: JSON.stringify(output) }]
+        });
+
+        // Commit offset within transaction (atomic!)
+        await transaction.sendOffsets({
+          consumerGroupId: 'eos-pipeline',
+          topics: [{
+            topic,
+            partitions: [{ partition, offset: (parseInt(message.offset) + 1).toString() }]
+          }]
+        });
+
+        // Commit transaction (both output + offset committed atomically)
+        await transaction.commit();
+        console.log('✅ Processed:', output.id || 'message');
+
+      } catch (error) {
+        console.error('❌ Error, aborting transaction:', error.message);
+        await transaction.abort();
+      }
+    }
+  });
+}
+
+process.on('SIGINT', async () => {
+  await consumer.disconnect();
+  await producer.disconnect();
+  process.exit(0);
+});
+
+run().catch(console.error);

package/plugins/specweave-kafka/examples/exactly-once-semantics/package.json

@@ -0,0 +1,11 @@
+{
+  "name": "kafka-exactly-once-semantics",
+  "version": "1.0.0",
+  "scripts": {
+    "eos-pipeline": "node eos-pipeline.js"
+  },
+  "dependencies": {
+    "kafkajs": "^2.2.4",
+    "dotenv": "^16.3.1"
+  }
+}

package/plugins/specweave-kafka/examples/kafka-streams-app/.env.example

@@ -0,0 +1,4 @@
+KAFKA_BROKERS=localhost:9092
+
+# For production, use RocksDB for state store:
+# STATE_STORE_DIR=./state-store

package/plugins/specweave-kafka/examples/kafka-streams-app/README.md

@@ -0,0 +1,30 @@
+# Kafka Streams Application Example
+
+**Real-time stream processing with windowing and aggregations**
+
+Demonstrates Kafka Streams DSL for stateful processing.
+
+## Features
+
+- **Windowed Aggregations** - Count events per user in 5-minute windows
+- **Stream-Table Joins** - Enrich events with user data
+- **Stateful Processing** - Maintain aggregated state
+- **RocksDB State Store** - Persistent local state
+
+## Run
+
+```bash
+npm install
+npm run streams
+```
+
+## Use Cases
+
+- Real-time analytics
+- User behavior tracking
+- Fraud detection
+- IoT data aggregation
+
+## Documentation
+
+See [Advanced Usage Guide](../../.specweave/docs/public/guides/kafka-advanced-usage.md#kafka-streams-applications)

package/plugins/specweave-kafka/examples/kafka-streams-app/package.json

@@ -0,0 +1,11 @@
+{
+  "name": "kafka-streams-windowing",
+  "version": "1.0.0",
+  "scripts": {
+    "streams": "node windowed-aggregation.js"
+  },
+  "dependencies": {
+    "kafkajs": "^2.2.4",
+    "dotenv": "^16.3.1"
+  }
+}

package/plugins/specweave-kafka/examples/kafka-streams-app/windowed-aggregation.js

@@ -0,0 +1,66 @@
+require('dotenv').config();
+const { Kafka } = require('kafkajs');
+
+const kafka = new Kafka({
+  brokers: (process.env.KAFKA_BROKERS || 'localhost:9092').split(',')
+});
+
+const WINDOW_SIZE_MS = 60000; // 1 minute tumbling windows
+const windows = new Map(); // In-memory state (use RocksDB for production)
+
+async function run() {
+  const consumer = kafka.consumer({ groupId: 'streams-aggregator' });
+  const producer = kafka.producer();
+
+  await consumer.connect();
+  await producer.connect();
+
+  await consumer.subscribe({ topic: 'user-events', fromBeginning: false });
+
+  console.log('✓ Kafka Streams app started');
+  console.log('⏳ Aggregating events in 1-minute windows...\n');
+
+  await consumer.run({
+    eachMessage: async ({ message }) => {
+      const event = JSON.parse(message.value.toString());
+
+      // Calculate window start
+      const timestamp = event.timestamp || Date.now();
+      const windowStart = Math.floor(timestamp / WINDOW_SIZE_MS) * WINDOW_SIZE_MS;
+      const windowEnd = windowStart + WINDOW_SIZE_MS;
+
+      // Aggregate by userId + window
+      const windowKey = `${event.userId}:${windowStart}`;
+
+      if (!windows.has(event.userId)) {
+        windows.set(event.userId, new Map());
+      }
+
+      const userWindows = windows.get(event.userId);
+      const count = (userWindows.get(windowStart) || 0) + 1;
+      userWindows.set(windowStart, count);
+
+      // Emit windowed count
+      const result = {
+        userId: event.userId,
+        count,
+        windowStart: new Date(windowStart).toISOString(),
+        windowEnd: new Date(windowEnd).toISOString()
+      };
+
+      await producer.send({
+        topic: 'user-event-counts-1min',
+        messages: [{
+          key: windowKey,
+          value: JSON.stringify(result)
+        }]
+      });
+
+      console.log(`📊 Window [${result.windowStart}]: User ${event.userId} → ${count} events`);
+    }
+  });
+}
+
+process.on('SIGINT', async () => process.exit(0));
+
+run().catch(console.error);

package/plugins/specweave-kafka/examples/n8n-workflow/README.md

@@ -0,0 +1,54 @@
+# n8n Kafka Workflow Example
+
+**No-code Kafka workflows with n8n automation**
+
+Demonstrates integrating Kafka with n8n for event-driven workflows.
+
+## Features
+
+- **Kafka Trigger** - Start workflows from Kafka messages
+- **Enrichment** - Call external APIs to enrich data
+- **Filtering** - Route messages based on conditions
+- **Multi-Sink** - Send to database, Slack, email, etc.
+
+## Setup
+
+```bash
+# Start n8n
+docker-compose up -d
+
+# Access n8n UI
+open http://localhost:5678
+```
+
+## Import Workflow
+
+1. Open n8n at http://localhost:5678
+2. Click "Import from File"
+3. Select `kafka-to-slack.json`
+4. Configure Kafka credentials
+5. Activate workflow
+
+## Example Workflow
+
+```
+Kafka Consumer (user-events)
+  ↓
+Filter (event_type === 'purchase')
+  ↓
+HTTP Request (fetch user details)
+  ↓
+Slack (send notification)
+```
+
+## Use Cases
+
+- Alerts and notifications
+- Data pipeline orchestration
+- Event-driven automation
+- Integration with 3rd party services
+
+## Documentation
+
+- [n8n Documentation](https://docs.n8n.io/)
+- [n8n Kafka Node](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.kafka/)

package/plugins/specweave-kafka/examples/n8n-workflow/docker-compose.yml

@@ -0,0 +1,19 @@
+version: '3.8'
+
+services:
+  n8n:
+    image: n8nio/n8n:latest
+    container_name: n8n
+    ports:
+      - "5678:5678"
+    environment:
+      - N8N_BASIC_AUTH_ACTIVE=true
+      - N8N_BASIC_AUTH_USER=admin
+      - N8N_BASIC_AUTH_PASSWORD=admin
+    volumes:
+      - n8n-data:/home/node/.n8n
+      - ./workflows:/home/node/workflows
+
+volumes:
+  n8n-data:
+    driver: local