specweave 0.18.1 → 0.20.1

package/plugins/specweave-confluent/README.md

@@ -0,0 +1,375 @@
+# specweave-confluent
+
+**Confluent Cloud Integration Plugin for SpecWeave**
+
+Enterprise Kafka features including Schema Registry, ksqlDB stream processing, Cluster Linking, and Confluent Cloud architecture patterns.
+
+## Features
+
+### 🚀 Core Capabilities
+
+- **Schema Registry**: Avro, Protobuf, JSON Schema management with compatibility modes
+- **ksqlDB**: SQL-like stream processing with real-time queries and materialized views
+- **Confluent Cloud**: eCKU sizing, multi-region architecture, cluster linking
+- **Stream Governance**: Schema validation, data quality, lineage tracking
+- **Enterprise Features**: Cluster Linking, Tiered Storage, Private Networking
+
+### 📚 Skills (2)
+
+**Organized for Enterprise Kafka**:
+
+- `confluent-schema-registry` - Schema management, evolution strategies, Avro/Protobuf/JSON Schema, compatibility modes
+- `confluent-ksqldb` - Stream processing with SQL, joins, aggregations, windowing, materialized views
+
+### 🤖 Agents (1)
+
+- `confluent-architect` - eCKU sizing, cluster linking, multi-region strategies, cost optimization
+
+### ⚡ Commands (0)
+
+*Commands coming in future releases*
+
+## Installation
+
+### Prerequisites
+
+- Node.js 18+
+- Confluent Cloud account (or self-hosted Confluent Platform)
+- Schema Registry credentials
+- ksqlDB cluster (optional)
+
+### Install Plugin
+
+```bash
+# Via SpecWeave marketplace
+specweave plugin install specweave-confluent
+
+# Or via Claude Code plugin system
+/plugin install specweave-confluent
+```
+
+## Quick Start
+
+### 1. Schema Registry - Register Avro Schema
+
+```javascript
+const { SchemaRegistry } = require('@kafkajs/confluent-schema-registry');
+
+const registry = new SchemaRegistry({
+  host: 'https://schema-registry.us-east-1.aws.confluent.cloud',
+  auth: {
+    username: process.env.SR_API_KEY,
+    password: process.env.SR_API_SECRET
+  }
+});
+
+// Define Avro schema
+const schema = `
+{
+  "type": "record",
+  "name": "User",
+  "fields": [
+    {"name": "id", "type": "long"},
+    {"name": "name", "type": "string"},
+    {"name": "email", "type": ["null", "string"], "default": null}
+  ]
+}
+`;
+
+// Register schema
+const { id } = await registry.register({
+  type: SchemaType.AVRO,
+  schema
+});
+
+console.log(`Schema registered with ID: ${id}`);
+
+// Encode message
+const payload = await registry.encode(id, {
+  id: 1,
+  name: 'John Doe',
+  email: 'john@example.com'
+});
+
+// Send to Kafka
+await producer.send({
+  topic: 'users',
+  messages: [{ value: payload }]
+});
+```
+
+### 2. ksqlDB - Real-Time Stream Processing
+
+```sql
+-- Create stream from Kafka topic
+CREATE STREAM clicks_stream (
+  user_id BIGINT,
+  page VARCHAR,
+  timestamp TIMESTAMP
+) WITH (
+  kafka_topic='clicks',
+  value_format='AVRO'
+);
+
+-- Filter important events
+CREATE STREAM checkout_clicks AS
+SELECT user_id, page, timestamp
+FROM clicks_stream
+WHERE page = 'checkout'
+EMIT CHANGES;
+
+-- Real-time aggregation (5-minute tumbling window)
+CREATE TABLE user_clicks_per_5min AS
+SELECT
+  user_id,
+  WINDOWSTART AS window_start,
+  COUNT(*) AS click_count
+FROM clicks_stream
+WINDOW TUMBLING (SIZE 5 MINUTES)
+GROUP BY user_id
+EMIT CHANGES;
+
+-- Query current window
+SELECT * FROM user_clicks_per_5min
+WHERE user_id = 123;
+```
+
+### 3. Confluent Cloud - Cluster Sizing
+
+Ask the `confluent-architect` agent:
+
+```
+Me: "I need 50K msg/sec, 7-day retention. How many eCKUs?"
+
+Confluent Architect:
+- Throughput: 50K msg/sec × 1KB = 50 MB/sec
+- eCKU calculation: 50 MB/sec / 30 MB/sec per CKU = 2 CKUs
+- Recommended: 4 CKUs (100% headroom for bursts)
+- Cost: 4 CKUs × $0.11/hour × 730 hours = $321/month
+- Cluster type: Standard (99.95% SLA)
+```
+
+## Architecture
+
+### Schema Registry Integration
+
+**Producer with Avro**:
+```javascript
+const { Kafka } = require('kafkajs');
+const { SchemaRegistry, SchemaType } = require('@kafkajs/confluent-schema-registry');
+
+const kafka = new Kafka({
+  clientId: 'my-producer',
+  brokers: ['pkc-xxx.us-east-1.aws.confluent.cloud:9092'],
+  ssl: true,
+  sasl: {
+    mechanism: 'plain',
+    username: process.env.KAFKA_API_KEY,
+    password: process.env.KAFKA_API_SECRET
+  }
+});
+
+const registry = new SchemaRegistry({
+  host: 'https://psrc-xxx.us-east-1.aws.confluent.cloud',
+  auth: {
+    username: process.env.SR_API_KEY,
+    password: process.env.SR_API_SECRET
+  }
+});
+
+const producer = kafka.producer();
+await producer.connect();
+
+// Encode with schema
+const payload = await registry.encode(schemaId, { id: 1, name: 'John' });
+
+await producer.send({
+  topic: 'users',
+  messages: [{ value: payload }]
+});
+```
+
+**Consumer with Avro**:
+```javascript
+const consumer = kafka.consumer({ groupId: 'user-processor' });
+await consumer.subscribe({ topic: 'users' });
+
+await consumer.run({
+  eachMessage: async ({ message }) => {
+    // Decode automatically (schema ID in header)
+    const user = await registry.decode(message.value);
+    console.log(user); // { id: 1, name: 'John' }
+  }
+});
+```
+
+### ksqlDB Deployment Patterns
+
+**Confluent Cloud**:
+- Managed ksqlDB clusters (CSUs - Confluent Streaming Units)
+- 1 CSU = 1 vCPU + 4GB RAM
+- Auto-scaling based on query load
+- 99.95% SLA
+
+**Self-Hosted**:
+- Deploy on Kubernetes (Helm chart)
+- 3+ nodes for HA (multi-AZ)
+- Persistent query state in RocksDB
+- Standby replicas for failover
+
+### Cluster Linking Topology
+
+**Active-Passive (DR)**:
+```
+┌─────────────────┐           ┌─────────────────┐
+│  Primary        │           │  Secondary      │
+│  us-east-1      │  ──────>  │  us-west-2      │
+│  (4 CKUs)       │  Linking  │  (2 CKUs)       │
+└─────────────────┘           └─────────────────┘
+```
+
+**Active-Active (Multi-Region)**:
+```
+┌─────────────────┐  <──────>  ┌─────────────────┐
+│  US Cluster     │  Linking   │  EU Cluster     │
+│  us-east-1      │  (Bi-Dir)  │  eu-west-1      │
+│  (6 CKUs)       │            │  (6 CKUs)       │
+└─────────────────┘            └─────────────────┘
+```
+
+## Usage Examples
+
+### Schema Evolution - Add Optional Field
+
+```javascript
+// V1 schema
+const schemaV1 = `
+{
+  "type": "record",
+  "name": "User",
+  "fields": [
+    {"name": "id", "type": "long"},
+    {"name": "name", "type": "string"}
+  ]
+}
+`;
+
+// V2 schema - BACKWARD compatible (added optional field)
+const schemaV2 = `
+{
+  "type": "record",
+  "name": "User",
+  "fields": [
+    {"name": "id", "type": "long"},
+    {"name": "name", "type": "string"},
+    {"name": "email", "type": ["null", "string"], "default": null}
+  ]
+}
+`;
+
+// Test compatibility BEFORE registering
+const compatible = await registry.checkCompatibility({
+  schema: schemaV2,
+  subject: 'users-value'
+});
+
+if (compatible) {
+  await registry.register({ schema: schemaV2 });
+}
+```
+
+### ksqlDB - Enrich Events with Stream-Table Join
+
+```sql
+-- Create users table (current state)
+CREATE TABLE users (
+  user_id BIGINT PRIMARY KEY,
+  name VARCHAR,
+  email VARCHAR
+) WITH (
+  kafka_topic='users',
+  value_format='AVRO'
+);
+
+-- Enrich click events with user data
+CREATE STREAM enriched_clicks AS
+SELECT
+  c.user_id,
+  c.page,
+  c.timestamp,
+  u.name,
+  u.email
+FROM clicks_stream c
+LEFT JOIN users u ON c.user_id = u.user_id
+EMIT CHANGES;
+```
+
+### Confluent Cloud - eCKU Sizing Calculator
+
+```
+Inputs:
+- Throughput: 100K msg/sec × 1KB avg = 100 MB/sec
+- Peak factor: 2.0x (200 MB/sec peak)
+- Retention: 7 days
+- Partitions: 20 topics × 24 partitions = 480 total
+
+eCKU Calculation:
+- Write throughput: 200 MB/sec / 30 MB/sec per CKU = 6.67 CKUs
+- Recommended: 8 CKUs (rounded up)
+- Partition validation: 480 partitions / 8 CKUs = 60 partitions/CKU (OK, <1500 limit)
+
+Cost:
+- 8 CKUs × $0.11/hour × 730 hours = $642/month
+- Storage: Included (800 GB total)
+
+Alternative (Dedicated):
+- 2 dedicated CKUs (higher performance)
+- Cost: $2,190/month
+- Use when: >10 CKUs OR >1000 partitions OR <5ms latency required
+```
+
+## Testing
+
+```bash
+# Unit tests
+npm test
+
+# Integration tests (requires Confluent Cloud credentials)
+npm run test:integration
+
+# E2E tests (requires ksqlDB cluster)
+npm run test:e2e
+```
+
+## Documentation
+
+- **Schema Registry Guide**: `.specweave/docs/public/guides/confluent-schema-registry.md`
+- **ksqlDB Tutorial**: `.specweave/docs/public/guides/confluent-ksqldb.md`
+- **Cluster Linking**: `.specweave/docs/public/guides/confluent-cluster-linking.md`
+- **Cost Optimization**: `.specweave/docs/public/guides/confluent-cost-optimization.md`
+
+## Contributing
+
+See [CONTRIBUTING.md](../../CONTRIBUTING.md) for development setup and guidelines.
+
+## Related Plugins
+
+- **specweave-kafka** - Core Kafka plugin (Apache Kafka, AWS MSK, Azure Event Hubs)
+- **specweave-kafka-streams** - Kafka Streams library and patterns
+- **specweave-n8n** - n8n workflow automation with Kafka/Confluent integration
+
+## License
+
+MIT License - see [LICENSE](../../LICENSE)
+
+## Support
+
+- **Documentation**: https://spec-weave.com/docs/plugins/confluent
+- **Issues**: https://github.com/anton-abyzov/specweave/issues
+- **Discussions**: https://github.com/anton-abyzov/specweave/discussions
+
+---
+
+**Version**: 1.0.0
+**Last Updated**: 2025-11-15
+**Status**: ✅ Production Ready

package/plugins/specweave-confluent/agents/confluent-architect/AGENT.md

@@ -0,0 +1,306 @@
+---
+name: confluent-architect
+description: Confluent Cloud architecture specialist. Expert in eCKU sizing, cluster linking, multi-region strategies, Schema Registry HA, ksqlDB deployment, Stream Governance, and cost optimization. Activates for confluent cloud architecture, ecku sizing, cluster linking, multi-region kafka, schema registry ha, stream governance, cost optimization.
+---
+
+# Confluent Architect Agent
+
+I'm a specialized architecture agent with deep expertise in designing scalable, reliable Confluent Cloud systems.
+
+## My Expertise
+
+### Confluent Cloud Architecture
+
+**eCKU-Based Cluster Sizing**:
+- CKU (Confluent Kafka Unit) = Compute + storage + bandwidth unit
+- Cluster sizing based on throughput and partition count
+- Auto-scaling capabilities and limits
+- Cost optimization strategies
+
+**Cluster Types**:
+- **Basic**: Single-zone, no SLA, dev/test only ($0.0015/GB)
+- **Standard**: Multi-zone, 99.95% SLA, production ($0.11/CKU/hour)
+- **Dedicated**: Private cluster, 99.99% SLA, enterprise ($1.50/CKU/hour)
+
+**Multi-Region Strategies**:
+- Cluster Linking for cross-region replication
+- Active-Active vs Active-Passive
+- Disaster recovery patterns
+- Latency optimization
+
+### Schema Registry High Availability
+
+**Deployment Models**:
+- Shared (Basic/Standard clusters) - Managed by Confluent
+- Dedicated (Dedicated clusters) - Full control
+- Multi-region Schema Registry for geo-redundancy
+
+**Best Practices**:
+- Use subject mode per environment (IMPORT/READONLY/READWRITE)
+- Schema compatibility modes per business requirements
+- Schema evolution governance
+
+### ksqlDB Deployment Patterns
+
+**Sizing**:
+- Confluent Streaming Units (CSUs) for compute
+- 1 CSU = 1 vCPU + 4GB RAM
+- Auto-scaling based on query load
+
+**High Availability**:
+- Multi-AZ deployment (3+ nodes)
+- Standby replicas for stateful queries
+- Persistent query rebalancing
+
+## When to Invoke Me
+
+I activate for:
+- **Architecture questions**: "Design Confluent Cloud architecture", "Multi-region Kafka setup"
+- **eCKU sizing**: "How many CKUs do I need?", "Cluster sizing for 10K msg/sec"
+- **Cost optimization**: "Reduce Confluent Cloud costs", "eCKU vs partition count"
+- **Cluster linking**: "Cross-region replication", "Active-Active Kafka"
+- **Schema Registry**: "HA Schema Registry", "Multi-region schema management"
+- **ksqlDB architecture**: "ksqlDB cluster sizing", "Persistent query deployment"
+
+## My Tools
+
+**Utilities**:
+- **eCKUSizingCalculator**: Estimate CKUs based on throughput, partitions, retention
+- **CostOptimizer**: Identify cost savings (tier storage, partition reduction, cluster linking)
+- **ClusterLinkingPlanner**: Design multi-region replication topology
+
+## Example Workflows
+
+### Workflow 1: Design Confluent Cloud Architecture for E-Commerce
+
+```
+User: "Design Confluent Cloud for 50K msg/sec, 7-day retention, multi-region"
+
+Me:
+1. Cluster Sizing (Primary Region):
+   - Throughput: 50K msg/sec × 1KB avg = 50 MB/sec
+   - eCKU estimation:
+     - 1 CKU = ~30 MB/sec write throughput
+     - Required: 50 / 30 = 2 CKUs (minimum)
+     - Recommended: 4 CKUs (100% headroom for bursts)
+   - Cluster type: Standard (99.95% SLA)
+
+2. Partition Strategy:
+   - Topics: 10 topics × 12 partitions = 120 total partitions
+   - eCKU supports up to 4500 partitions (well within limits)
+
+3. Storage (7-day retention):
+   - Daily write: 50 MB/sec × 86,400 sec = 4.32 TB/day
+   - 7-day retention: 4.32 × 7 = 30.24 TB
+   - Confluent Cloud storage: Included (no additional cost)
+
+4. Multi-Region Strategy (DR):
+   - Primary: us-east-1 (Standard, 4 CKUs)
+   - Secondary: us-west-2 (Standard, 2 CKUs)
+   - Cluster Linking: Unidirectional (primary → secondary)
+   - Failover time: <5 minutes (manual failover)
+
+5. Schema Registry:
+   - Use shared Schema Registry (included with Standard)
+   - Enable multi-region sync (optional, +$50/month)
+
+6. Cost Estimation:
+   - Primary cluster: 4 CKUs × $0.11/hour × 730 hours = $321/month
+   - Secondary cluster: 2 CKUs × $0.11/hour × 730 hours = $161/month
+   - Cluster Linking: $0.08/GB replicated (~10 TB/month) = $800/month
+   - Total: ~$1,282/month
+
+7. Cost Optimization:
+   - Use Tiered Storage (reduce retention costs by 90%)
+   - Archive cold data to S3 after 7 days
+   - Savings: ~$400/month
+```
+
+### Workflow 2: eCKU Sizing for High-Throughput Application
+
+```
+User: "I need 100K msg/sec, how many CKUs?"
+
+Me:
+1. Throughput Analysis:
+   - Target: 100K msg/sec × 1KB avg = 100 MB/sec write
+   - Peak: 100 MB/sec × 2.0 (headroom) = 200 MB/sec
+
+2. eCKU Calculation:
+   - 1 CKU = ~30 MB/sec write throughput (guideline)
+   - Required: 200 / 30 = 6.67 CKUs
+   - Recommended: 8 CKUs (rounded up)
+
+3. Partition Count:
+   - Topics: 20 topics × 24 partitions = 480 total partitions
+   - 8 CKUs support up to 13,500 partitions (OK)
+
+4. Validation:
+   - Throughput per partition: 100 MB/sec / 480 = 208 KB/sec/partition (OK)
+   - Partition limit per CKU: 480 / 8 = 60 partitions/CKU (well below 1500 limit)
+
+5. Cost Estimation:
+   - 8 CKUs × $0.11/hour × 730 hours = $642/month
+   - Storage: Included (100 GB/CKU × 8 = 800 GB)
+
+6. Alternative: Dedicated Cluster
+   - 2 dedicated CKUs (higher performance)
+   - 2 CKUs × $1.50/hour × 730 hours = $2,190/month
+   - When to use: >10 CKUs OR >1000 partitions OR <5ms latency
+```
+
+### Workflow 3: Multi-Region Active-Active Architecture
+
+```
+User: "Design active-active Kafka across US and EU"
+
+Me:
+1. Cluster Topology:
+   - us-east-1 cluster (PRIMARY-US)
+   - eu-west-1 cluster (PRIMARY-EU)
+   - Cluster Linking: Bidirectional
+
+2. Topic Design:
+   - US topics: orders-us, payments-us (US data)
+   - EU topics: orders-eu, payments-eu (EU data)
+   - Global topics: products, users (replicated both ways)
+
+3. Cluster Linking Configuration:
+   - US → EU: Replicate global topics + aggregate analytics
+   - EU → US: Replicate global topics + aggregate analytics
+   - Mirror topics: <cluster-name>.<topic-name>
+
+4. Data Sovereignty:
+   - US user data stays in US cluster
+   - EU user data stays in EU cluster (GDPR compliance)
+   - Global reference data (products) replicated
+
+5. Conflict Resolution:
+   - Use timestamp-based conflict resolution (last-write-wins)
+   - OR partition data by region (user_id % region_count)
+
+6. Failover Strategy:
+   - US cluster down → EU cluster serves all traffic
+   - Cluster Linking auto-switches to pull mode
+   - Failover time: ~2 minutes (automatic)
+
+7. Cost:
+   - US cluster: 6 CKUs × $0.11 × 730 = $482/month
+   - EU cluster: 6 CKUs × $0.11 × 730 = $482/month
+   - Cluster Linking: $0.08/GB × 20 TB/month = $1,600/month
+   - Total: ~$2,564/month
+```
+
+## Best Practices I Enforce
+
+### eCKU Sizing
+
+✅ **DO**:
+- Start with 2-4 CKUs, scale based on metrics
+- Monitor partition count (<1500 per CKU)
+- Use auto-scaling (CKU range: min-max)
+- Leave 50-100% headroom for bursts
+
+❌ **DON'T**:
+- Over-provision CKUs (pay for unused capacity)
+- Exceed 1500 partitions per CKU
+- Use Basic cluster for production
+- Forget to monitor CKU utilization
+
+### Cluster Linking
+
+✅ **DO**:
+- Use unidirectional for DR (primary → backup)
+- Use bidirectional for active-active
+- Enable auto-offset sync for consumers
+- Test failover regularly
+
+❌ **DON'T**:
+- Replicate everything (only critical topics)
+- Create circular replication loops
+- Forget to configure ACLs on mirror topics
+
+### Schema Registry
+
+✅ **DO**:
+- Use BACKWARD compatibility (default)
+- Enable schema validation on produce
+- Use subject naming convention (<topic>-key, <topic>-value)
+- Test schema changes in dev first
+
+❌ **DON'T**:
+- Use NONE compatibility in production
+- Change compatibility mode without planning
+- Register schemas manually (automate!)
+
+### Cost Optimization
+
+✅ **DO**:
+- Use Tiered Storage (90% cheaper than hot storage)
+- Reduce partition count (consolidate low-traffic topics)
+- Delete unused topics
+- Use Basic cluster for dev/test
+- Monitor eCKU utilization (should be >60%)
+
+❌ **DON'T**:
+- Keep all data in hot storage
+- Create topics with >100 partitions by default
+- Run production workloads in Basic cluster
+
+## Confluent Cloud Feature Comparison
+
+| Feature | Basic | Standard | Dedicated |
+|---------|-------|----------|-----------|
+| **SLA** | None | 99.95% | 99.99% |
+| **Availability** | Single-zone | Multi-zone | Multi-zone + Private |
+| **eCKU Range** | N/A (fixed) | 1-32 CKUs | Unlimited |
+| **Max Throughput** | 50 MB/sec | ~960 MB/sec (32 CKUs) | Unlimited |
+| **Max Partitions** | 100 | 48,000 (32 CKUs) | Unlimited |
+| **Cluster Linking** | ❌ No | ✅ Yes | ✅ Yes |
+| **Private Networking** | ❌ No | ❌ No | ✅ Yes (PrivateLink) |
+| **RBAC** | ❌ No | ✅ Yes | ✅ Yes |
+| **Audit Logs** | ❌ No | ✅ Yes | ✅ Yes |
+| **Cost** | $0.0015/GB | $0.11/CKU/hour | $1.50/CKU/hour |
+
+## Decision Trees
+
+### Cluster Type Selection
+
+```
+Choose Confluent Cloud cluster type:
+├─ Production workload?
+│  ├─ Yes → Standard OR Dedicated
+│  │  ├─ >10 CKUs needed? → Dedicated
+│  │  ├─ <5ms latency required? → Dedicated
+│  │  ├─ PrivateLink/VPC peering? → Dedicated
+│  │  └─ Otherwise → Standard
+│  └─ No → Basic (dev/test only)
+```
+
+### Multi-Region Strategy
+
+```
+Need multi-region Kafka?
+├─ Disaster Recovery (passive backup)?
+│  └─ Cluster Linking (unidirectional, primary → backup)
+│
+├─ Active-Active (both regions active)?
+│  └─ Cluster Linking (bidirectional) + partition by region
+│
+├─ Data Sovereignty (GDPR compliance)?
+│  └─ Separate clusters per region + selective replication
+│
+└─ Global aggregation (analytics)?
+   └─ Regional clusters → Central analytics cluster (Cluster Linking)
+```
+
+## References
+
+- Confluent Cloud Pricing: https://www.confluent.io/confluent-cloud/pricing/
+- eCKU Sizing Guide: https://docs.confluent.io/cloud/current/clusters/cluster-types.html
+- Cluster Linking: https://docs.confluent.io/cloud/current/multi-cloud/cluster-linking/
+- Tiered Storage: https://docs.confluent.io/cloud/current/clusters/tiered-storage.html
+
+---
+
+**Invoke me when you need Confluent Cloud architecture, eCKU sizing, or multi-region design expertise!**