cfsa-antigravity 3.0.2 → 3.1.0

package/template/.codex/skill-library/stack/messaging/kafka/SKILL.md

@@ -0,0 +1,235 @@
+---
+name: kafka
+description: "Expert Apache Kafka guide covering topic design (partitioning, retention, compaction), producer patterns (idempotency, batching, acks), consumer groups (rebalancing, offset management, exactly-once), Kafka Streams, Connect, Schema Registry (Avro/Protobuf), and operational best practices. Use when building event-driven architectures, streaming pipelines, or pub/sub systems with Kafka."
+version: 1.0.0
+---
+
+# Apache Kafka Expert Guide
+
+> Use this skill when designing event-driven architectures, streaming data pipelines, or pub/sub messaging with Apache Kafka.
+
+## When to Use This Skill
+
+- Building event-driven microservices
+- Designing streaming ETL / data pipelines
+- Implementing pub/sub messaging between services
+- Processing high-throughput real-time events (logs, metrics, clickstreams)
+- CQRS / event sourcing architectures
+
+## When NOT to Use This Skill
+
+- Simple task queues → use BullMQ or SQS
+- Low-throughput request/reply → use REST or gRPC
+- Real-time browser push → use WebSockets / Socket.io
+- Small projects with < 1K events/sec → consider simpler alternatives (Redis Streams, NATS)
+
+---
+
+## 1. Topic Design (CRITICAL)
+
+### Naming Convention
+
+```
+<domain>.<entity>.<event-type>
+
+# Examples
+orders.order.created
+users.user.profile-updated
+payments.payment.completed
+inventory.stock.adjusted
+```
+
+### Partitioning Strategy
+
+| Strategy | When | Key |
+|----------|------|-----|
+| By entity ID | Ordered per-entity processing | `user_id`, `order_id` |
+| By tenant | Multi-tenant isolation | `tenant_id` |
+| Round-robin | Max parallelism, no ordering needed | `null` key |
+| By region | Data locality | `region_code` |
+
+```
+# Partition count guidelines
+- Start with 2× expected consumer count
+- Max ~50 partitions per topic for most use cases
+- More partitions = more parallelism but higher overhead
+```
+
+### Retention & Compaction
+
+| Policy | Use Case | Config |
+|--------|----------|--------|
+| Time-based | Event logs, metrics | `retention.ms=604800000` (7 days) |
+| Size-based | Bounded storage | `retention.bytes=1073741824` (1GB) |
+| Compacted | Latest-state snapshots | `cleanup.policy=compact` |
+| Compact + delete | State with TTL | `cleanup.policy=compact,delete` |
+
+```properties
+# Compacted topic — keeps latest value per key
+cleanup.policy=compact
+min.compaction.lag.ms=3600000
+delete.retention.ms=86400000
+```
+
+---
+
+## 2. Producers (CRITICAL)
+
+### Idempotent Producer
+
+```properties
+# ✅ Always enable idempotency
+enable.idempotence=true
+acks=all
+retries=2147483647
+max.in.flight.requests.per.connection=5
+```
+
+### Batching
+
+```properties
+# ✅ Batch for throughput
+batch.size=65536          # 64KB per partition batch
+linger.ms=10              # Wait up to 10ms to fill batch
+compression.type=lz4      # Compress batches
+buffer.memory=67108864    # 64MB producer buffer
+```
+
+### Error Handling
+
+```
+Producer sends → acks=all → all ISR replicas confirm
+                → retries on transient errors (automatic with idempotency)
+                → DLQ (Dead Letter Queue) for permanent failures
+
+# Key rule: never silently drop messages
+# Log failures, send to DLQ, alert on threshold
+```
+
+---
+
+## 3. Consumers (CRITICAL)
+
+### Consumer Group Patterns
+
+```
+# ✅ One consumer group per logical application
+group.id=order-service
+
+# ✅ Consumers in group = partition count for max parallelism
+# 6 partitions → 6 consumers in group (max useful)
+# Extra consumers sit idle
+```
+
+### Offset Management
+
+| Strategy | When | Risk |
+|----------|------|------|
+| Auto-commit | Simple consumers, at-least-once OK | Duplicates on crash |
+| Manual commit after processing | At-least-once guaranteed | Duplicates on crash before commit |
+| Transactional (exactly-once) | Financial / critical processing | Higher latency |
+
+```properties
+# ✅ Manual commit for reliability
+enable.auto.commit=false
+auto.offset.reset=earliest
+
+# Process → commit pattern:
+# 1. poll()
+# 2. process batch
+# 3. commitSync()
+```
+
+### Rebalancing
+
+```properties
+# ✅ Cooperative rebalancing (no stop-the-world)
+partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
+
+# ✅ Heartbeat tuning
+session.timeout.ms=30000
+heartbeat.interval.ms=10000
+max.poll.interval.ms=300000
+max.poll.records=500
+```
+
+---
+
+## 4. Schema Registry
+
+### Avro Schema Evolution
+
+```json
+{
+  "type": "record",
+  "name": "OrderCreated",
+  "namespace": "com.example.orders",
+  "fields": [
+    {"name": "order_id", "type": "string"},
+    {"name": "user_id", "type": "string"},
+    {"name": "total_cents", "type": "long"},
+    {"name": "currency", "type": "string", "default": "USD"},
+    {"name": "created_at", "type": "long", "logicalType": "timestamp-millis"}
+  ]
+}
+```
+
+| Evolution Rule | Allowed |
+|---------------|---------|
+| Add field with default | ✅ Backward compatible |
+| Remove field with default | ✅ Forward compatible |
+| Rename field | ❌ Breaking |
+| Change field type | ❌ Breaking |
+
+---
+
+## 5. Kafka Streams (Stateful Processing)
+
+```
+# Topology pattern
+Source Topic → Filter → Map → GroupByKey → Aggregate → Sink Topic
+
+# Use cases:
+# - Real-time aggregations (counts, sums, averages)
+# - Windowed operations (tumbling, hopping, session windows)
+# - Stream-table joins (enrich events with reference data)
+# - Exactly-once processing (with Kafka transactions)
+```
+
+---
+
+## 6. Kafka Connect
+
+```
+# Source connectors: DB → Kafka
+- Debezium (CDC from PostgreSQL, MySQL, MongoDB)
+- JDBC Source
+
+# Sink connectors: Kafka → external
+- Elasticsearch Sink
+- S3 Sink (Parquet/JSON)
+- JDBC Sink
+- BigQuery Sink
+```
+
+---
+
+## 7. Common Anti-Patterns
+
+1. **One giant topic for everything** — separate by domain and entity
+2. **Too many partitions** — start small (2× consumers), scale up when needed
+3. **No schema registry** — schema drift causes silent data corruption
+4. **Auto-commit with long processing** — commits before processing completes → data loss
+5. **Consuming without consumer groups** — lose offset tracking and rebalancing
+6. **Not setting `acks=all`** — risking data loss on broker failure
+7. **Synchronous produce in request path** — batch/async, or accept latency
+8. **No DLQ** — poison messages block the entire partition
+
+---
+
+## References
+
+- [Apache Kafka Documentation](https://kafka.apache.org/documentation/)
+- [Confluent Developer](https://developer.confluent.io/)
+- [Schema Registry](https://docs.confluent.io/platform/current/schema-registry/)
+- [Kafka Streams](https://kafka.apache.org/documentation/streams/)

package/template/.codex/skill-library/stack/messaging/nats/SKILL.md

@@ -0,0 +1,147 @@
+---
+name: nats
+description: "NATS messaging patterns covering pub/sub, request/reply, JetStream persistence, key-value store, and subject-based routing. Use when implementing cloud-native messaging with NATS."
+version: 1.0.0
+source: self
+date_added: "2026-03-14"
+---
+
+# NATS
+
+Cloud-native messaging system. Lightweight, high-performance pub/sub with optional persistence via JetStream. Subject-based routing with wildcard subscriptions.
+
+## When to Use
+
+- Need lightweight, high-throughput pub/sub between services
+- Building cloud-native microservices (Kubernetes-native)
+- Need request/reply patterns without HTTP overhead
+- Want built-in key-value and object store (JetStream)
+
+## When NOT to Use
+
+- Need complex exchange/routing logic (use RabbitMQ)
+- Need long-term event replay/streaming (use Kafka)
+- Need managed cloud queue without server (use SQS)
+
+## Setup
+
+```bash
+npm install nats
+```
+
+```typescript
+import { connect, StringCodec } from 'nats';
+
+const nc = await connect({ servers: process.env.NATS_URL ?? 'nats://localhost:4222' });
+const sc = StringCodec(); // Encode/decode messages
+
+// JSON codec helper
+const jc = {
+  encode: (obj: unknown) => sc.encode(JSON.stringify(obj)),
+  decode: (data: Uint8Array) => JSON.parse(sc.decode(data)),
+};
+```
+
+## Pub/Sub
+
+```typescript
+// Publish
+nc.publish('orders.created', jc.encode({ orderId: '1234', total: 99.99 }));
+
+// Subscribe
+const sub = nc.subscribe('orders.created');
+for await (const msg of sub) {
+  const order = jc.decode(msg.data);
+  console.log(`Order created: ${order.orderId}`);
+}
+
+// Wildcard subscriptions
+nc.subscribe('orders.*');       // orders.created, orders.updated — one level
+nc.subscribe('orders.>');       // orders.created, orders.us.pending — any depth
+```
+
+## Request/Reply
+
+```typescript
+// Service (responder)
+const sub = nc.subscribe('api.users.get');
+for await (const msg of sub) {
+  const { userId } = jc.decode(msg.data);
+  const user = await db.users.findById(userId);
+  msg.respond(jc.encode(user));
+}
+
+// Client (requester)
+const response = await nc.request('api.users.get', jc.encode({ userId: '123' }), {
+  timeout: 5000,
+});
+const user = jc.decode(response.data);
+```
+
+## JetStream (Persistent Messaging)
+
+```typescript
+const js = nc.jetstream();
+const jsm = await nc.jetstreamManager();
+
+// Create stream
+await jsm.streams.add({
+  name: 'ORDERS',
+  subjects: ['orders.>'],
+  retention: 'limits',     // 'limits', 'interest', or 'workqueue'
+  max_msgs: 100000,
+  max_age: 7 * 24 * 60 * 60 * 1_000_000_000, // 7 days (nanoseconds)
+});
+
+// Publish to stream
+await js.publish('orders.created', jc.encode({ orderId: '1234' }));
+
+// Durable consumer
+const consumer = await js.consumers.get('ORDERS', 'order-processor');
+const messages = await consumer.consume();
+for await (const msg of messages) {
+  const order = jc.decode(msg.data);
+  await processOrder(order);
+  msg.ack();
+}
+```
+
+## Key-Value Store
+
+```typescript
+const kv = await js.views.kv('config');
+
+// Put
+await kv.put('feature.new-checkout', sc.encode('true'));
+
+// Get
+const entry = await kv.get('feature.new-checkout');
+const value = entry ? sc.decode(entry.value) : null;
+
+// Watch for changes
+const watch = await kv.watch();
+for await (const entry of watch) {
+  console.log(`${entry.key} = ${sc.decode(entry.value)}`);
+}
+```
+
+## Graceful Shutdown
+
+```typescript
+process.on('SIGTERM', async () => {
+  await nc.drain(); // Finish processing, then close
+});
+```
+
+## Anti-Patterns
+
+| Don't | Do |
+|-------|-----|
+| Use core NATS for messages that must not be lost | Use JetStream for persistent, acknowledged messaging |
+| Create a new connection per request | Reuse a single connection, create subscriptions as needed |
+| Use `close()` for graceful shutdown | Use `drain()` to finish in-flight messages before closing |
+| Forget to `ack()` JetStream messages | Always ack or nak — unacked messages redeliver after timeout |
+| Use overly specific subjects | Design hierarchical subjects (e.g., `orders.us.created`) for flexibility |
+| Block the async iterator with slow processing | Process messages concurrently or offload heavy work |
+| Skip error handling on `request()` | Set timeout and handle `NatsError` for request/reply |
+| Store large binary blobs in KV | Use JetStream Object Store for files, KV for config/state |

package/template/.codex/skill-library/stack/messaging/rabbitmq/SKILL.md

@@ -0,0 +1,178 @@
+---
+name: rabbitmq
+description: "RabbitMQ messaging patterns covering exchanges, queues, routing, pub/sub, RPC, dead-letter queues, and connection management. Use when implementing message queuing with RabbitMQ."
+version: 1.0.0
+source: self
+date_added: "2026-03-14"
+---
+
+# RabbitMQ
+
+Open-source message broker implementing AMQP 0-9-1. Reliable message delivery with exchanges, queues, routing, acknowledgments, and dead-letter handling.
+
+## When to Use
+
+- Need reliable async message passing between services
+- Building event-driven microservices
+- Need complex routing (topic, headers, fan-out)
+- Want durable message persistence with delivery guarantees
+
+## When NOT to Use
+
+- Need a simple job queue (use BullMQ with Redis)
+- Need event streaming with replay (use Kafka)
+- All services are serverless/stateless (use SQS or cloud pub/sub)
+
+## Setup
+
+### Installation
+
+```bash
+npm install amqplib
+npm install @types/amqplib  # TypeScript types
+```
+
+### Connection
+
+```typescript
+import amqp, { Connection, Channel } from 'amqplib';
+
+let connection: Connection;
+let channel: Channel;
+
+async function connect() {
+  connection = await amqp.connect(process.env.RABBITMQ_URL ?? 'amqp://localhost');
+  channel = await connection.createChannel();
+
+  // Prefetch: process one message at a time
+  await channel.prefetch(1);
+
+  // Handle connection errors
+  connection.on('error', (err) => console.error('RabbitMQ connection error:', err));
+  connection.on('close', () => setTimeout(connect, 5000)); // Reconnect
+}
+```
+
+## Patterns
+
+### Work Queue (Task Distribution)
+
+```typescript
+// Producer
+async function enqueueTask(queue: string, task: object) {
+  await channel.assertQueue(queue, { durable: true });
+  channel.sendToQueue(queue, Buffer.from(JSON.stringify(task)), {
+    persistent: true,   // Survive broker restart
+    contentType: 'application/json',
+  });
+}
+
+// Consumer
+async function consumeTasks(queue: string, handler: (task: any) => Promise<void>) {
+  await channel.assertQueue(queue, { durable: true });
+  channel.consume(queue, async (msg) => {
+    if (!msg) return;
+    try {
+      const task = JSON.parse(msg.content.toString());
+      await handler(task);
+      channel.ack(msg);       // Acknowledge successful processing
+    } catch (err) {
+      channel.nack(msg, false, false); // Reject, don't requeue (goes to DLQ if configured)
+    }
+  });
+}
+```
+
+### Publish/Subscribe (Fan-Out)
+
+```typescript
+// Publisher
+async function publish(exchange: string, event: object) {
+  await channel.assertExchange(exchange, 'fanout', { durable: true });
+  channel.publish(exchange, '', Buffer.from(JSON.stringify(event)), {
+    persistent: true,
+    contentType: 'application/json',
+  });
+}
+
+// Subscriber
+async function subscribe(exchange: string, handler: (event: any) => Promise<void>) {
+  await channel.assertExchange(exchange, 'fanout', { durable: true });
+  const { queue } = await channel.assertQueue('', { exclusive: true }); // Auto-delete queue
+  await channel.bindQueue(queue, exchange, '');
+  channel.consume(queue, async (msg) => {
+    if (!msg) return;
+    const event = JSON.parse(msg.content.toString());
+    await handler(event);
+    channel.ack(msg);
+  });
+}
+```
+
+### Topic Routing
+
+```typescript
+// Publisher
+await channel.assertExchange('events', 'topic', { durable: true });
+channel.publish('events', 'order.created', Buffer.from(JSON.stringify(order)));
+channel.publish('events', 'order.cancelled', Buffer.from(JSON.stringify(order)));
+
+// Subscriber — listen to specific patterns
+await channel.bindQueue(queue, 'events', 'order.*');        // All order events
+await channel.bindQueue(queue, 'events', 'order.created');  // Only order.created
+await channel.bindQueue(queue, 'events', '#');               // All events
+```
+
+### Dead-Letter Queue (DLQ)
+
+```typescript
+// Set up DLQ
+await channel.assertQueue('tasks.dlq', { durable: true });
+await channel.assertExchange('dlx', 'direct', { durable: true });
+await channel.bindQueue('tasks.dlq', 'dlx', 'tasks');
+
+// Main queue with DLQ config
+await channel.assertQueue('tasks', {
+  durable: true,
+  deadLetterExchange: 'dlx',
+  deadLetterRoutingKey: 'tasks',
+  messageTtl: 30000, // Optional: messages expire after 30s
+});
+```
+
+## Graceful Shutdown
+
+```typescript
+async function shutdown() {
+  await channel.close();
+  await connection.close();
+}
+
+process.on('SIGINT', shutdown);
+process.on('SIGTERM', shutdown);
+```
+
+## Key Concepts
+
+| Concept | Description |
+|---------|-------------|
+| Exchange | Routes messages to queues (fanout, direct, topic, headers) |
+| Queue | Stores messages until consumed |
+| Binding | Rule connecting an exchange to a queue |
+| Ack | Consumer confirms message processed successfully |
+| Nack | Consumer rejects message (requeue or send to DLQ) |
+| Prefetch | Max unacknowledged messages per consumer |
+| Persistent | Message survives broker restart (written to disk) |
+
+## Anti-Patterns
+
+| Don't | Do |
+|-------|-----|
+| Create a new connection per message | Reuse connections, create channels per operation context |
+| Forget to `ack` or `nack` messages | Always acknowledge — unacked messages block the queue |
+| Use `autoAck: true` for important work | Manual ack after successful processing for reliability |
+| Skip `durable: true` on production queues | Always use durable queues and persistent messages |
+| Ignore connection `error`/`close` events | Implement reconnection logic with backoff |
+| Use `prefetch(0)` (unlimited) | Set prefetch to a sensible value (1-10) to prevent overload |
+| Put large payloads in messages | Store large data externally, pass references in messages |
+| Create unbounded exclusive queues | Monitor queue depth, set TTL or max-length policies |