@harness-engineering/cli 1.23.2 → 1.24.1

package/dist/agents/skills/codex/events-kafka-patterns/SKILL.md

@@ -0,0 +1,190 @@
+# Events: Kafka Patterns
+
+> Produce and consume Kafka messages with partitioning, consumer groups, and offset management.
+
+## When to Use
+
+- You need high-throughput, durable event streaming (millions of events/day)
+- You need event replay — re-consume historical events to rebuild state or debug
+- You have multiple independent consumer groups that each need all events
+- You need ordered processing within a partition (e.g., all events for one user in order)
+- You're building event sourcing, CDC pipelines, or stream processing
+
+## Instructions
+
+**KafkaJS producer with partitioning:**
+
+```typescript
+import { Kafka, Partitioners, CompressionTypes } from 'kafkajs';
+
+const kafka = new Kafka({
+  clientId: 'order-service',
+  brokers: ['kafka:9092'],
+  retry: { initialRetryTime: 300, retries: 8 },
+});
+
+const producer = kafka.producer({
+  createPartitioner: Partitioners.LegacyPartitioner,
+  idempotent: true, // enable exactly-once for the producer
+  transactionTimeout: 30_000,
+});
+
+await producer.connect();
+
+// Send with a partition key — ensures ordering for the same key
+await producer.send({
+  topic: 'order.events',
+  messages: [
+    {
+      key: `order:${orderId}`, // same key → same partition → ordered
+      value: JSON.stringify({
+        eventType: 'ORDER_CREATED',
+        orderId,
+        userId,
+        amount,
+        timestamp: new Date().toISOString(),
+      }),
+      headers: {
+        'event-type': 'ORDER_CREATED',
+        'source-service': 'order-service',
+        'schema-version': '1',
+      },
+    },
+  ],
+  compression: CompressionTypes.GZIP,
+  acks: -1, // wait for all ISR replicas to ack
+});
+```
+
+**Consumer with consumer group and manual offset commit:**
+
+```typescript
+const consumer = kafka.consumer({
+  groupId: 'fulfillment-service', // all instances share this group — each partition assigned to one
+  sessionTimeout: 30_000,
+  heartbeatInterval: 3_000,
+});
+
+await consumer.connect();
+await consumer.subscribe({ topic: 'order.events', fromBeginning: false });
+
+await consumer.run({
+  eachBatchAutoResolve: false, // manual offset commit for fine-grained control
+  eachBatch: async ({ batch, resolveOffset, heartbeat, commitOffsetsIfNecessary }) => {
+    for (const message of batch.messages) {
+      const event = JSON.parse(message.value!.toString());
+
+      try {
+        await processOrderEvent(event);
+        resolveOffset(message.offset); // mark this message as processed
+      } catch (err) {
+        console.error(`Failed to process ${message.offset}:`, err);
+        // Do NOT resolve offset — message will be reprocessed
+        break; // stop processing this batch
+      }
+
+      await heartbeat(); // prevent session timeout during slow processing
+    }
+
+    await commitOffsetsIfNecessary(); // commit resolved offsets
+  },
+});
+```
+
+**Transactional producer (atomic produce + consume):**
+
+```typescript
+const transactionalProducer = kafka.producer({
+  transactionalId: 'order-processor-1', // unique per producer instance
+  idempotent: true,
+  maxInFlightRequests: 1,
+});
+
+await transactionalProducer.connect();
+
+// Atomic: consume from input topic, produce to output topic
+const transaction = await transactionalProducer.transaction();
+try {
+  await transaction.send({
+    topic: 'shipping.commands',
+    messages: [{ key: `order:${orderId}`, value: JSON.stringify({ orderId, address }) }],
+  });
+
+  // Commit offsets as part of the transaction
+  await transaction.sendOffsets({
+    consumerGroupId: 'order-processor',
+    topics: [{ topic: 'order.events', partitions: [{ partition: 0, offset: '42' }] }],
+  });
+
+  await transaction.commit();
+} catch (err) {
+  await transaction.abort();
+  throw err;
+}
+```
+
+**Topic and partition strategy:**
+
+```typescript
+// Partition key selection strategy
+function getPartitionKey(event: OrderEvent): string {
+  // User-scoped events: partition by userId for ordering
+  if ('userId' in event) return `user:${event.userId}`;
+  // Order events: partition by orderId
+  if ('orderId' in event) return `order:${event.orderId}`;
+  // Global events: null key → round-robin
+  return '';
+}
+
+// Topic naming convention
+const TOPICS = {
+  ORDER_EVENTS: 'order.events', // all order lifecycle events
+  FULFILLMENT_COMMANDS: 'fulfillment.commands', // commands to fulfillment
+  NOTIFICATION_EVENTS: 'notification.events', // fan-out notifications
+  DLQ: 'order.events.dlq', // failed processing
+} as const;
+```
+
+## Details
+
+**Consumer group mechanics:** All consumers in the same `groupId` share partitions — each partition assigned to one consumer. Add consumers to scale (up to `partitionCount` consumers can process in parallel). Different `groupId` values create independent consumers that each receive all messages.
+
+**Offset management:**
+
+- `fromBeginning: true` — start from the earliest retained offset (useful for new consumer groups)
+- `fromBeginning: false` — start from the latest offset (only new messages)
+- Manual offset commit gives exactly-once semantics when combined with idempotent processing
+
+**Retention and compaction:**
+
+- Default retention: time-based (7 days). Good for audit logs, replay.
+- Log compaction: keeps only the latest message per key. Good for state materialization (like a distributed KV store).
+
+**Anti-patterns:**
+
+- Single partition for all messages — no parallelism, no ordering control
+- Committing offsets before processing — risk of data loss on crash
+- Consumer that blocks `eachMessage` without heartbeats — causes session timeout and partition reassignment
+- Sharing `transactionalId` across multiple producer instances — causes transaction fencing errors
+
+**Schema evolution:** Use Avro or Protobuf with a schema registry. Never break consumers by removing fields without versioning. See `events-event-schema` skill.
+
+## Source
+
+kafka.apache.org/documentation/
+
+## Process
+
+1. Read the instructions and examples in this document.
+2. Apply the patterns to your implementation, adapting to your specific context.
+3. Verify your implementation against the details and edge cases listed above.
+
+## Harness Integration
+
+- **Type:** knowledge — this skill is a reference document, not a procedural workflow.
+- **No tools or state** — consumed as context by other skills and agents.
+
+## Success Criteria
+
+- The patterns described in this document are applied correctly in the implementation.
+- Edge cases and anti-patterns listed in this document are avoided.

package/dist/agents/skills/codex/events-kafka-patterns/skill.yaml

@@ -0,0 +1,45 @@
+name: events-kafka-patterns
+version: "1.0.0"
+description: Produce and consume Kafka messages with partitioning, consumer groups, and offset management
+stability: static
+cognitive_mode: advisory-guide
+type: knowledge
+tier: 3
+triggers:
+  - manual
+platforms:
+  - claude-code
+  - gemini-cli
+  - cursor
+  - codex
+tools: []
+paths:
+  - "**/events/**"
+  - "**/queues/**"
+  - "**/handlers/**"
+  - "**/socket/**"
+  - "**/pub-sub/**"
+related_skills:
+  - events-message-queue
+  - events-event-schema
+  - events-idempotency
+stack_signals:
+  - nodejs
+  - typescript
+  - kafka
+  - redis
+  - rabbitmq
+keywords:
+  - kafka
+  - producer
+  - consumer
+  - partitions
+  - consumer-groups
+  - offset-management
+metadata:
+  author: community
+  upstream: kafka.apache.org/documentation/
+state:
+  persistent: false
+  files: []
+depends_on: []

package/dist/agents/skills/codex/events-message-queue/SKILL.md

@@ -0,0 +1,190 @@
+# Events: Message Queue
+
+> Use message queues for reliable async delivery with competing consumers and dead letter queues.
+
+## When to Use
+
+- You need reliable, at-least-once delivery — messages must not be lost even if the consumer crashes
+- You have work that should be processed by exactly one consumer (competing consumers pattern)
+- You need decoupling between producer and consumer throughput
+- You want dead letter queues (DLQ) for failed message handling
+- Background job processing, email sending, report generation, order fulfillment
+
+## Instructions
+
+**BullMQ (Redis-backed queue — recommended for Node.js):**
+
+```typescript
+import { Queue, Worker, Job } from 'bullmq';
+import { Redis } from 'ioredis';
+
+const connection = new Redis({ host: 'localhost', port: 6379 });
+
+// Define job data types
+interface SendEmailJobData {
+  to: string;
+  subject: string;
+  body: string;
+  templateId?: string;
+}
+
+// Producer — enqueue jobs
+const emailQueue = new Queue<SendEmailJobData>('email', { connection });
+
+async function scheduleWelcomeEmail(userId: string, email: string): Promise<void> {
+  await emailQueue.add(
+    'welcome',
+    { to: email, subject: 'Welcome!', body: 'Thanks for joining.', templateId: 'welcome-v2' },
+    {
+      delay: 5_000, // wait 5s before processing
+      attempts: 3, // retry up to 3 times
+      backoff: { type: 'exponential', delay: 2_000 }, // 2s, 4s, 8s
+      removeOnComplete: { count: 100 }, // keep last 100 completed jobs
+      removeOnFail: { count: 50 }, // keep last 50 failed jobs
+    }
+  );
+}
+
+// Consumer — process jobs
+const emailWorker = new Worker<SendEmailJobData>(
+  'email',
+  async (job: Job<SendEmailJobData>) => {
+    console.log(`Processing job ${job.id}: send email to ${job.data.to}`);
+    await sendEmail(job.data.to, job.data.subject, job.data.body);
+    // Returning from the processor marks the job complete
+  },
+  {
+    connection,
+    concurrency: 5, // process 5 jobs in parallel
+  }
+);
+
+emailWorker.on('completed', (job) => {
+  console.log(`Job ${job.id} completed`);
+});
+
+emailWorker.on('failed', (job, err) => {
+  console.error(`Job ${job?.id} failed after all retries:`, err.message);
+  // At this point, job moves to the failed set (acts as DLQ)
+});
+```
+
+**RabbitMQ with amqplib:**
+
+```typescript
+import amqp from 'amqplib';
+
+const QUEUE = 'order.processing';
+const DLQ = 'order.processing.dlq';
+
+async function setupQueue(): Promise<void> {
+  const conn = await amqp.connect('amqp://localhost');
+  const channel = await conn.createChannel();
+
+  // Dead letter queue
+  await channel.assertQueue(DLQ, { durable: true });
+
+  // Main queue with DLQ routing
+  await channel.assertQueue(QUEUE, {
+    durable: true,
+    arguments: {
+      'x-dead-letter-exchange': '',
+      'x-dead-letter-routing-key': DLQ,
+      'x-message-ttl': 3_600_000, // messages expire after 1h
+    },
+  });
+
+  channel.prefetch(1); // process one message at a time per consumer
+  return channel;
+}
+
+// Producer
+async function publishOrder(orderId: string, amount: number): Promise<void> {
+  const channel = await setupQueue();
+  const message = JSON.stringify({ orderId, amount, timestamp: new Date() });
+  channel.sendToQueue(QUEUE, Buffer.from(message), { persistent: true });
+}
+
+// Consumer
+async function startConsumer(): Promise<void> {
+  const channel = await setupQueue();
+  await channel.consume(QUEUE, async (msg) => {
+    if (!msg) return;
+    try {
+      const data = JSON.parse(msg.content.toString());
+      await processOrder(data.orderId, data.amount);
+      channel.ack(msg); // acknowledge on success
+    } catch (err) {
+      console.error('Processing failed:', err);
+      // nack with requeue=false — sends to DLQ after retries exhausted
+      channel.nack(msg, false, false);
+    }
+  });
+}
+```
+
+**Competing consumers pattern:**
+
+```typescript
+// Run multiple workers — each message processed by exactly ONE worker
+// Scale by adding more worker instances
+
+// Worker 1 (process 1)
+const worker1 = new Worker('orders', processOrder, { connection, concurrency: 3 });
+
+// Worker 2 (process 2 or separate machine)
+const worker2 = new Worker('orders', processOrder, { connection, concurrency: 3 });
+
+// BullMQ guarantees only one worker processes each job
+```
+
+## Details
+
+**Delivery guarantees:**
+
+- **At-most-once:** Fire and forget. Publisher sends, never tracks. Simple but messages can be lost.
+- **At-least-once:** Publisher gets confirmation; consumer acks after processing. May process twice — handlers must be idempotent.
+- **Exactly-once:** At-least-once + deduplication. Hard to achieve; usually simulated with idempotency keys.
+
+**Dead letter queue strategy:**
+
+1. Job fails after all retries
+2. Moved to DLQ automatically
+3. Monitoring alerts on DLQ depth
+4. Operators inspect, fix the bug, and replay from DLQ
+
+**Anti-patterns:**
+
+- Consumers that ack before processing (risk losing work on crash)
+- Long-running consumers without heartbeats (broker requeues "stalled" jobs)
+- Unbounded queue depth (apply backpressure via rate limiting producers)
+- Processing heavy jobs in HTTP request handlers (enqueue them instead)
+
+**Monitoring queues:**
+
+```typescript
+// BullMQ — check queue health
+const counts = await emailQueue.getJobCounts('waiting', 'active', 'completed', 'failed', 'delayed');
+console.log(counts);
+// Expose as /metrics endpoint for Prometheus
+```
+
+## Source
+
+microservices.io/patterns/communication-style/messaging.html
+
+## Process
+
+1. Read the instructions and examples in this document.
+2. Apply the patterns to your implementation, adapting to your specific context.
+3. Verify your implementation against the details and edge cases listed above.
+
+## Harness Integration
+
+- **Type:** knowledge — this skill is a reference document, not a procedural workflow.
+- **No tools or state** — consumed as context by other skills and agents.
+
+## Success Criteria
+
+- The patterns described in this document are applied correctly in the implementation.
+- Edge cases and anti-patterns listed in this document are avoided.

package/dist/agents/skills/codex/events-message-queue/skill.yaml

@@ -0,0 +1,44 @@
+name: events-message-queue
+version: "1.0.0"
+description: Use message queues for reliable async delivery with competing consumers and dead letter queues
+stability: static
+cognitive_mode: advisory-guide
+type: knowledge
+tier: 3
+triggers:
+  - manual
+platforms:
+  - claude-code
+  - gemini-cli
+  - cursor
+  - codex
+tools: []
+paths:
+  - "**/events/**"
+  - "**/queues/**"
+  - "**/handlers/**"
+  - "**/socket/**"
+  - "**/pub-sub/**"
+related_skills:
+  - events-pubsub-pattern
+  - events-idempotency
+  - events-kafka-patterns
+stack_signals:
+  - nodejs
+  - typescript
+  - kafka
+  - redis
+  - rabbitmq
+keywords:
+  - message-queue
+  - at-least-once
+  - competing-consumers
+  - dead-letter-queue
+  - async-messaging
+metadata:
+  author: community
+  upstream: microservices.io/patterns/communication-style/messaging.html
+state:
+  persistent: false
+  files: []
+depends_on: []

package/dist/agents/skills/codex/events-outbox-pattern/SKILL.md

@@ -0,0 +1,212 @@
+# Events: Outbox Pattern
+
+> Reliably publish domain events using the transactional outbox and CDC polling approach.
+
+## When to Use
+
+- You need to guarantee that domain events are published whenever a database transaction commits
+- You can't afford to lose events (payment processed, order created, user registered)
+- You want to avoid the dual-write problem: "write to DB and publish to queue atomically"
+- You're building event-sourced systems or microservices with eventual consistency
+
+## Instructions
+
+**The problem the outbox solves:**
+
+```
+BAD:
+1. Write to DB ✓
+2. Publish to Kafka ✗  ← crash here → event lost, DB and queue out of sync
+
+GOOD (outbox):
+1. Write to DB + write to outbox table (one transaction) ✓
+2. Separate poller reads outbox, publishes to Kafka ✓
+3. Mark outbox records as published ✓
+```
+
+**Outbox table schema (SQL):**
+
+```sql
+CREATE TABLE outbox (
+  id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  event_type  TEXT NOT NULL,
+  payload     JSONB NOT NULL,
+  aggregate_id TEXT NOT NULL,
+  created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+  published_at TIMESTAMPTZ,
+  publish_attempts INT NOT NULL DEFAULT 0,
+  error       TEXT
+);
+
+CREATE INDEX outbox_unpublished ON outbox (created_at)
+  WHERE published_at IS NULL;
+```
+
+**Write to DB + outbox in one transaction (Prisma example):**
+
+```typescript
+import { PrismaClient } from '@prisma/client';
+
+const prisma = new PrismaClient();
+
+async function createOrder(data: CreateOrderInput): Promise<Order> {
+  return prisma.$transaction(async (tx) => {
+    // Primary write
+    const order = await tx.order.create({
+      data: {
+        userId: data.userId,
+        items: data.items,
+        status: 'PENDING',
+        total: calculateTotal(data.items),
+      },
+    });
+
+    // Outbox write — same transaction
+    await tx.outbox.create({
+      data: {
+        eventType: 'order.created',
+        aggregateId: order.id,
+        payload: {
+          orderId: order.id,
+          userId: order.userId,
+          total: order.total,
+          items: order.items,
+          occurredAt: new Date().toISOString(),
+        },
+      },
+    });
+
+    return order;
+  });
+  // If transaction commits: both writes succeed atomically
+  // If transaction rolls back: neither write happens
+}
+```
+
+**Polling publisher (separate process):**
+
+```typescript
+class OutboxPublisher {
+  private running = false;
+
+  constructor(
+    private readonly db: PrismaClient,
+    private readonly producer: KafkaProducer,
+    private readonly pollIntervalMs = 1_000,
+    private readonly batchSize = 100
+  ) {}
+
+  start(): void {
+    this.running = true;
+    this.poll();
+  }
+
+  stop(): void {
+    this.running = false;
+  }
+
+  private async poll(): Promise<void> {
+    while (this.running) {
+      try {
+        await this.publishBatch();
+      } catch (err) {
+        console.error('Outbox poll error:', err);
+      }
+      await new Promise((r) => setTimeout(r, this.pollIntervalMs));
+    }
+  }
+
+  private async publishBatch(): Promise<void> {
+    // Fetch unpublished events — oldest first, batch
+    const events = await this.db.outbox.findMany({
+      where: { publishedAt: null, publishAttempts: { lt: 5 } },
+      orderBy: { createdAt: 'asc' },
+      take: this.batchSize,
+    });
+
+    if (events.length === 0) return;
+
+    for (const event of events) {
+      try {
+        // Publish to Kafka with event ID as key for deduplication
+        await this.producer.send({
+          topic: event.eventType,
+          messages: [
+            {
+              key: event.aggregateId,
+              value: JSON.stringify(event.payload),
+              headers: { 'outbox-id': event.id },
+            },
+          ],
+        });
+
+        // Mark as published
+        await this.db.outbox.update({
+          where: { id: event.id },
+          data: { publishedAt: new Date() },
+        });
+      } catch (err) {
+        await this.db.outbox.update({
+          where: { id: event.id },
+          data: {
+            publishAttempts: { increment: 1 },
+            error: (err as Error).message,
+          },
+        });
+      }
+    }
+  }
+}
+```
+
+**CDC alternative (Debezium):** Instead of polling, use Change Data Capture to stream the outbox table changes directly to Kafka. No polling process needed — lower latency, zero overhead on DB.
+
+```yaml
+# Debezium connector config (Kafka Connect)
+connector.class: io.debezium.connector.postgresql.PostgresConnector
+database.hostname: postgres
+database.dbname: myapp
+table.include.list: public.outbox
+transforms: outbox
+transforms.outbox.type: io.debezium.transforms.outbox.EventRouter
+```
+
+## Details
+
+**Polling interval vs. latency trade-off:** Polling every 100ms gives ~100ms event latency. Polling every 5s gives 5s latency. CDC gives near-zero latency but adds infrastructure complexity.
+
+**Exactly-once delivery:** The outbox gives at-least-once. To achieve exactly-once, consumers must deduplicate using the outbox record ID (stored in message headers). See `events-idempotency` skill.
+
+**Anti-patterns:**
+
+- Publishing inside the application after the transaction (dual-write) — defeats the purpose
+- Outbox table that grows unboundedly — prune published events after a retention period
+- Missing index on `WHERE published_at IS NULL` — polling becomes a full table scan under load
+
+**Retention cleanup:**
+
+```sql
+DELETE FROM outbox
+WHERE published_at IS NOT NULL
+  AND published_at < NOW() - INTERVAL '7 days';
+```
+
+## Source
+
+microservices.io/patterns/data/transactional-outbox.html
+
+## Process
+
+1. Read the instructions and examples in this document.
+2. Apply the patterns to your implementation, adapting to your specific context.
+3. Verify your implementation against the details and edge cases listed above.
+
+## Harness Integration
+
+- **Type:** knowledge — this skill is a reference document, not a procedural workflow.
+- **No tools or state** — consumed as context by other skills and agents.
+
+## Success Criteria
+
+- The patterns described in this document are applied correctly in the implementation.
+- Edge cases and anti-patterns listed in this document are avoided.