cortex-agents 2.3.1 → 4.0.0

package/.opencode/skills/data-engineering/SKILL.md

@@ -0,0 +1,221 @@
+---
+name: data-engineering
+description: ETL pipelines, data validation, streaming patterns, message queues, and data partitioning strategies
+license: Apache-2.0
+compatibility: opencode
+---
+
+# Data Engineering Skill
+
+This skill provides patterns for building reliable data pipelines, processing large datasets, and managing data infrastructure.
+
+## When to Use
+
+Use this skill when:
+- Designing ETL/ELT pipelines
+- Implementing data validation and schema enforcement
+- Working with message queues (Kafka, RabbitMQ, SQS)
+- Building streaming data processing systems
+- Designing data partitioning and sharding strategies
+- Handling batch vs real-time data processing
+
+## ETL Pipeline Design
+
+### Batch vs Streaming
+
+| Aspect | Batch Processing | Stream Processing |
+|--------|-----------------|-------------------|
+| **Latency** | Minutes to hours | Milliseconds to seconds |
+| **Data volume** | Large datasets at once | Continuous flow |
+| **Complexity** | Simpler error handling | Complex state management |
+| **Use cases** | Reports, analytics, migrations | Real-time dashboards, alerts, events |
+| **Tools** | Airflow, dbt, Spark | Kafka Streams, Flink, Pulsar |
+
+### ETL vs ELT
+
+| Pattern | When to Use |
+|---------|-------------|
+| **ETL** (Extract → Transform → Load) | Data warehouse with strict schema, transform before loading |
+| **ELT** (Extract → Load → Transform) | Cloud data lakes, transform after loading using SQL/Spark |
+
+### Pipeline Design Principles
+- **Idempotency** — Running the same pipeline twice produces the same result
+- **Incremental processing** — Process only new/changed data, not full reloads
+- **Schema evolution** — Handle schema changes gracefully (add columns, not remove)
+- **Backfill capability** — Ability to reprocess historical data
+- **Monitoring** — Track pipeline health, data quality, processing lag
+
+### Pipeline Architecture
+
+```
+Source → Extract → Validate → Transform → Load → Verify
+                      ↓
+              Dead Letter Queue (failed records)
+```
+
+### Error Handling Strategies
+- **Skip and log** — Log bad records, continue processing (good for analytics)
+- **Dead letter queue** — Route failures to a separate queue for manual review
+- **Fail fast** — Stop pipeline on first error (good for critical data)
+- **Retry with backoff** — Retry transient errors with exponential backoff
+
+## Data Validation
+
+### Schema Enforcement
+- Validate data types, required fields, and constraints at ingestion
+- Use schema registries (Avro, Protobuf, JSON Schema) for contract enforcement
+- Version schemas — never break backward compatibility
+
+### Validation Layers
+
+| Layer | What to Check | Example |
+|-------|---------------|---------|
+| **Structural** | Schema conformance, types, required fields | Missing `email` field, wrong type |
+| **Semantic** | Business rules, value ranges, relationships | Age < 0, end_date before start_date |
+| **Referential** | Foreign key integrity, cross-dataset consistency | Order references non-existent customer |
+| **Statistical** | Distribution anomalies, volume checks | 10x fewer records than yesterday |
+
+### Data Quality Dimensions
+- **Completeness** — Are all required fields populated?
+- **Accuracy** — Do values reflect reality?
+- **Consistency** — Are the same facts represented the same way?
+- **Timeliness** — Is data available when needed?
+- **Uniqueness** — Are there duplicate records?
+
+## Idempotency Patterns
+
+### Why Idempotency Matters
+Pipelines fail and retry. Without idempotency, retries cause:
+- Duplicate records in the target
+- Incorrect aggregations (double-counting)
+- Inconsistent state
+
+### Patterns
+
+| Pattern | How It Works | Trade-off |
+|---------|-------------|-----------|
+| **Upsert (MERGE)** | Insert or update based on key | Requires natural/business key |
+| **Delete + Insert** | Delete partition, then insert | Simple but risky window of missing data |
+| **Deduplication** | Assign unique IDs, deduplicate at read or write | Extra storage for IDs |
+| **Exactly-once semantics** | Transactional writes with offset tracking | Complex, framework-dependent |
+| **Tombstone + Compact** | Write delete markers, compact later | Kafka log compaction pattern |
+
+### Idempotency Keys
+- Use deterministic IDs: `hash(source + key + timestamp)`
+- Store processing watermarks: "last processed offset/timestamp"
+- Use database transactions: read offset + write data atomically
+
+## Message Queue Patterns
+
+### When to Use Which
+
+| Queue | Best For | Key Feature |
+|-------|----------|-------------|
+| **Kafka** | High-throughput event streaming, log-based | Durable, ordered, replayable |
+| **RabbitMQ** | Task queues, RPC, complex routing | Flexible routing, acknowledgments |
+| **SQS** | Simple cloud-native queuing | Managed, auto-scaling, no ops |
+| **Redis Streams** | Lightweight streaming with existing Redis | Low latency, familiar API |
+| **NATS** | High-performance pub/sub | Ultra-low latency, cloud-native |
+
+### Consumer Patterns
+
+| Pattern | Description | Use Case |
+|---------|-------------|----------|
+| **Competing consumers** | Multiple consumers share a queue | Parallel task processing |
+| **Fan-out** | One message delivered to all consumers | Event notifications |
+| **Consumer groups** | Partitioned consumption across group members | Kafka-style parallel processing |
+| **Request-reply** | Send request, await response on reply queue | Async RPC |
+
+### Delivery Guarantees
+
+| Guarantee | Meaning | Trade-off |
+|-----------|---------|-----------|
+| **At-most-once** | Message may be lost, never duplicated | Fastest, lossy |
+| **At-least-once** | Message never lost, may be duplicated | Requires idempotent consumers |
+| **Exactly-once** | Message processed exactly once | Complex, performance overhead |
+
+### Backpressure Handling
+- **Bounded queues** — Reject/block producers when queue is full
+- **Rate limiting** — Limit consumer processing rate
+- **Circuit breaker** — Stop consuming when downstream is unhealthy
+- **Autoscaling** — Add consumers when queue depth exceeds threshold
+
+## Streaming Patterns
+
+### Windowing
+
+| Window Type | Description | Use Case |
+|-------------|-------------|----------|
+| **Tumbling** | Fixed-size, non-overlapping | Hourly aggregation |
+| **Sliding** | Fixed-size, overlapping | Moving average |
+| **Session** | Gap-based, variable size | User session activity |
+| **Global** | All events in one window | Running totals |
+
+### Event Time vs Processing Time
+- **Event time** — When the event actually occurred (embedded in data)
+- **Processing time** — When the system processes the event
+- **Watermarks** — Track progress through event time, handle late arrivals
+- Always prefer event time for correctness; use processing time only for real-time approximation
+
+### Stateful Stream Processing
+- **State stores** — Local key-value stores for aggregations, joins
+- **Changelog topics** — Back up state to Kafka for fault tolerance
+- **State checkpointing** — Periodic snapshots for recovery (Flink pattern)
+
+## Data Partitioning & Sharding
+
+### Partitioning Strategies
+
+| Strategy | How | Best For |
+|----------|-----|----------|
+| **Range partitioning** | Partition by value range (date, ID range) | Time-series data, sequential access |
+| **Hash partitioning** | Hash key modulo partition count | Even distribution, point lookups |
+| **List partitioning** | Partition by discrete values (country, region) | Known categories, geographic data |
+| **Composite** | Combine strategies (hash + range) | Multi-tenant time-series |
+
+### Partition Key Selection
+- Choose keys with **high cardinality** (many distinct values)
+- Avoid **hot partitions** (one key getting disproportionate traffic)
+- Consider **query patterns** — partition by how data is most often read
+- Plan for **partition growth** — avoid partition count that requires redistribution
+
+### Sharding Considerations
+- **Shard key immutability** — Changing a shard key requires data migration
+- **Cross-shard queries** — Avoid joins across shards (denormalize instead)
+- **Rebalancing** — Use consistent hashing to minimize data movement
+- **Shard splitting** — Plan for splitting hot shards without downtime
+
+## Data Pipeline Tools
+
+### Orchestration
+- **Airflow** — DAG-based workflow orchestration (Python)
+- **Dagster** — Software-defined assets, strong typing
+- **Prefect** — Python-native, dynamic workflows
+- **Temporal** — Durable execution for long-running pipelines
+
+### Transformation
+- **dbt** — SQL-based transformations in the warehouse
+- **Spark** — Distributed processing for large datasets
+- **Pandas/Polars** — Single-machine data transformation
+- **Flink** — Stream and batch processing (JVM)
+
+### Storage
+- **Data Lake** — Raw, unstructured (S3, GCS, ADLS)
+- **Data Warehouse** — Structured, optimized for analytics (BigQuery, Snowflake, Redshift)
+- **Data Lakehouse** — Combines both (Delta Lake, Iceberg, Hudi)
+
+## Checklist
+
+When building a data pipeline:
+- [ ] Idempotent operations — safe to retry without side effects
+- [ ] Schema validation at ingestion boundary
+- [ ] Dead letter queue for failed records
+- [ ] Monitoring: processing lag, error rate, throughput
+- [ ] Backfill capability — can reprocess historical data
+- [ ] Incremental processing — not full reloads on every run
+- [ ] Data quality checks after transformation
+- [ ] Partition strategy aligned with query patterns
+- [ ] Exactly-once or at-least-once with idempotent consumers
+- [ ] Schema evolution plan (backward compatible changes)
+- [ ] Alerting on pipeline failures and data quality anomalies
+- [ ] Documentation of data lineage and transformation logic

package/.opencode/skills/monitoring-observability/SKILL.md

@@ -0,0 +1,251 @@
+---
+name: monitoring-observability
+description: Structured logging, metrics instrumentation, distributed tracing, health checks, and alerting patterns
+license: Apache-2.0
+compatibility: opencode
+---
+
+# Monitoring & Observability Skill
+
+This skill provides patterns for making applications observable in production through logging, metrics, tracing, and alerting.
+
+## When to Use
+
+Use this skill when:
+- Adding logging to new features or services
+- Instrumenting code with metrics (counters, histograms, gauges)
+- Implementing distributed tracing across services
+- Designing health check endpoints
+- Setting up alerting and SLO definitions
+- Debugging production issues through observability data
+
+## The Three Pillars of Observability
+
+### 1. Logs — What Happened
+Structured, contextual records of discrete events.
+
+### 2. Metrics — How Much / How Fast
+Numeric measurements aggregated over time.
+
+### 3. Traces — The Journey
+End-to-end request paths across service boundaries.
+
+## Structured Logging
+
+### Principles
+- **Always use structured logging** (JSON) — never unstructured `console.log` in production
+- **Log levels matter**: ERROR (action needed), WARN (degraded), INFO (business events), DEBUG (development)
+- **Include context**: correlation IDs, user IDs, request IDs, operation names
+- **Never log secrets**: passwords, tokens, PII, credit card numbers
+
+### Log Levels Guide
+
+| Level | When to Use | Example |
+|-------|-------------|---------|
+| **ERROR** | Something failed and needs attention | Database connection lost, payment failed |
+| **WARN** | Degraded but still functioning | Cache miss fallback, retry attempt, rate limit approaching |
+| **INFO** | Significant business events | User signed up, order placed, deployment completed |
+| **DEBUG** | Development/troubleshooting detail | SQL query executed, cache hit/miss, function entry/exit |
+
+### Structured Log Format
+
+```json
+{
+  "timestamp": "2025-01-15T10:30:00.000Z",
+  "level": "info",
+  "message": "Order placed successfully",
+  "service": "order-service",
+  "traceId": "abc123",
+  "spanId": "def456",
+  "userId": "user_789",
+  "orderId": "order_012",
+  "amount": 99.99,
+  "currency": "USD",
+  "duration_ms": 145
+}
+```
+
+### Correlation IDs
+- Generate a unique request ID at the entry point (API gateway, load balancer)
+- Propagate it through all downstream calls via headers (`X-Request-ID`, `traceparent`)
+- Include it in every log line for cross-service correlation
+
+### What to Log
+
+**DO log:**
+- Request/response boundaries (method, path, status, duration)
+- Business events (user actions, state transitions, transactions)
+- Error details with stack traces and context
+- Performance-relevant data (query times, cache hit rates)
+- Security events (auth failures, permission denials, rate limits)
+
+**DO NOT log:**
+- Passwords, tokens, API keys, secrets
+- Full credit card numbers, SSNs, or PII without masking
+- High-frequency debug data in production (use sampling)
+- Request/response bodies containing sensitive data
+
+## Metrics Instrumentation
+
+### Metric Types
+
+| Type | Use Case | Example |
+|------|----------|---------|
+| **Counter** | Monotonically increasing value | Total requests, errors, orders placed |
+| **Gauge** | Value that goes up and down | Active connections, queue depth, memory usage |
+| **Histogram** | Distribution of values | Request latency, response size, batch processing time |
+| **Summary** | Pre-calculated quantiles | P50/P95/P99 latency (client-side) |
+
+### Naming Conventions
+- Use snake_case: `http_requests_total`, `request_duration_seconds`
+- Include units in the name: `_seconds`, `_bytes`, `_total`
+- Use `_total` suffix for counters
+- Prefix with service/subsystem: `api_http_requests_total`
+
+### Key Metrics to Track
+
+**RED Method (Request-driven services):**
+- **R**ate — Requests per second
+- **E**rrors — Error rate (4xx, 5xx)
+- **D**uration — Request latency distribution
+
+**USE Method (Resource-oriented):**
+- **U**tilization — % of resource capacity used
+- **S**aturation — Queue depth, backpressure
+- **E**rrors — Error count per resource
+
+### Cardinality Warning
+- Avoid high-cardinality labels (user IDs, request IDs, URLs with path params)
+- Keep label combinations < 1000 per metric
+- Use bounded values: HTTP methods (GET, POST), status codes (2xx, 4xx, 5xx), endpoints (normalized)
+
+## Distributed Tracing
+
+### OpenTelemetry Patterns
+- **Span** — A single operation within a trace (e.g., HTTP request, DB query, function call)
+- **Trace** — A tree of spans representing an end-to-end request
+- **Context Propagation** — Passing trace context across service boundaries via headers
+
+### What to Trace
+- HTTP requests (client and server)
+- Database queries
+- Cache operations
+- Message queue publish/consume
+- External API calls
+- Significant business operations
+
+### Span Attributes
+```
+http.method: GET
+http.url: /api/users/123
+http.status_code: 200
+db.system: postgresql
+db.statement: SELECT * FROM users WHERE id = $1
+messaging.system: kafka
+messaging.destination: orders
+```
+
+### Sampling Strategies
+- **Head-based sampling**: Decide at trace start (e.g., sample 10% of requests)
+- **Tail-based sampling**: Decide after trace completes (keep errors, slow requests, sample normal)
+- **Priority sampling**: Always sample errors, high-value transactions; sample routine requests
+
+## Health Check Endpoints
+
+### Liveness vs Readiness
+
+| Check | Purpose | Failure Action |
+|-------|---------|----------------|
+| **Liveness** (`/healthz`) | Is the process alive? | Restart the container |
+| **Readiness** (`/readyz`) | Can it serve traffic? | Remove from load balancer |
+| **Startup** (`/startupz`) | Has it finished initializing? | Wait before liveness checks |
+
+### Health Check Response Format
+```json
+{
+  "status": "healthy",
+  "checks": {
+    "database": { "status": "healthy", "latency_ms": 5 },
+    "cache": { "status": "healthy", "latency_ms": 1 },
+    "external_api": { "status": "degraded", "latency_ms": 2500, "message": "Slow response" }
+  },
+  "version": "1.2.3",
+  "uptime_seconds": 86400
+}
+```
+
+### Best Practices
+- Health checks should be **fast** (< 1 second)
+- Liveness should check the process only — NOT external dependencies
+- Readiness should check critical dependencies (database, cache)
+- Return appropriate HTTP status: 200 (healthy), 503 (unhealthy)
+- Include dependency health in readiness but not liveness
+
+## Alerting & SLOs
+
+### SLO Definitions
+- **SLI** (Service Level Indicator): The metric you measure (e.g., request latency P99)
+- **SLO** (Service Level Objective): The target (e.g., P99 latency < 500ms for 99.9% of requests)
+- **Error Budget**: Allowable failures (e.g., 0.1% of requests can exceed 500ms)
+
+### Alert Design Principles
+- **Alert on symptoms, not causes** — Alert on "users can't log in", not "CPU is high"
+- **Alert on SLO burn rate** — Alert when error budget is being consumed too fast
+- **Avoid alert fatigue** — Every alert should require human action
+- **Include runbook links** — Every alert should link to resolution steps
+
+### Severity Levels
+
+| Severity | Response Time | Example |
+|----------|--------------|---------|
+| **P1 — Critical** | Immediate (< 5 min) | Service down, data loss, security breach |
+| **P2 — High** | Within 1 hour | Degraded performance, partial outage |
+| **P3 — Medium** | Within 1 business day | Non-critical feature broken, elevated error rate |
+| **P4 — Low** | Next sprint | Performance degradation, tech debt alert |
+
+### Useful Alert Patterns
+- Error rate exceeds N% for M minutes
+- Latency P99 exceeds threshold for M minutes
+- Error budget burn rate > 1x for 1 hour (fast burn)
+- Error budget burn rate > 0.1x for 6 hours (slow burn)
+- Queue depth exceeds threshold (backpressure)
+- Certificate expiry within N days
+- Disk usage exceeds N%
+
+## Technology Selection
+
+### Logging
+- **Node.js**: pino, winston, bunyan
+- **Python**: structlog, python-json-logger
+- **Go**: zerolog, zap, slog (stdlib)
+- **Rust**: tracing, log + env_logger
+
+### Metrics
+- **Prometheus** — Pull-based, widely adopted, great with Kubernetes
+- **StatsD/Datadog** — Push-based, hosted
+- **OpenTelemetry Metrics** — Vendor-neutral, emerging standard
+
+### Tracing
+- **OpenTelemetry** — Vendor-neutral standard (recommended)
+- **Jaeger** — Open-source trace backend
+- **Zipkin** — Lightweight trace backend
+
+### Dashboards
+- **Grafana** — Open-source, works with Prometheus/Loki/Tempo
+- **Datadog** — Hosted all-in-one
+- **New Relic** — Hosted APM
+
+## Checklist
+
+When adding observability to a feature:
+- [ ] Structured logging with correlation IDs at request boundaries
+- [ ] Error logging with stack traces and context
+- [ ] Business event logging (significant state changes)
+- [ ] RED metrics for request-driven endpoints
+- [ ] Histogram for latency-sensitive operations
+- [ ] Trace spans for cross-service calls and database queries
+- [ ] Health check endpoint updated if new dependency added
+- [ ] No secrets or PII in logs
+- [ ] Appropriate log levels (not everything is INFO)
+- [ ] Dashboard updated with new metrics
+- [ ] Alerts defined for SLO violations