@namch/agent-assistant 1.0.0

package/skills/microservices-architect/references/patterns.md

@@ -0,0 +1,603 @@
+# Resilience and Reliability Patterns
+
+Essential patterns for building fault-tolerant distributed systems.
+
+## Resilience Patterns
+
+### Circuit Breaker
+
+**Purpose:** Prevent cascading failures by failing fast when a dependency is unhealthy.
+
+**How It Works:**
+```
+States:
+1. CLOSED (normal operation)
+   - Requests pass through
+   - Track failure rate
+   - If failures exceed threshold → OPEN
+
+2. OPEN (failing fast)
+   - Immediately reject requests
+   - Return fallback response
+   - After timeout period → HALF_OPEN
+
+3. HALF_OPEN (testing recovery)
+   - Allow limited test requests
+   - If successful → CLOSED
+   - If failed → OPEN
+
+Configuration:
+- Failure threshold: 50% failures in 10 requests
+- Timeout: 30 seconds in OPEN state
+- Success threshold: 2 consecutive successes in HALF_OPEN
+```
+
+**Implementation Example:**
+```python
+# Using resilience4j-like pattern
+@CircuitBreaker(
+    name="payment-service",
+    fallbackMethod="paymentFallback",
+    failureThreshold=50,
+    waitDurationInOpenState=30000,  # 30s
+    permittedNumberOfCallsInHalfOpenState=3
+)
+async def process_payment(order_id: str, amount: float):
+    async with httpx.AsyncClient() as client:
+        response = await client.post(
+            f"{PAYMENT_SERVICE_URL}/payments",
+            json={"orderId": order_id, "amount": amount},
+            timeout=5.0
+        )
+        return response.json()
+
+async def paymentFallback(order_id: str, amount: float, exception):
+    # Log the failure
+    logger.error(f"Payment service unavailable: {exception}")
+    # Return graceful degradation
+    return {
+        "status": "pending",
+        "message": "Payment processing delayed, will retry"
+    }
+```
+
+**When to Use:**
+```
+Apply circuit breakers to:
+✓ External service calls
+✓ Database queries
+✓ Third-party APIs
+✓ Microservice-to-microservice calls
+
+Configuration Guidelines:
+- Fast services (p99 < 100ms): 5s timeout, 10s circuit open
+- Medium services (p99 < 1s): 10s timeout, 30s circuit open
+- Slow services (p99 > 1s): 30s timeout, 60s circuit open
+```
+
+### Retry Pattern
+
+**Purpose:** Handle transient failures by retrying operations.
+
+**Strategies:**
+
+**1. Exponential Backoff:**
+```
+Retry delays: 100ms, 200ms, 400ms, 800ms, 1600ms
+
+Benefits:
+- Reduces load during incidents
+- Gives service time to recover
+- Prevents thundering herd
+
+Implementation:
+attempts = 0
+max_attempts = 5
+base_delay = 0.1  # 100ms
+
+while attempts < max_attempts:
+    try:
+        return await make_request()
+    except TransientError as e:
+        attempts += 1
+        if attempts == max_attempts:
+            raise
+        delay = base_delay * (2 ** attempts) + random.uniform(0, 0.1)
+        await asyncio.sleep(delay)
+```
+
+**2. Retry with Jitter:**
+```
+Why: Prevents synchronized retries (thundering herd)
+
+Full Jitter:
+delay = random.uniform(0, base_delay * (2 ** attempt))
+
+Decorrelated Jitter:
+delay = min(cap, random.uniform(base, previous_delay * 3))
+
+Recommended: Decorrelated jitter for production systems
+```
+
+**3. Idempotency Keys:**
+```
+Problem: Retries can cause duplicate operations
+
+Solution: Idempotency keys
+POST /api/v1/payments
+Headers:
+  Idempotency-Key: uuid-12345
+
+Server Logic:
+1. Check if operation with this key already processed
+2. If yes, return cached response
+3. If no, process and cache result
+4. Cache for 24 hours
+
+Ensures safe retries even for non-idempotent operations
+```
+
+**Retry Best Practices:**
+```
+DO:
+✓ Only retry transient errors (timeout, 503, 429)
+✓ Use exponential backoff with jitter
+✓ Set maximum retry attempts (3-5)
+✓ Implement overall timeout
+✓ Use idempotency keys for writes
+✓ Log each retry attempt
+
+DON'T:
+✗ Retry client errors (400, 401, 404)
+✗ Retry without backoff (causes load spikes)
+✗ Infinite retries
+✗ Retry non-idempotent operations without safeguards
+```
+
+### Bulkhead Pattern
+
+**Purpose:** Isolate resources to prevent total system failure.
+
+**Thread Pool Isolation:**
+```
+Concept: Separate thread pools for different operations
+
+Example:
+- Payment Service Thread Pool: 20 threads
+- Inventory Service Thread Pool: 20 threads
+- Notification Service Thread Pool: 10 threads
+
+If payment service becomes slow:
+- Only payment thread pool exhausted
+- Inventory and notification still work
+- System partially degraded, not completely down
+```
+
+**Connection Pool Isolation:**
+```
+Database Connection Pools:
+- Read-only queries: 50 connections
+- Write queries: 20 connections
+- Reporting queries: 10 connections
+
+Heavy reporting query won't starve transactional operations
+```
+
+**Rate Limiting per Tenant:**
+```
+Multi-tenant SaaS application:
+
+tenant-a: 1000 requests/minute
+tenant-b: 1000 requests/minute
+tenant-c: 1000 requests/minute
+
+If tenant-a floods the system:
+- Only tenant-a throttled
+- tenant-b and tenant-c unaffected
+```
+
+**Implementation:**
+```python
+# Using semaphores for concurrency limits
+class BulkheadExecutor:
+    def __init__(self):
+        self.payment_semaphore = asyncio.Semaphore(20)
+        self.inventory_semaphore = asyncio.Semaphore(20)
+        self.notification_semaphore = asyncio.Semaphore(10)
+
+    async def call_payment_service(self, data):
+        async with self.payment_semaphore:
+            return await payment_service.call(data)
+
+    async def call_inventory_service(self, data):
+        async with self.inventory_semaphore:
+            return await inventory_service.call(data)
+```
+
+### Timeout Pattern
+
+**Purpose:** Prevent indefinite waiting for responses.
+
+**Timeout Types:**
+
+**1. Connection Timeout:**
+```
+Time allowed to establish connection
+
+Recommended: 2-5 seconds
+If takes longer, network likely has issues
+
+httpx.AsyncClient(timeout=httpx.Timeout(connect=3.0))
+```
+
+**2. Read Timeout:**
+```
+Time allowed to receive response after connection
+
+Varies by service:
+- Fast APIs: 5 seconds
+- Database queries: 10 seconds
+- Complex processing: 30 seconds
+
+httpx.AsyncClient(timeout=httpx.Timeout(read=10.0))
+```
+
+**3. Total Timeout:**
+```
+Overall time budget for entire operation
+
+Example: User checkout flow
+- Total budget: 30 seconds
+- Payment service: 10 seconds
+- Inventory check: 5 seconds
+- Order creation: 5 seconds
+- Buffer: 10 seconds
+
+async with asyncio.timeout(30):
+    result = await complete_checkout()
+```
+
+**Timeout Best Practices:**
+```
+Timeouts Hierarchy:
+Parent timeout > sum of child timeouts
+
+Request → API Gateway (30s timeout)
+  → Service A (10s timeout)
+    → Service B (5s timeout)
+      → Database (2s timeout)
+
+Set timeouts everywhere:
+✓ HTTP clients
+✓ Database connections
+✓ Message consumers
+✓ gRPC calls
+✓ Cache operations
+```
+
+## Distributed Transaction Patterns
+
+### Saga Pattern
+
+**Purpose:** Manage distributed transactions across services.
+
+**Choreography-Based Saga:**
+```
+Example: Order Creation Saga
+
+Events:
+1. OrderService: order.created
+2. PaymentService: payment.completed OR payment.failed
+3. InventoryService: inventory.reserved OR inventory.reservation.failed
+4. ShippingService: shipment.created
+
+Compensating Transactions:
+If inventory.reservation.failed:
+  → PaymentService listens → refund.initiated
+  → OrderService listens → order.cancelled
+
+Pros:
+- Decentralized
+- No single point of failure
+- Services autonomous
+
+Cons:
+- Difficult to track saga state
+- Complex debugging
+- No saga-wide timeout
+```
+
+**Orchestration-Based Saga:**
+```
+Example: Order Saga Orchestrator
+
+Saga Steps:
+1. Create Order (OrderService)
+2. Charge Payment (PaymentService)
+3. Reserve Inventory (InventoryService)
+4. Create Shipment (ShippingService)
+
+Orchestrator Logic:
+step1_result = await order_service.create_order()
+if not step1_result.success:
+    return failure("Order creation failed")
+
+step2_result = await payment_service.charge(amount)
+if not step2_result.success:
+    await order_service.cancel_order(step1_result.order_id)
+    return failure("Payment failed")
+
+step3_result = await inventory_service.reserve(items)
+if not step3_result.success:
+    await payment_service.refund(step2_result.payment_id)
+    await order_service.cancel_order(step1_result.order_id)
+    return failure("Inventory unavailable")
+
+# Continue saga...
+
+Pros:
+- Clear workflow
+- Centralized monitoring
+- Easy to understand
+
+Cons:
+- Orchestrator complexity
+- Potential bottleneck
+- Coupling to orchestrator
+```
+
+**Saga State Management:**
+```
+Persist saga state to handle failures:
+
+CREATE TABLE saga_instances (
+    saga_id UUID PRIMARY KEY,
+    saga_type VARCHAR(50),
+    current_step VARCHAR(50),
+    status VARCHAR(20),
+    payload JSONB,
+    created_at TIMESTAMP,
+    updated_at TIMESTAMP
+);
+
+On orchestrator restart:
+- Load incomplete sagas
+- Resume from last completed step
+- Execute remaining steps or compensations
+```
+
+### Event Sourcing
+
+**Purpose:** Store all state changes as events, derive current state by replaying.
+
+**Implementation:**
+```
+Traditional Approach:
+UPDATE orders SET status = 'shipped' WHERE id = 123;
+(Lost: when shipped, by whom, from where)
+
+Event Sourcing Approach:
+Events:
+1. OrderPlaced { orderId, customerId, items, timestamp }
+2. PaymentReceived { orderId, amount, paymentId, timestamp }
+3. OrderShipped { orderId, trackingNumber, carrier, timestamp }
+
+Current state = replay all events
+```
+
+**Event Store:**
+```
+CREATE TABLE events (
+    event_id UUID PRIMARY KEY,
+    aggregate_id UUID,
+    aggregate_type VARCHAR(50),
+    event_type VARCHAR(100),
+    event_data JSONB,
+    version INTEGER,
+    timestamp TIMESTAMP,
+    correlation_id UUID
+);
+
+CREATE INDEX idx_aggregate ON events(aggregate_id, version);
+
+Guarantees:
+- Events immutable
+- Events ordered by version
+- Optimistic locking prevents conflicts
+```
+
+**Benefits:**
+```
+✓ Full audit trail
+✓ Time travel (replay to any point)
+✓ Event replay for debugging
+✓ Multiple read models from same events
+✓ Temporal queries ("show orders as of yesterday")
+
+Challenges:
+✗ Eventual consistency
+✗ Event schema evolution
+✗ Snapshot strategy needed
+✗ Increased storage
+```
+
+### CQRS (Command Query Responsibility Segregation)
+
+**Purpose:** Separate read and write models for different optimization strategies.
+
+**Architecture:**
+```
+Write Side (Command):
+- Receives commands (CreateOrder, UpdateInventory)
+- Validates business rules
+- Stores events in event store
+- Optimized for consistency and writes
+
+Read Side (Query):
+- Listens to events
+- Updates denormalized read models
+- Optimized for queries
+- Eventual consistency
+
+Example:
+Command: CreateOrder
+  → Order aggregate validates
+  → Publishes OrderCreated event
+  → Event stored in event store
+
+Query Side:
+  → Listens to OrderCreated
+  → Updates order_summary table (denormalized)
+  → Updates customer_order_history (different view)
+  → Updates order_analytics (aggregated metrics)
+```
+
+**Read Models:**
+```
+Multiple specialized views from same events:
+
+1. Order Detail View (for customer):
+   { orderId, items, status, total, estimatedDelivery }
+
+2. Order List View (for admin):
+   { orderId, customerName, orderDate, status, total }
+
+3. Analytics View:
+   { date, totalOrders, totalRevenue, averageOrderValue }
+
+Each optimized for specific query patterns
+```
+
+## Fault Tolerance Patterns
+
+### Health Checks
+
+**Types:**
+
+**1. Liveness Probe:**
+```
+Purpose: Is the service alive?
+
+Endpoint: GET /health/live
+
+Returns 200 if:
+- Application process running
+- Not deadlocked
+
+Kubernetes Action:
+- If fails: Restart container
+```
+
+**2. Readiness Probe:**
+```
+Purpose: Is the service ready to receive traffic?
+
+Endpoint: GET /health/ready
+
+Returns 200 if:
+- Database connection pool healthy
+- Cache accessible
+- Downstream dependencies responsive
+
+Kubernetes Action:
+- If fails: Remove from load balancer
+- Don't send traffic until ready
+```
+
+**3. Startup Probe:**
+```
+Purpose: Has the service finished initialization?
+
+Endpoint: GET /health/startup
+
+For slow-starting applications:
+- Prevents premature liveness checks
+- Allows longer startup time
+```
+
+**Implementation:**
+```python
+@app.get("/health/live")
+async def liveness():
+    return {"status": "alive"}
+
+@app.get("/health/ready")
+async def readiness():
+    checks = {
+        "database": await check_database(),
+        "cache": await check_cache(),
+        "payment_service": await check_payment_service()
+    }
+
+    all_healthy = all(checks.values())
+    status_code = 200 if all_healthy else 503
+
+    return JSONResponse(
+        status_code=status_code,
+        content={"status": "ready" if all_healthy else "not ready", "checks": checks}
+    )
+```
+
+### Graceful Degradation
+
+**Purpose:** Provide reduced functionality when dependencies fail.
+
+**Strategies:**
+
+**1. Cached Responses:**
+```
+async def get_product_recommendations(user_id):
+    try:
+        async with circuit_breaker:
+            return await ml_service.get_recommendations(user_id)
+    except ServiceUnavailable:
+        # Fallback to cached popular products
+        return await cache.get_popular_products()
+```
+
+**2. Default Values:**
+```
+async def get_user_preferences(user_id):
+    try:
+        return await preferences_service.get(user_id)
+    except ServiceUnavailable:
+        # Return sensible defaults
+        return {
+            "language": "en",
+            "currency": "USD",
+            "theme": "light"
+        }
+```
+
+**3. Feature Toggles:**
+```
+if feature_flags.is_enabled("personalized_recommendations"):
+    recommendations = await ml_service.get_recommendations()
+else:
+    # Fallback to simple algorithm
+    recommendations = await get_popular_products()
+```
+
+## Summary
+
+Resilience patterns are mandatory in distributed systems. Layer multiple patterns for defense in depth:
+
+**Essential Stack:**
+1. Timeouts (prevent hanging)
+2. Retries with backoff (handle transient errors)
+3. Circuit breakers (prevent cascading failures)
+4. Bulkheads (isolate failures)
+5. Health checks (enable auto-healing)
+6. Graceful degradation (maintain partial functionality)
+
+**Choose Saga Pattern When:**
+- Distributed transaction needed
+- Strong consistency not required
+- Compensating transactions possible
+
+**Choose Event Sourcing When:**
+- Full audit trail required
+- Temporal queries needed
+- Multiple read models beneficial
+
+Always test failure scenarios. Use chaos engineering to validate resilience.

package/skills/ml-pipeline/SKILL.md

@@ -0,0 +1,111 @@
+---
+name: ml-pipeline
+description: Use when building ML pipelines, orchestrating training workflows, automating model lifecycle, implementing feature stores, or managing experiment tracking systems.
+triggers:
+  - ML pipeline
+  - MLflow
+  - Kubeflow
+  - feature engineering
+  - model training
+  - experiment tracking
+  - feature store
+  - hyperparameter tuning
+  - pipeline orchestration
+  - model registry
+  - training workflow
+  - MLOps
+  - model deployment
+  - data pipeline
+  - model versioning
+role: expert
+scope: implementation
+output-format: code
+---
+
+# ML Pipeline Expert
+
+Senior ML pipeline engineer specializing in production-grade machine learning infrastructure, orchestration systems, and automated training workflows.
+
+## Role Definition
+
+You are a senior ML pipeline expert specializing in end-to-end machine learning workflows. You design and implement scalable feature engineering pipelines, orchestrate distributed training jobs, manage experiment tracking, and automate the complete model lifecycle from data ingestion to production deployment. You build robust, reproducible, and observable ML systems.
+
+## When to Use This Skill
+
+- Building feature engineering pipelines and feature stores
+- Orchestrating training workflows with Kubeflow, Airflow, or custom systems
+- Implementing experiment tracking with MLflow, Weights & Biases, or Neptune
+- Creating automated hyperparameter tuning pipelines
+- Setting up model registries and versioning systems
+- Designing data validation and preprocessing workflows
+- Implementing model evaluation and validation strategies
+- Building reproducible training environments
+- Automating model retraining and deployment pipelines
+
+## Core Workflow
+
+1. **Design pipeline architecture** - Map data flow, identify stages, define interfaces between components
+2. **Implement feature engineering** - Build transformation pipelines, feature stores, validation checks
+3. **Orchestrate training** - Configure distributed training, hyperparameter tuning, resource allocation
+4. **Track experiments** - Log metrics, parameters, artifacts; enable comparison and reproducibility
+5. **Validate and deploy** - Implement model validation, A/B testing, automated deployment workflows
+
+## Reference Guide
+
+Load detailed guidance based on context:
+
+| Topic | Reference | Load When |
+|-------|-----------|-----------|
+| Feature Engineering | `references/feature-engineering.md` | Feature pipelines, transformations, feature stores, Feast, data validation |
+| Training Pipelines | `references/training-pipelines.md` | Training orchestration, distributed training, hyperparameter tuning, resource management |
+| Experiment Tracking | `references/experiment-tracking.md` | MLflow, Weights & Biases, experiment logging, model registry |
+| Pipeline Orchestration | `references/pipeline-orchestration.md` | Kubeflow Pipelines, Airflow, Prefect, DAG design, workflow automation |
+| Model Validation | `references/model-validation.md` | Evaluation strategies, validation workflows, A/B testing, shadow deployment |
+
+## Constraints
+
+### MUST DO
+- Version all data, code, and models explicitly
+- Implement reproducible training environments (pinned dependencies, seeds)
+- Log all hyperparameters and metrics to experiment tracking
+- Validate data quality before training (schema checks, distribution validation)
+- Use containerized environments for training jobs
+- Implement proper error handling and retry logic
+- Store artifacts in versioned object storage
+- Enable pipeline monitoring and alerting
+- Document pipeline dependencies and data lineage
+- Implement automated testing for pipeline components
+
+### MUST NOT DO
+- Run training without experiment tracking
+- Deploy models without validation metrics
+- Hardcode hyperparameters in training scripts
+- Skip data validation and quality checks
+- Use non-reproducible random states
+- Store credentials in pipeline code
+- Train on production data without proper access controls
+- Deploy models without versioning
+- Ignore pipeline failures silently
+- Mix training and inference code without clear separation
+
+## Output Templates
+
+When implementing ML pipelines, provide:
+1. Complete pipeline definition (Kubeflow/Airflow DAG or equivalent)
+2. Feature engineering code with data validation
+3. Training script with experiment logging
+4. Model evaluation and validation code
+5. Deployment configuration
+6. Brief explanation of architecture decisions and reproducibility measures
+
+## Knowledge Reference
+
+MLflow, Kubeflow Pipelines, Apache Airflow, Prefect, Feast, Weights & Biases, Neptune, DVC, Great Expectations, Ray, Horovod, Kubernetes, Docker, S3/GCS/Azure Blob, model registry patterns, feature store architecture, distributed training, hyperparameter optimization
+
+## Related Skills
+
+- **DevOps Engineer** - CI/CD integration for ML workflows
+- **Kubernetes Specialist** - ML workload orchestration on K8s
+- **Cloud Architect** - Cloud infrastructure for ML pipelines
+- **Python Pro** - Python best practices for ML code
+- **Data Engineer** - Data pipeline integration