4runr-os 2.10.85 → 2.10.86

package/apps/os-server/TROUBLESHOOTING.md

@@ -0,0 +1,541 @@
+# Troubleshooting Guide
+
+Common issues and solutions for the 4Runr Gateway.
+
+## Table of Contents
+
+1. [Startup Issues](#startup-issues)
+2. [Database Issues](#database-issues)
+3. [Authentication Issues](#authentication-issues)
+4. [Queue Issues](#queue-issues)
+5. [Rate Limiting Issues](#rate-limiting-issues)
+6. [Performance Issues](#performance-issues)
+7. [Error Messages](#error-messages)
+
+## Startup Issues
+
+### Gateway Won't Start
+
+**Symptoms:**
+- Container exits immediately
+- "Cannot find module" errors
+- Port already in use
+
+**Solutions:**
+
+1. **Check logs:**
+```bash
+docker-compose logs gateway
+# or
+pm2 logs gateway
+```
+
+2. **Port conflict:**
+```bash
+# Check what's using port 3000
+lsof -i :3000
+# or
+netstat -tulpn | grep 3000
+
+# Change port in docker-compose.yml or .env
+PORT=3001
+```
+
+3. **Missing dependencies:**
+```bash
+# Rebuild
+docker-compose build --no-cache gateway
+
+# Or reinstall
+pnpm install
+```
+
+4. **Prisma client not generated:**
+```bash
+cd apps/os-server
+pnpm db:generate
+```
+
+### Health Check Failing
+
+**Symptoms:**
+- `/health` returns 500 or times out
+- Container marked as unhealthy
+
+**Solutions:**
+
+1. **Check if server is actually running:**
+```bash
+curl http://localhost:3000/health
+```
+
+2. **Check dependencies:**
+```bash
+curl http://localhost:3000/ready
+# Check which dependencies are failing
+```
+
+3. **Review startup logs:**
+```bash
+docker-compose logs gateway | grep -i error
+```
+
+## Database Issues
+
+### Cannot Connect to Database
+
+**Symptoms:**
+- "Connection refused" errors
+- "Authentication failed" errors
+- Timeout errors
+
+**Solutions:**
+
+1. **Verify database is running:**
+```bash
+docker-compose ps postgres
+# or
+psql -h localhost -U 4runr -d 4runr_gateway
+```
+
+2. **Check connection string:**
+```bash
+echo $DATABASE_URL
+# Should be: postgresql://user:password@host:port/database
+```
+
+3. **Verify network connectivity:**
+```bash
+# From gateway container
+docker-compose exec gateway ping postgres
+```
+
+4. **Check firewall rules:**
+```bash
+# Ensure port 5432 is accessible
+telnet db-host 5432
+```
+
+### Migration Errors
+
+**Symptoms:**
+- "Migration failed" errors
+- Schema mismatch errors
+
+**Solutions:**
+
+1. **Check migration status:**
+```bash
+cd apps/os-server
+pnpm prisma migrate status
+```
+
+2. **Reset database (development only):**
+```bash
+pnpm prisma migrate reset
+```
+
+3. **Apply pending migrations:**
+```bash
+pnpm db:migrate
+```
+
+4. **Manual migration:**
+```bash
+pnpm prisma migrate dev --name migration-name
+```
+
+### Prisma Client Not Found
+
+**Symptoms:**
+- "Cannot find module '@prisma/client'"
+- "Prisma client not generated"
+
+**Solutions:**
+
+1. **Generate Prisma client:**
+```bash
+cd apps/os-server
+pnpm db:generate
+```
+
+2. **Verify generation:**
+```bash
+ls -la node_modules/.prisma/client
+```
+
+3. **Rebuild:**
+```bash
+pnpm clean
+pnpm build
+```
+
+## Authentication Issues
+
+### API Key Not Working
+
+**Symptoms:**
+- 401 Unauthorized errors
+- "Invalid API key" messages
+
+**Solutions:**
+
+1. **Verify API key format:**
+```bash
+# Check header name (should be x-api-key)
+curl -H "x-api-key: your-key" http://localhost:3000/api/runs
+```
+
+2. **Check if key exists in database:**
+```bash
+# PostgreSQL mode
+psql -h localhost -U 4runr -d 4runr_gateway -c "SELECT * FROM api_keys;"
+
+# Memory mode - check API_KEY env var
+echo $API_KEY
+```
+
+3. **Verify AUTH_ENABLED:**
+```bash
+# Should be true
+echo $AUTH_ENABLED
+```
+
+4. **Check key is not revoked:**
+```bash
+psql -h localhost -U 4runr -d 4runr_gateway -c "SELECT * FROM api_keys WHERE revoked = false;"
+```
+
+### JWT Authentication Failing
+
+**Symptoms:**
+- "Invalid token" errors
+- "Token expired" errors
+
+**Solutions:**
+
+1. **Verify JWT_SECRET matches:**
+```bash
+echo $JWT_SECRET
+# Must match the secret used to sign the token
+```
+
+2. **Check token expiration:**
+```bash
+# Decode JWT (use jwt.io or similar)
+# Verify 'exp' claim is in the future
+```
+
+3. **Verify algorithm:**
+```bash
+# Check JWT_ALG matches token algorithm
+echo $JWT_ALG
+```
+
+4. **Check token format:**
+```bash
+# Should be: Authorization: Bearer <token>
+curl -H "Authorization: Bearer your-token" http://localhost:3000/api/runs
+```
+
+## Queue Issues
+
+### Jobs Not Processing
+
+**Symptoms:**
+- Runs stuck in "queued" status
+- No job activity
+
+**Solutions:**
+
+1. **Check Redis connection:**
+```bash
+# Verify Redis is running
+docker-compose ps redis
+redis-cli ping
+```
+
+2. **Check REDIS_URL:**
+```bash
+echo $REDIS_URL
+# Should be: redis://host:port
+```
+
+3. **Verify queue initialization:**
+```bash
+# Check logs for queue initialization
+docker-compose logs gateway | grep -i queue
+```
+
+4. **Check queue status:**
+```bash
+# Access Redis and check queue
+redis-cli
+> KEYS bull:run-execution:*
+> LLEN bull:run-execution:waiting
+```
+
+### Jobs Failing
+
+**Symptoms:**
+- Jobs marked as "failed"
+- Error logs in processor
+
+**Solutions:**
+
+1. **Check job error logs:**
+```bash
+docker-compose logs gateway | grep -i "job.*fail"
+```
+
+2. **Verify processor context:**
+```bash
+# Check if Sentinel/SSE context is set
+docker-compose logs gateway | grep -i "processor context"
+```
+
+3. **Check run store:**
+```bash
+# Verify runs can be retrieved
+curl http://localhost:3000/api/runs -H "x-api-key: your-key"
+```
+
+## Rate Limiting Issues
+
+### Rate Limits Too Strict
+
+**Symptoms:**
+- 429 errors too frequently
+- Legitimate requests blocked
+
+**Solutions:**
+
+1. **Increase limits:**
+```bash
+RATE_LIMIT_MAX=100
+RATE_LIMIT_WINDOW_MS=60000
+```
+
+2. **Check per-route limits:**
+- Read operations: Full limit
+- Write operations: 75% of limit
+- Expensive operations: 50% of limit
+
+3. **Verify Redis is working:**
+```bash
+# Rate limiting uses Redis
+redis-cli ping
+```
+
+4. **Check identity tracking:**
+```bash
+# Rate limits are per-identity (API key, JWT sub, or IP)
+# Verify same identity isn't being used for multiple clients
+```
+
+### Rate Limits Not Working
+
+**Symptoms:**
+- No 429 errors even with high load
+- Rate limiting disabled
+
+**Solutions:**
+
+1. **Verify RATE_LIMIT_ENABLED:**
+```bash
+echo $RATE_LIMIT_ENABLED
+# Should be true
+```
+
+2. **Check Redis connection:**
+```bash
+# Rate limiting requires Redis
+redis-cli ping
+```
+
+3. **Verify middleware is registered:**
+```bash
+# Check logs for rate limit middleware
+docker-compose logs gateway | grep -i "rate limit"
+```
+
+## Performance Issues
+
+### Slow Response Times
+
+**Symptoms:**
+- High latency
+- Timeout errors
+
+**Solutions:**
+
+1. **Check database performance:**
+```bash
+# Monitor database connections
+psql -c "SELECT count(*) FROM pg_stat_activity;"
+
+# Check slow queries
+psql -c "SELECT * FROM pg_stat_statements ORDER BY total_time DESC LIMIT 10;"
+```
+
+2. **Check queue depth:**
+```bash
+# High queue depth indicates bottleneck
+redis-cli LLEN bull:run-execution:waiting
+```
+
+3. **Monitor resource usage:**
+```bash
+# CPU and memory
+docker stats 4runr-gateway
+# or
+top -p $(pgrep -f "node.*gateway")
+```
+
+4. **Check for memory leaks:**
+```bash
+# Monitor memory over time
+watch -n 1 'docker stats --no-stream 4runr-gateway | grep gateway'
+```
+
+### High Memory Usage
+
+**Symptoms:**
+- Container OOM errors
+- Memory usage growing over time
+
+**Solutions:**
+
+1. **Check for memory leaks:**
+```bash
+# Use Node.js memory profiling
+node --inspect dist/index.js
+```
+
+2. **Review queue job retention:**
+```bash
+# Check queue config for job retention
+# Reduce removeOnComplete/removeOnFail if too many jobs retained
+```
+
+3. **Limit concurrent jobs:**
+```bash
+# Configure queue concurrency
+# In queue config, set concurrency limit
+```
+
+4. **Increase container memory:**
+```bash
+# In docker-compose.yml
+deploy:
+  resources:
+    limits:
+      memory: 2G
+```
+
+## Error Messages
+
+### "Cannot find module '@prisma/client'"
+
+**Solution:**
+```bash
+cd apps/os-server
+pnpm db:generate
+```
+
+### "Prisma client not generated yet"
+
+**Solution:**
+```bash
+cd apps/os-server
+pnpm db:generate
+pnpm build
+```
+
+### "Queue system not available (Redis required)"
+
+**Solution:**
+- Verify Redis is running: `docker-compose ps redis`
+- Check REDIS_URL: `echo $REDIS_URL`
+- Gateway will fall back to synchronous execution
+
+### "Security validation failed"
+
+**Solution:**
+- Check JWT_SECRET length (min 32 chars)
+- Verify no insecure default values
+- See security validation logs for details
+
+### "Database connection required"
+
+**Solution:**
+- Verify DATABASE_URL is set
+- Check database is accessible
+- In development, can use `GATEWAY_PERSISTENCE=memory`
+
+### "Rate limit exceeded"
+
+**Solution:**
+- Wait for rate limit window to reset
+- Check retry_after_ms in error response
+- Consider increasing RATE_LIMIT_MAX if legitimate
+
+## Getting Help
+
+### Collecting Diagnostic Information
+
+```bash
+# Health status
+curl http://localhost:3000/health
+curl http://localhost:3000/ready
+
+# Metrics
+curl http://localhost:3000/metrics
+
+# Logs
+docker-compose logs gateway --tail=100
+
+# Environment
+docker-compose exec gateway env | grep -E "(NODE_ENV|DATABASE|REDIS|AUTH|RATE)"
+
+# Database status
+psql -h localhost -U 4runr -d 4runr_gateway -c "SELECT version();"
+
+# Redis status
+redis-cli INFO
+```
+
+### Log Levels
+
+Increase log verbosity for debugging:
+
+```bash
+LOG_LEVEL=debug
+```
+
+**Warning**: Debug logs can be verbose. Use only for troubleshooting.
+
+## Common Patterns
+
+### Service Won't Start After Update
+
+1. Check for breaking changes in changelog
+2. Verify environment variables still valid
+3. Check database migrations needed
+4. Review dependency updates
+
+### Intermittent Errors
+
+1. Check for resource limits (CPU, memory)
+2. Verify network stability
+3. Check database connection pool
+4. Review queue job processing
+
+### Production Issues
+
+1. Check error logs first
+2. Verify health endpoints
+3. Review metrics for anomalies
+4. Check recent deployments
+5. Verify dependencies (DB, Redis) are healthy
+

package/apps/os-server/dist/apps/os-server/src/adapters/gateway-cancel-adapter.d.ts

@@ -0,0 +1,3 @@
+import type { CancelAdapter } from '@4runr/sentinel';
+export declare function createGatewayCancelAdapter(): CancelAdapter;
+//# sourceMappingURL=gateway-cancel-adapter.d.ts.map

package/apps/os-server/dist/apps/os-server/src/adapters/gateway-cancel-adapter.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"gateway-cancel-adapter.d.ts","sourceRoot":"","sources":["../../../../../src/adapters/gateway-cancel-adapter.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,EAAE,aAAa,EAAgB,MAAM,iBAAiB,CAAC;AAGnE,wBAAgB,0BAA0B,IAAI,aAAa,CAwB1D"}

package/apps/os-server/dist/apps/os-server/src/adapters/gateway-cancel-adapter.js

@@ -0,0 +1,28 @@
+import { applyRunKill } from '../runs/run-kill.js';
+export function createGatewayCancelAdapter() {
+    return {
+        async cancelRun(runId, reason) {
+            try {
+                const result = await applyRunKill(runId, reason);
+                return {
+                    success: result.applied,
+                    reason: result.idempotent
+                        ? 'Run already terminal'
+                        : result.applied
+                            ? 'Run killed in store'
+                            : 'Run kill not applied',
+                    timestamp: Date.now(),
+                };
+            }
+            catch (error) {
+                const message = error instanceof Error ? error.message : String(error);
+                return {
+                    success: false,
+                    reason: message,
+                    timestamp: Date.now(),
+                };
+            }
+        },
+    };
+}
+//# sourceMappingURL=gateway-cancel-adapter.js.map

package/apps/os-server/dist/apps/os-server/src/adapters/gateway-cancel-adapter.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"gateway-cancel-adapter.js","sourceRoot":"","sources":["../../../../../src/adapters/gateway-cancel-adapter.ts"],"names":[],"mappings":"AAKA,OAAO,EAAE,YAAY,EAAE,MAAM,qBAAqB,CAAC;AAEnD,MAAM,UAAU,0BAA0B;IACxC,OAAO;QACL,KAAK,CAAC,SAAS,CAAC,KAAa,EAAE,MAAc;YAC3C,IAAI,CAAC;gBACH,MAAM,MAAM,GAAG,MAAM,YAAY,CAAC,KAAK,EAAE,MAAM,CAAC,CAAC;gBACjD,OAAO;oBACL,OAAO,EAAE,MAAM,CAAC,OAAO;oBACvB,MAAM,EAAE,MAAM,CAAC,UAAU;wBACvB,CAAC,CAAC,sBAAsB;wBACxB,CAAC,CAAC,MAAM,CAAC,OAAO;4BACd,CAAC,CAAC,qBAAqB;4BACvB,CAAC,CAAC,sBAAsB;oBAC5B,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE;iBACtB,CAAC;YACJ,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,MAAM,OAAO,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;gBACvE,OAAO;oBACL,OAAO,EAAE,KAAK;oBACd,MAAM,EAAE,OAAO;oBACf,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE;iBACtB,CAAC;YACJ,CAAC;QACH,CAAC;KACF,CAAC;AACJ,CAAC"}

package/apps/os-server/dist/apps/os-server/src/adapters/redis-sentinel-publisher.d.ts

@@ -0,0 +1,23 @@
+export interface SentinelEvent {
+    type: 'shield.decision' | 'judge.verdict' | 'audit.log' | 'policy.violation' | 'kill';
+    runId: string;
+    timestamp: number;
+    data: Record<string, unknown>;
+}
+export interface PolicyViolationEventData {
+    policy: string;
+    reason: string;
+    threshold: number;
+    actual: number;
+    action: 'deny';
+}
+export type SentinelKillTrigger = 'policy' | 'manual' | 'operator';
+export declare function publishSentinelEvent(event: SentinelEvent): Promise<void>;
+export declare function buildPolicyViolationEvent(runId: string, violation: PolicyViolationEventData, timestamp?: number): SentinelEvent;
+export declare function buildKillEvent(runId: string, reason: string, trigger: SentinelKillTrigger, timestamp?: number): SentinelEvent;
+export declare function buildAuditLogEvent(runId: string, action: 'kill' | 'policy_violation', payload: Record<string, unknown>, timestamp?: number): SentinelEvent;
+export declare function publishSentinelRunKill(runId: string, reason: string): Promise<void>;
+export declare function publishSentinelPolicyViolation(runId: string, violation: PolicyViolationEventData): Promise<void>;
+export declare function publishSentinelRunKillFireAndForget(runId: string, reason: string): void;
+export declare function publishSentinelPolicyViolationFireAndForget(runId: string, violation: PolicyViolationEventData): void;
+//# sourceMappingURL=redis-sentinel-publisher.d.ts.map

package/apps/os-server/dist/apps/os-server/src/adapters/redis-sentinel-publisher.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"redis-sentinel-publisher.d.ts","sourceRoot":"","sources":["../../../../../src/adapters/redis-sentinel-publisher.ts"],"names":[],"mappings":"AAkBA,MAAM,WAAW,aAAa;IAC5B,IAAI,EAAE,iBAAiB,GAAG,eAAe,GAAG,WAAW,GAAG,kBAAkB,GAAG,MAAM,CAAC;IACtF,KAAK,EAAE,MAAM,CAAC;IACd,SAAS,EAAE,MAAM,CAAC;IAClB,IAAI,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;CAC/B;AAED,MAAM,WAAW,wBAAwB;IACvC,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;IACf,SAAS,EAAE,MAAM,CAAC;IAClB,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,MAAM,mBAAmB,GAAG,QAAQ,GAAG,QAAQ,GAAG,UAAU,CAAC;AAKnE,wBAAsB,oBAAoB,CAAC,KAAK,EAAE,aAAa,GAAG,OAAO,CAAC,IAAI,CAAC,CAuB9E;AAED,wBAAgB,yBAAyB,CACvC,KAAK,EAAE,MAAM,EACb,SAAS,EAAE,wBAAwB,EACnC,SAAS,SAAa,GACrB,aAAa,CAOf;AAED,wBAAgB,cAAc,CAC5B,KAAK,EAAE,MAAM,EACb,MAAM,EAAE,MAAM,EACd,OAAO,EAAE,mBAAmB,EAC5B,SAAS,SAAa,GACrB,aAAa,CAaf;AAED,wBAAgB,kBAAkB,CAChC,KAAK,EAAE,MAAM,EACb,MAAM,EAAE,MAAM,GAAG,kBAAkB,EACnC,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,EAChC,SAAS,SAAa,GACrB,aAAa,CAOf;AAgBD,wBAAsB,sBAAsB,CAC1C,KAAK,EAAE,MAAM,EACb,MAAM,EAAE,MAAM,GACb,OAAO,CAAC,IAAI,CAAC,CAYf;AAKD,wBAAsB,8BAA8B,CAClD,KAAK,EAAE,MAAM,EACb,SAAS,EAAE,wBAAwB,GAClC,OAAO,CAAC,IAAI,CAAC,CAaf;AAGD,wBAAgB,mCAAmC,CACjD,KAAK,EAAE,MAAM,EACb,MAAM,EAAE,MAAM,GACb,IAAI,CAEN;AAED,wBAAgB,2CAA2C,CACzD,KAAK,EAAE,MAAM,EACb,SAAS,EAAE,wBAAwB,GAClC,IAAI,CAEN"}

package/apps/os-server/dist/apps/os-server/src/adapters/redis-sentinel-publisher.js

@@ -0,0 +1,95 @@
+import { getRedisClient } from '../db/redis.js';
+import { createLogger } from '@4runr/shared';
+import { buildSentinelFailureOutput, parsePolicyFromKillReason, } from '../security/sentinel-run-failure.js';
+import { sentinelKillsTotal, sentinelPolicyViolationsTotal, } from '../metrics/index.js';
+const logger = createLogger('Gateway:Sentinel:RedisPublisher');
+export async function publishSentinelEvent(event) {
+    try {
+        const redisClient = await getRedisClient();
+        if (!redisClient) {
+            return;
+        }
+        const channel = `sentinel:${event.type}`;
+        const message = JSON.stringify(event);
+        await redisClient.publish(channel, message);
+        logger.debug('Published Sentinel event to Redis', {
+            channel,
+            runId: event.runId,
+            type: event.type,
+        });
+    }
+    catch (error) {
+        logger.error('Failed to publish Sentinel event to Redis', {
+            runId: event.runId,
+            type: event.type,
+            error: error instanceof Error ? error.message : String(error),
+        });
+    }
+}
+export function buildPolicyViolationEvent(runId, violation, timestamp = Date.now()) {
+    return {
+        type: 'policy.violation',
+        runId,
+        timestamp,
+        data: { ...violation },
+    };
+}
+export function buildKillEvent(runId, reason, trigger, timestamp = Date.now()) {
+    return {
+        type: 'kill',
+        runId,
+        timestamp,
+        data: {
+            reason,
+            trigger,
+            source: 'sentinel',
+            output: buildSentinelFailureOutput(reason),
+            policy: parsePolicyFromKillReason(reason),
+        },
+    };
+}
+export function buildAuditLogEvent(runId, action, payload, timestamp = Date.now()) {
+    return {
+        type: 'audit.log',
+        runId,
+        timestamp,
+        data: { action, ...payload },
+    };
+}
+function resolveKillTrigger(reason) {
+    if (reason === 'manual_cancellation') {
+        return 'manual';
+    }
+    if (reason.startsWith('policy_violation:')) {
+        return 'policy';
+    }
+    return 'operator';
+}
+export async function publishSentinelRunKill(runId, reason) {
+    const trigger = resolveKillTrigger(reason);
+    sentinelKillsTotal.inc({ trigger });
+    await publishSentinelEvent(buildKillEvent(runId, reason, trigger));
+    await publishSentinelEvent(buildAuditLogEvent(runId, 'kill', {
+        reason,
+        trigger,
+        output: buildSentinelFailureOutput(reason),
+    }));
+}
+export async function publishSentinelPolicyViolation(runId, violation) {
+    sentinelPolicyViolationsTotal.inc({ policy: violation.policy });
+    await publishSentinelEvent(buildPolicyViolationEvent(runId, violation));
+    await publishSentinelEvent(buildAuditLogEvent(runId, 'policy_violation', {
+        policy: violation.policy,
+        reason: violation.reason,
+        threshold: violation.threshold,
+        actual: violation.actual,
+        action: 'deny',
+    }));
+}
+export function publishSentinelRunKillFireAndForget(runId, reason) {
+    void publishSentinelRunKill(runId, reason);
+}
+export function publishSentinelPolicyViolationFireAndForget(runId, violation) {
+    void publishSentinelPolicyViolation(runId, violation);
+}
+//# sourceMappingURL=redis-sentinel-publisher.js.map

package/apps/os-server/dist/apps/os-server/src/adapters/redis-sentinel-publisher.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"redis-sentinel-publisher.js","sourceRoot":"","sources":["../../../../../src/adapters/redis-sentinel-publisher.ts"],"names":[],"mappings":"AAKA,OAAO,EAAE,cAAc,EAAE,MAAM,gBAAgB,CAAC;AAChD,OAAO,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AAC7C,OAAO,EACL,0BAA0B,EAC1B,yBAAyB,GAC1B,MAAM,qCAAqC,CAAC;AAC7C,OAAO,EACL,kBAAkB,EAClB,6BAA6B,GAC9B,MAAM,qBAAqB,CAAC;AAE7B,MAAM,MAAM,GAAG,YAAY,CAAC,iCAAiC,CAAC,CAAC;AAsB/D,MAAM,CAAC,KAAK,UAAU,oBAAoB,CAAC,KAAoB;IAC7D,IAAI,CAAC;QACH,MAAM,WAAW,GAAG,MAAM,cAAc,EAAE,CAAC;QAC3C,IAAI,CAAC,WAAW,EAAE,CAAC;YACjB,OAAO;QACT,CAAC;QAED,MAAM,OAAO,GAAG,YAAY,KAAK,CAAC,IAAI,EAAE,CAAC;QACzC,MAAM,OAAO,GAAG,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC;QAEtC,MAAM,WAAW,CAAC,OAAO,CAAC,OAAO,EAAE,OAAO,CAAC,CAAC;QAC5C,MAAM,CAAC,KAAK,CAAC,mCAAmC,EAAE;YAChD,OAAO;YACP,KAAK,EAAE,KAAK,CAAC,KAAK;YAClB,IAAI,EAAE,KAAK,CAAC,IAAI;SACjB,CAAC,CAAC;IACL,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,CAAC,KAAK,CAAC,2CAA2C,EAAE;YACxD,KAAK,EAAE,KAAK,CAAC,KAAK;YAClB,IAAI,EAAE,KAAK,CAAC,IAAI;YAChB,KAAK,EAAE,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC;SAC9D,CAAC,CAAC;IACL,CAAC;AACH,CAAC;AAED,MAAM,UAAU,yBAAyB,CACvC,KAAa,EACb,SAAmC,EACnC,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE;IAEtB,OAAO;QACL,IAAI,EAAE,kBAAkB;QACxB,KAAK;QACL,SAAS;QACT,IAAI,EAAE,EAAE,GAAG,SAAS,EAAE;KACvB,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,cAAc,CAC5B,KAAa,EACb,MAAc,EACd,OAA4B,EAC5B,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE;IAEtB,OAAO;QACL,IAAI,EAAE,MAAM;QACZ,KAAK;QACL,SAAS;QACT,IAAI,EAAE;YACJ,MAAM;YACN,OAAO;YACP,MAAM,EAAE,UAAU;YAClB,MAAM,EAAE,0BAA0B,CAAC,MAAM,CAAC;YAC1C,MAAM,EAAE,yBAAyB,CAAC,MAAM,CAAC;SAC1C;KACF,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,kBAAkB,CAChC,KAAa,EACb,MAAmC,EACnC,OAAgC,EAChC,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE;IAEtB,OAAO;QACL,IAAI,EAAE,WAAW;QACjB,KAAK;QACL,SAAS;QACT,IAAI,EAAE,EAAE,MAAM,EAAE,GAAG,OAAO,EAAE;KAC7B,CAAC;AACJ,CAAC;AAED,SAAS,kBAAkB,CAAC,MAAc;IACxC,IAAI,MAAM,KAAK,qBAAqB,EAAE,CAAC;QACrC,OAAO,QAAQ,CAAC;IAClB,CAAC;IACD,IAAI,MAAM,CAAC,UAAU,CAAC,mBAAmB,CAAC,EAAE,CAAC;QAC3C,OAAO,QAAQ,CAAC;IAClB,CAAC;IACD,OAAO,UAAU,CAAC;AACpB,CAAC;AAMD,MAAM,CAAC,KAAK,UAAU,sBAAsB,CAC1C,KAAa,EACb,MAAc;IAEd,MAAM,OAAO,GAAG,kBAAkB,CAAC,MAAM,CAAC,CAAC;IAC3C,kBAAkB,CAAC,GAAG,CAAC,EAAE,OAAO,EAAE,CAAC,CAAC;IAEpC,MAAM,oBAAoB,CAAC,cAAc,CAAC,KAAK,EAAE,MAAM,EAAE,OAAO,CAAC,CAAC,CAAC;IACnE,MAAM,oBAAoB,CACxB,kBAAkB,CAAC,KAAK,EAAE,MAAM,EAAE;QAChC,MAAM;QACN,OAAO;QACP,MAAM,EAAE,0BAA0B,CAAC,MAAM,CAAC;KAC3C,CAAC,CACH,CAAC;AACJ,CAAC;AAKD,MAAM,CAAC,KAAK,UAAU,8BAA8B,CAClD,KAAa,EACb,SAAmC;IAEnC,6BAA6B,CAAC,GAAG,CAAC,EAAE,MAAM,EAAE,SAAS,CAAC,MAAM,EAAE,CAAC,CAAC;IAEhE,MAAM,oBAAoB,CAAC,yBAAyB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC,CAAC;IACxE,MAAM,oBAAoB,CACxB,kBAAkB,CAAC,KAAK,EAAE,kBAAkB,EAAE;QAC5C,MAAM,EAAE,SAAS,CAAC,MAAM;QACxB,MAAM,EAAE,SAAS,CAAC,MAAM;QACxB,SAAS,EAAE,SAAS,CAAC,SAAS;QAC9B,MAAM,EAAE,SAAS,CAAC,MAAM;QACxB,MAAM,EAAE,MAAM;KACf,CAAC,CACH,CAAC;AACJ,CAAC;AAGD,MAAM,UAAU,mCAAmC,CACjD,KAAa,EACb,MAAc;IAEd,KAAK,sBAAsB,CAAC,KAAK,EAAE,MAAM,CAAC,CAAC;AAC7C,CAAC;AAED,MAAM,UAAU,2CAA2C,CACzD,KAAa,EACb,SAAmC;IAEnC,KAAK,8BAA8B,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;AACxD,CAAC"}