@techwavedev/agi-agent-kit 1.5.3 → 1.6.0

package/templates/skills/extended/other/comfyui-gateway/references/troubleshooting.md

@@ -0,0 +1,1082 @@
+# ComfyUI Gateway -- Troubleshooting Guide
+
+Comprehensive troubleshooting reference for diagnosing and resolving issues with the
+ComfyUI Gateway. Every section follows the **Symptom -> Cause -> Solution** format
+with concrete commands you can run immediately.
+
+---
+
+## Table of Contents
+
+1. [ComfyUI Not Reachable](#1-comfyui-not-reachable)
+2. [OOM (Out of Memory) Errors](#2-oom-out-of-memory-errors)
+3. [Slow Generation](#3-slow-generation)
+4. [Webhook Failures](#4-webhook-failures)
+5. [Redis Connection Issues](#5-redis-connection-issues)
+6. [Storage Errors](#6-storage-errors)
+7. [Database Issues](#7-database-issues)
+8. [Job Stuck in "running"](#8-job-stuck-in-running)
+9. [Rate Limiting Issues](#9-rate-limiting-issues)
+10. [Authentication Problems](#10-authentication-problems)
+
+---
+
+## 1. ComfyUI Not Reachable
+
+The gateway returns `COMFYUI_UNREACHABLE` and the `/health` endpoint shows
+`comfyui.reachable: false`.
+
+### 1a. Wrong COMFYUI_URL
+
+**Symptom**: Gateway starts fine but every job fails with `COMFYUI_UNREACHABLE`.
+The health endpoint returns `{ ok: false, comfyui: { reachable: false } }`.
+
+**Cause**: The `COMFYUI_URL` in `.env` does not point to a running ComfyUI instance.
+
+**Solution**:
+
+```bash
+# 1. Verify what you have configured
+grep COMFYUI_URL .env
+
+# 2. Test connectivity from the gateway host
+curl -s http://127.0.0.1:8188/
+# Expected: HTML page or JSON from ComfyUI
+
+# 3. If ComfyUI is on a different port or host, update .env
+# Example: COMFYUI_URL=http://192.168.1.50:8188
+
+# 4. Restart the gateway after changing .env
+npm run dev
+```
+
+### 1b. Firewall Blocking the Port
+
+**Symptom**: `curl` to the ComfyUI URL times out or returns `Connection refused`,
+but ComfyUI is confirmed running on that machine.
+
+**Cause**: A host firewall (Windows Defender, iptables, ufw) is blocking the port.
+
+**Solution**:
+
+```bash
+# Linux (ufw)
+sudo ufw allow 8188/tcp
+sudo ufw reload
+
+# Linux (iptables)
+sudo iptables -A INPUT -p tcp --dport 8188 -j ACCEPT
+
+# Windows (PowerShell, run as Admin)
+New-NetFirewallRule -DisplayName "ComfyUI" -Direction Inbound -Port 8188 -Protocol TCP -Action Allow
+
+# Verify the port is listening
+# Linux
+ss -tlnp | grep 8188
+# Windows
+netstat -an | findstr 8188
+```
+
+### 1c. Docker Networking
+
+**Symptom**: Gateway running inside Docker cannot reach ComfyUI on `127.0.0.1:8188`.
+
+**Cause**: `127.0.0.1` inside a Docker container refers to the container itself,
+not the host machine.
+
+**Solution**:
+
+```bash
+# Option A: Use Docker's special host DNS (Linux + Docker Desktop)
+COMFYUI_URL=http://host.docker.internal:8188
+
+# Option B: Use the host network mode
+docker run --network host comfyui-gateway
+
+# Option C: Put both containers on the same Docker network
+docker network create comfy-net
+docker run --name comfyui --network comfy-net ...
+docker run --name gateway --network comfy-net -e COMFYUI_URL=http://comfyui:8188 ...
+
+# Verify from inside the gateway container
+docker exec -it gateway sh -c "wget -qO- http://comfyui:8188/ || echo FAIL"
+```
+
+### 1d. WSL2 Networking
+
+**Symptom**: Gateway running on Windows/WSL2 cannot reach ComfyUI running on the
+other side (host vs WSL or vice-versa).
+
+**Cause**: WSL2 uses a virtual network adapter. The WSL2 guest and Windows host
+have different IP addresses.
+
+**Solution**:
+
+```bash
+# From WSL2, get the Windows host IP
+cat /etc/resolv.conf | grep nameserver | awk '{print $2}'
+# Example output: 172.25.192.1
+
+# Set COMFYUI_URL to that IP
+COMFYUI_URL=http://172.25.192.1:8188
+
+# Alternatively, if ComfyUI runs inside WSL2 and the gateway is on Windows:
+# Find WSL2 IP
+wsl hostname -I
+# Example output: 172.25.198.5
+# Set: COMFYUI_URL=http://172.25.198.5:8188
+
+# Make sure ComfyUI is listening on 0.0.0.0, not just 127.0.0.1
+# Launch ComfyUI with: python main.py --listen 0.0.0.0
+```
+
+### 1e. ComfyUI Not Started or Crashed
+
+**Symptom**: Port is not listening at all.
+
+**Cause**: ComfyUI process is not running.
+
+**Solution**:
+
+```bash
+# Check if the process is running
+# Linux
+ps aux | grep "main.py"
+# Windows
+tasklist | findstr python
+
+# Start ComfyUI
+cd /path/to/ComfyUI
+python main.py --listen 0.0.0.0 --port 8188
+
+# Check logs for startup errors
+python main.py --listen 0.0.0.0 --port 8188 2>&1 | tail -50
+
+# Verify it is accepting connections
+curl -s http://127.0.0.1:8188/ && echo "OK" || echo "NOT REACHABLE"
+```
+
+---
+
+## 2. OOM (Out of Memory) Errors
+
+The gateway classifies these as `COMFYUI_OOM` with `retryable: false`.
+
+### 2a. Resolution or Batch Size Too Large
+
+**Symptom**: Job fails with error containing "CUDA out of memory",
+"allocator backend out of memory", or "failed to allocate".
+
+**Cause**: The requested image dimensions or batch size exceeds available VRAM.
+
+**Solution**:
+
+```bash
+# 1. Reduce resolution in your job request
+# Instead of 2048x2048, try 1024x1024 or 768x768
+curl -X POST http://localhost:3000/jobs \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-key" \
+  -d '{
+    "workflowId": "sdxl_realism_v1",
+    "inputs": {
+      "prompt": "a mountain landscape",
+      "width": 1024,
+      "height": 1024
+    }
+  }'
+
+# 2. Reduce batch size to 1
+# Set in your job inputs: "batch_size": 1
+
+# 3. Lower the gateway-level limits in .env
+MAX_IMAGE_SIZE=1024
+MAX_BATCH_SIZE=2
+```
+
+### 2b. Too Many Steps
+
+**Symptom**: OOM occurs mid-generation, not immediately at submission.
+
+**Cause**: The sampler accumulates intermediate tensors over many steps.
+
+**Solution**:
+
+```bash
+# Reduce steps in the job inputs
+# Instead of 50 steps, try 20-30
+curl -X POST http://localhost:3000/jobs \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-key" \
+  -d '{
+    "workflowId": "sdxl_realism_v1",
+    "inputs": {
+      "prompt": "a portrait photo",
+      "steps": 20,
+      "width": 1024,
+      "height": 1024
+    }
+  }'
+```
+
+### 2c. Model Quantization
+
+**Symptom**: Even at low resolution, OOM errors occur because the model is too
+large for the GPU (common on 8 GB VRAM cards with SDXL).
+
+**Cause**: Full-precision (fp32) or half-precision (fp16) model weights exceed
+available VRAM.
+
+**Solution**:
+
+```bash
+# In ComfyUI, use fp8 or quantized checkpoints
+# Update your workflow template to use a quantized model:
+# e.g., "ckpt_name": "sdxl_base_1.0_fp8.safetensors"
+
+# Or add --fp8_e4m3fn-unet flag when starting ComfyUI
+python main.py --listen 0.0.0.0 --fp8_e4m3fn-unet
+
+# Monitor VRAM usage
+nvidia-smi -l 2
+```
+
+### 2d. VAE Tiling
+
+**Symptom**: OOM happens during the VAE decode step (after sampling completes).
+
+**Cause**: The VAE decoder processes the entire latent at once, which can be very
+memory-intensive at high resolutions.
+
+**Solution**:
+
+```
+Enable VAE tiling in your ComfyUI workflow by adding a "VAEDecodeTiled" node
+instead of "VAEDecode". Tile size of 512 is a good default.
+
+In the workflow JSON template:
+{
+  "10": {
+    "class_type": "VAEDecodeTiled",
+    "inputs": {
+      "samples": ["3", 0],
+      "vae": ["4", 2],
+      "tile_size": 512
+    }
+  }
+}
+```
+
+---
+
+## 3. Slow Generation
+
+### 3a. GPU Not Being Utilized
+
+**Symptom**: Jobs complete but take much longer than expected. GPU utilization
+stays near 0%.
+
+**Cause**: ComfyUI is falling back to CPU inference, or the wrong GPU is selected.
+
+**Solution**:
+
+```bash
+# 1. Check GPU utilization during a job
+nvidia-smi -l 1
+# Look for "GPU-Util" column -- should be 80-100% during sampling
+
+# 2. Verify CUDA is available in ComfyUI
+# Check ComfyUI startup logs for "Using device: cuda"
+
+# 3. Force GPU selection (multi-GPU systems)
+CUDA_VISIBLE_DEVICES=0 python main.py --listen 0.0.0.0
+
+# 4. Verify PyTorch sees the GPU
+python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"
+```
+
+### 3b. Model Loading on Every Job
+
+**Symptom**: First job is slow, subsequent jobs with the same workflow are faster,
+but switching workflows causes long delays.
+
+**Cause**: ComfyUI loads the model from disk each time a different checkpoint is
+requested. This can take 10-30 seconds per model load.
+
+**Solution**:
+
+```bash
+# 1. Increase ComfyUI's model cache
+# Start ComfyUI with a larger cache (default is 1 model):
+python main.py --listen 0.0.0.0 --cache-size 3
+
+# 2. Use the same checkpoint across workflows when possible
+# Standardize on one checkpoint (e.g., sdxl_base_1.0.safetensors)
+
+# 3. Place models on an SSD, not an HDD
+# Move ComfyUI/models/ to an NVMe drive for faster load times
+```
+
+### 3c. Queue Depth / Concurrency
+
+**Symptom**: Jobs are queued for a long time before starting.
+The job stays in `status: "queued"` for minutes.
+
+**Cause**: The worker concurrency is set to 1 (default) and multiple jobs are
+queued, or the single slot is occupied by a long-running job.
+
+**Solution**:
+
+```bash
+# 1. Check current queue state
+curl -s http://localhost:3000/jobs?status=queued | jq '.count'
+curl -s http://localhost:3000/jobs?status=running | jq '.count'
+
+# 2. Increase concurrency if your GPU can handle it (multi-batch)
+# Edit .env:
+MAX_CONCURRENCY=2
+
+# WARNING: Only increase if you have enough VRAM for parallel jobs.
+# Two concurrent 1024x1024 SDXL jobs need ~20+ GB VRAM.
+
+# 3. For multi-GPU setups, run multiple worker processes
+# Terminal 1: CUDA_VISIBLE_DEVICES=0 npm run start:worker
+# Terminal 2: CUDA_VISIBLE_DEVICES=1 npm run start:worker
+# Both connect to the same Redis queue
+```
+
+### 3d. ComfyUI Startup Time
+
+**Symptom**: The very first job after starting ComfyUI takes 30-60 seconds even
+for a simple generation.
+
+**Cause**: ComfyUI performs initialization (loading nodes, compiling, warming up
+CUDA) on the first prompt.
+
+**Solution**:
+
+```bash
+# 1. Send a warm-up job immediately after starting ComfyUI
+# This is a tiny 64x64 generation that forces initialization
+curl -X POST http://localhost:3000/jobs \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-key" \
+  -d '{
+    "workflowId": "sdxl_realism_v1",
+    "inputs": {
+      "prompt": "test",
+      "width": 64,
+      "height": 64,
+      "steps": 1
+    }
+  }'
+
+# 2. Increase the gateway timeout to account for cold starts
+COMFYUI_TIMEOUT_MS=600000
+```
+
+---
+
+## 4. Webhook Failures
+
+Webhook errors appear in logs as `WEBHOOK_DELIVERY_FAILED`.
+
+### 4a. DNS Resolution Failure
+
+**Symptom**: Webhook fails with "getaddrinfo ENOTFOUND" or "DNS lookup failed".
+
+**Cause**: The callback URL hostname cannot be resolved.
+
+**Solution**:
+
+```bash
+# 1. Test DNS resolution from the gateway host
+nslookup your-webhook-domain.com
+dig your-webhook-domain.com
+
+# 2. If using a local hostname (e.g., within Docker), make sure it is resolvable
+# Add to /etc/hosts if needed:
+echo "192.168.1.50 my-webhook-server" | sudo tee -a /etc/hosts
+
+# 3. Verify the callback URL is correct in your job request
+curl -X POST http://localhost:3000/jobs \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-key" \
+  -d '{
+    "workflowId": "sdxl_realism_v1",
+    "inputs": { "prompt": "test" },
+    "callbackUrl": "https://your-valid-domain.com/webhook"
+  }'
+```
+
+### 4b. SSL Certificate Errors
+
+**Symptom**: Webhook fails with "self signed certificate", "CERT_HAS_EXPIRED",
+or "unable to verify the first certificate".
+
+**Cause**: The webhook receiver uses an invalid, expired, or self-signed SSL certificate.
+
+**Solution**:
+
+```bash
+# 1. Test the certificate manually
+openssl s_client -connect your-webhook-domain.com:443 -servername your-webhook-domain.com < /dev/null 2>&1 | head -20
+
+# 2. Check expiration
+echo | openssl s_client -connect your-webhook-domain.com:443 2>/dev/null | openssl x509 -noout -dates
+
+# 3. For development with self-signed certs, set NODE_TLS_REJECT_UNAUTHORIZED
+# WARNING: Do NOT use this in production
+NODE_TLS_REJECT_UNAUTHORIZED=0 npm run dev
+
+# 4. For production, fix the certificate (use Let's Encrypt or a valid CA)
+```
+
+### 4c. Webhook Timeout
+
+**Symptom**: Webhook logs show "AbortError" or "Webhook POST timed out".
+
+**Cause**: The webhook receiver takes longer than 10 seconds to respond.
+The gateway has a hardcoded 10-second timeout per webhook attempt with
+3 retries and exponential backoff.
+
+**Solution**:
+
+```bash
+# 1. Ensure your webhook receiver responds quickly
+# The receiver should return 200 immediately and process asynchronously
+# BAD:  app.post("/webhook", async (req, res) => { await longProcess(); res.send("ok"); })
+# GOOD: app.post("/webhook", (req, res) => { res.send("ok"); enqueueWork(req.body); })
+
+# 2. Test receiver response time
+time curl -s -o /dev/null -w "%{time_total}" -X POST https://your-webhook.com/callback \
+  -H "Content-Type: application/json" -d '{"test": true}'
+# Should be < 2 seconds
+```
+
+### 4d. Domain Not in Allowlist
+
+**Symptom**: Job creation fails with `Callback domain "example.com" is not in the
+allowed domains list`.
+
+**Cause**: `WEBHOOK_ALLOWED_DOMAINS` is configured and does not include the
+callback URL's domain.
+
+**Solution**:
+
+```bash
+# 1. Check current setting
+grep WEBHOOK_ALLOWED_DOMAINS .env
+
+# 2. Add the domain (comma-separated list)
+WEBHOOK_ALLOWED_DOMAINS=your-app.com,n8n.your-domain.com,*.internal.company.com
+
+# 3. Or allow all domains (less secure, suitable for development)
+WEBHOOK_ALLOWED_DOMAINS=*
+
+# 4. Restart the gateway
+npm run dev
+```
+
+### 4e. HMAC Signature Mismatch
+
+**Symptom**: Your webhook receiver receives the POST but HMAC validation fails
+on your end.
+
+**Cause**: The `WEBHOOK_SECRET` configured in the gateway does not match the secret
+your receiver uses to validate signatures, or the signature computation differs.
+
+**Solution**:
+
+```bash
+# 1. Verify the WEBHOOK_SECRET matches on both sides
+grep WEBHOOK_SECRET .env
+
+# 2. The gateway sends: X-Signature: sha256=<hex>
+# Computed as: HMAC-SHA256(secret, raw_body_string)
+# Verify in Node.js:
+node -e "
+const crypto = require('crypto');
+const secret = 'your-webhook-secret';
+const body = '{\"jobId\":\"test\",\"status\":\"succeeded\"}';
+const sig = crypto.createHmac('sha256', secret).update(body, 'utf8').digest('hex');
+console.log('Expected header: sha256=' + sig);
+"
+
+# 3. Common mistakes:
+# - Parsing the body before computing HMAC (must use raw string)
+# - Using different encodings (gateway uses utf8)
+# - Comparing strings case-sensitively (hex is lowercase)
+```
+
+---
+
+## 5. Redis Connection Issues
+
+### 5a. Cannot Connect to Redis
+
+**Symptom**: Gateway crashes at startup with "Redis connection error" or
+"ECONNREFUSED" targeting the Redis port.
+
+**Cause**: Redis server is not running, or the `REDIS_URL` is wrong.
+
+**Solution**:
+
+```bash
+# 1. Check if Redis is running
+redis-cli ping
+# Expected: PONG
+
+# 2. Verify the URL format
+# Correct formats:
+#   redis://localhost:6379
+#   redis://:yourpassword@redis-host:6379/0
+#   rediss://user:password@host:6380/0  (TLS)
+
+# 3. Test connectivity
+redis-cli -u "redis://localhost:6379" ping
+
+# 4. If Redis is not needed, remove REDIS_URL to use in-memory queue
+# Edit .env:
+REDIS_URL=
+# The gateway falls back to an in-memory queue automatically
+```
+
+### 5b. Redis Authentication Failure
+
+**Symptom**: Error message contains "NOAUTH Authentication required" or
+"ERR invalid password".
+
+**Cause**: Redis requires a password but `REDIS_URL` does not include one,
+or the password is wrong.
+
+**Solution**:
+
+```bash
+# 1. Include the password in the URL
+REDIS_URL=redis://:your_redis_password@localhost:6379/0
+
+# 2. Test with redis-cli
+redis-cli -a "your_redis_password" ping
+
+# 3. Check Redis config for requirepass
+redis-cli CONFIG GET requirepass
+```
+
+### 5c. Fallback to In-Memory Queue
+
+**Symptom**: Logs show "No Redis URL configured, using in-memory queue" and
+you expected BullMQ.
+
+**Cause**: `REDIS_URL` is empty or not set in `.env`.
+
+**Solution**:
+
+```bash
+# 1. Set REDIS_URL in .env
+REDIS_URL=redis://localhost:6379
+
+# 2. Verify Redis is running
+redis-cli ping
+
+# 3. Restart the gateway
+npm run dev
+
+# 4. Confirm in logs: should show "Redis URL configured, using BullMQ worker"
+```
+
+> **Note**: The in-memory queue is fine for single-instance development deployments.
+> For production with multiple workers or durability requirements, use Redis + BullMQ.
+
+---
+
+## 6. Storage Errors
+
+### 6a. Local Disk Permission Denied
+
+**Symptom**: Job fails at the output storage step with "EACCES: permission denied"
+or `STORAGE_READ_ERROR`.
+
+**Cause**: The gateway process does not have write permissions to `STORAGE_LOCAL_PATH`.
+
+**Solution**:
+
+```bash
+# 1. Check the configured path
+grep STORAGE_LOCAL_PATH .env
+# Default: ./data/outputs
+
+# 2. Ensure the directory exists and is writable
+mkdir -p ./data/outputs
+chmod 755 ./data/outputs
+
+# 3. Check ownership
+ls -la ./data/
+
+# 4. If running as a different user (e.g., in Docker)
+chown -R node:node ./data/outputs
+
+# 5. For Docker, mount a volume with correct permissions
+# docker run -v /host/path/outputs:/app/data/outputs ...
+```
+
+### 6b. S3 Credentials Invalid
+
+**Symptom**: Job fails with `STORAGE_S3_PUT_ERROR` and the underlying error
+mentions "InvalidAccessKeyId", "SignatureDoesNotMatch", or "AccessDenied".
+
+**Cause**: The `S3_ACCESS_KEY` / `S3_SECRET_KEY` are wrong, expired, or the
+IAM policy does not grant `s3:PutObject` permission.
+
+**Solution**:
+
+```bash
+# 1. Verify credentials are set
+grep S3_ACCESS_KEY .env
+grep S3_SECRET_KEY .env
+grep S3_BUCKET .env
+
+# 2. Test with AWS CLI
+aws s3 ls s3://your-bucket/ \
+  --endpoint-url http://your-minio:9000 \
+  --region us-east-1
+
+# 3. Test a put operation
+echo "test" > /tmp/test.txt
+aws s3 cp /tmp/test.txt s3://your-bucket/test.txt \
+  --endpoint-url http://your-minio:9000
+
+# 4. Minimum IAM policy for the gateway:
+# {
+#   "Version": "2012-10-17",
+#   "Statement": [{
+#     "Effect": "Allow",
+#     "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListBucket"],
+#     "Resource": ["arn:aws:s3:::your-bucket", "arn:aws:s3:::your-bucket/*"]
+#   }]
+# }
+```
+
+### 6c. MinIO Configuration
+
+**Symptom**: S3 storage fails with "socket hang up", "ECONNREFUSED", or
+"Bucket does not exist".
+
+**Cause**: MinIO endpoint is wrong, the bucket has not been created, or
+`forcePathStyle` is not enabled (handled automatically by the gateway).
+
+**Solution**:
+
+```bash
+# 1. Verify MinIO is running
+curl http://localhost:9000/minio/health/live
+# Expected: HTTP 200
+
+# 2. Set the correct endpoint in .env
+S3_ENDPOINT=http://localhost:9000
+S3_BUCKET=comfyui-outputs
+S3_ACCESS_KEY=minioadmin
+S3_SECRET_KEY=minioadmin
+S3_REGION=us-east-1
+
+# 3. Create the bucket if it does not exist
+# Using mc (MinIO Client)
+mc alias set local http://localhost:9000 minioadmin minioadmin
+mc mb local/comfyui-outputs
+
+# Or using AWS CLI
+aws s3 mb s3://comfyui-outputs --endpoint-url http://localhost:9000
+```
+
+---
+
+## 7. Database Issues
+
+### 7a. SQLite WAL Lock Errors
+
+**Symptom**: Intermittent "SQLITE_BUSY" or "database is locked" errors under
+concurrent load.
+
+**Cause**: Multiple processes or threads are writing to the SQLite database
+simultaneously. SQLite WAL mode supports concurrent readers but only one writer.
+
+**Solution**:
+
+```bash
+# 1. The gateway already sets optimal pragmas:
+#    journal_mode = WAL
+#    synchronous = NORMAL
+#    busy_timeout = 5000 (5 seconds)
+
+# 2. If running multiple gateway instances, switch to Postgres
+DATABASE_URL=postgresql://user:password@localhost:5432/comfyui_gateway
+
+# 3. If you must use SQLite with a single instance, increase busy timeout
+# (requires code change or env override):
+# The default 5000ms should be sufficient for most single-instance use cases
+
+# 4. Check for stuck WAL files
+ls -la ./data/gateway.db*
+# You should see: gateway.db, gateway.db-wal, gateway.db-shm
+
+# 5. If the database is corrupted, try recovery
+sqlite3 ./data/gateway.db "PRAGMA integrity_check;"
+# If it reports errors, back up and recreate:
+cp ./data/gateway.db ./data/gateway.db.bak
+sqlite3 ./data/gateway.db ".recover" | sqlite3 ./data/gateway_recovered.db
+```
+
+### 7b. Postgres Connection Pooling
+
+**Symptom**: Errors like "too many clients already", "remaining connection slots
+are reserved", or intermittent "Connection terminated unexpectedly".
+
+**Cause**: The gateway opens too many connections to Postgres, exceeding
+`max_connections`, or connections are not being properly returned to the pool.
+
+**Solution**:
+
+```bash
+# 1. Check current connections in Postgres
+psql -c "SELECT count(*) FROM pg_stat_activity WHERE datname = 'comfyui_gateway';"
+
+# 2. Check max_connections setting
+psql -c "SHOW max_connections;"
+
+# 3. Use a connection pooler like PgBouncer
+# Install PgBouncer and point DATABASE_URL to it
+DATABASE_URL=postgresql://user:password@localhost:6432/comfyui_gateway
+
+# 4. If running multiple gateway instances, ensure the total pool size
+# across all instances does not exceed Postgres max_connections
+```
+
+### 7c. Database URL Format
+
+**Symptom**: Gateway crashes at startup with "Invalid connection string" or
+uses SQLite when you intended Postgres.
+
+**Cause**: The `DATABASE_URL` format is wrong. The gateway checks if the URL
+starts with `postgres://` or `postgresql://` to select the Postgres backend.
+
+**Solution**:
+
+```bash
+# SQLite formats (all valid):
+DATABASE_URL=./data/gateway.db
+DATABASE_URL=/absolute/path/to/gateway.db
+
+# Postgres formats (must start with postgres:// or postgresql://):
+DATABASE_URL=postgresql://user:password@localhost:5432/comfyui_gateway
+DATABASE_URL=postgres://user:password@host:5432/dbname?sslmode=require
+```
+
+---
+
+## 8. Job Stuck in "running"
+
+### 8a. ComfyUI Crashed During Execution
+
+**Symptom**: A job shows `status: "running"` indefinitely. No progress updates.
+The gateway health endpoint may show `comfyui.reachable: false`.
+
+**Cause**: ComfyUI crashed (segfault, CUDA error, killed by OOM killer) while
+processing the job, and the gateway's WebSocket connection was severed.
+
+**Solution**:
+
+```bash
+# 1. Check job status
+curl -s http://localhost:3000/jobs/<jobId> | jq '.status'
+
+# 2. Check if ComfyUI is still running
+curl -s http://localhost:3000/health | jq '.comfyui.reachable'
+
+# 3. If ComfyUI crashed, restart it
+cd /path/to/ComfyUI
+python main.py --listen 0.0.0.0
+
+# 4. The stuck job will eventually time out (COMFYUI_TIMEOUT_MS, default 5 min)
+# and be marked as failed with COMFYUI_TIMEOUT
+
+# 5. To immediately cancel the stuck job
+curl -X POST http://localhost:3000/jobs/<jobId>/cancel \
+  -H "X-API-Key: your-key"
+
+# 6. To reduce timeout for faster failure detection
+COMFYUI_TIMEOUT_MS=120000
+```
+
+### 8b. WebSocket Disconnection
+
+**Symptom**: Job stays "running" but ComfyUI is actually done. The output
+exists in ComfyUI's history.
+
+**Cause**: The WebSocket connection dropped mid-execution, and the polling
+fallback failed to pick up the result.
+
+**Solution**:
+
+```bash
+# 1. Check ComfyUI history directly
+curl -s http://127.0.0.1:8188/history | jq 'keys | length'
+
+# 2. The gateway automatically falls back to HTTP polling if WebSocket fails.
+# If polling also fails, the job times out.
+
+# 3. Restart the gateway to reset connections
+npm run dev
+
+# 4. Check network stability between gateway and ComfyUI
+ping -c 10 <comfyui-host>
+```
+
+### 8c. Restart Recovery
+
+**Symptom**: After restarting the gateway, jobs that were "running" remain
+in that state permanently.
+
+**Cause**: The in-memory queue loses track of running jobs when the process
+restarts. There is no automatic recovery for in-memory jobs.
+
+**Solution**:
+
+```bash
+# 1. For production, use Redis (BullMQ) for durable job queues
+REDIS_URL=redis://localhost:6379
+
+# 2. Manually fail stuck jobs via the database
+sqlite3 ./data/gateway.db \
+  "UPDATE jobs SET status='failed', errorJson='{\"code\":\"GATEWAY_RESTART\",\"message\":\"Job interrupted by gateway restart\"}', completedAt=datetime('now') WHERE status='running';"
+
+# 3. Verify
+sqlite3 ./data/gateway.db "SELECT id, status FROM jobs WHERE status='running';"
+```
+
+---
+
+## 9. Rate Limiting Issues
+
+### 9a. Identifying You Are Being Rate Limited
+
+**Symptom**: API returns HTTP 429 with body `{ "error": "RATE_LIMITED" }` and
+a `Retry-After` header.
+
+**Cause**: You exceeded `RATE_LIMIT_MAX` requests within the `RATE_LIMIT_WINDOW_MS`
+window. Limits are applied per API key or per IP.
+
+**Solution**:
+
+```bash
+# 1. Check the response headers
+curl -v http://localhost:3000/health -H "X-API-Key: your-key" 2>&1 | grep -i "x-ratelimit"
+# X-RateLimit-Limit: 100
+# X-RateLimit-Remaining: 0
+# Retry-After: 42
+
+# 2. Wait for the Retry-After period, then retry
+
+# 3. Implement exponential backoff in your client
+```
+
+### 9b. Adjusting Rate Limits
+
+**Symptom**: Legitimate usage is being throttled.
+
+**Cause**: Default limits (100 requests/minute) are too low for your workload.
+
+**Solution**:
+
+```bash
+# 1. Increase the limit in .env
+RATE_LIMIT_MAX=500
+RATE_LIMIT_WINDOW_MS=60000
+
+# 2. For burst workloads, widen the window
+RATE_LIMIT_MAX=1000
+RATE_LIMIT_WINDOW_MS=300000
+
+# 3. Restart the gateway
+npm run dev
+
+# 4. Note: Rate limits are per API key (if authenticated) or per IP.
+# Different API keys have independent counters.
+```
+
+### 9c. Rate Limit Per API Key vs Per IP
+
+**Symptom**: Different clients sharing the same IP are interfering with each
+other's rate limits.
+
+**Cause**: Without API keys, all requests from the same IP share a single
+rate-limit bucket.
+
+**Solution**:
+
+```bash
+# 1. Assign unique API keys to each client
+API_KEYS=client1-key:user,client2-key:user,admin-key:admin
+
+# 2. Each client uses its own X-API-Key header
+# Client 1: -H "X-API-Key: client1-key"
+# Client 2: -H "X-API-Key: client2-key"
+
+# 3. Each key gets its own independent rate-limit counter
+```
+
+---
+
+## 10. Authentication Problems
+
+### 10a. API Key Not Accepted
+
+**Symptom**: Every request returns HTTP 401 with `{ "error": "AUTH_FAILED",
+"message": "Invalid API key" }`.
+
+**Cause**: The `X-API-Key` header value does not match any entry in `API_KEYS`.
+
+**Solution**:
+
+```bash
+# 1. Check configured keys
+grep API_KEYS .env
+# Format: key1:admin,key2:user
+
+# 2. Ensure your request uses the exact key (no extra whitespace)
+curl -H "X-API-Key: mykey123" http://localhost:3000/health
+
+# 3. Keys are case-sensitive and matched exactly
+
+# 4. If API_KEYS is empty, authentication is DISABLED (development mode)
+# All requests are treated as admin. Set keys for production:
+API_KEYS=sk-prod-abc123:admin,sk-user-xyz789:user
+```
+
+### 10b. JWT Token Expired
+
+**Symptom**: Request returns `{ "error": "AUTH_FAILED", "message": "JWT token
+has expired" }`.
+
+**Cause**: The JWT `exp` claim is in the past.
+
+**Solution**:
+
+```bash
+# 1. Decode the JWT to check expiration (without verification)
+echo "<your-token>" | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '.exp'
+
+# 2. Compare with current time
+date +%s
+
+# 3. Generate a new token with a longer TTL
+# Example using Node.js:
+node -e "
+const crypto = require('crypto');
+const secret = 'your-jwt-secret';
+const header = Buffer.from(JSON.stringify({alg:'HS256',typ:'JWT'})).toString('base64url');
+const payload = Buffer.from(JSON.stringify({
+  sub: 'user-1',
+  role: 'admin',
+  iat: Math.floor(Date.now()/1000),
+  exp: Math.floor(Date.now()/1000) + 86400  // 24 hours
+})).toString('base64url');
+const sig = crypto.createHmac('sha256', secret).update(header+'.'+payload).digest('base64url');
+console.log(header+'.'+payload+'.'+sig);
+"
+```
+
+### 10c. JWT Signature Invalid
+
+**Symptom**: Request returns `{ "error": "AUTH_FAILED", "message": "Invalid JWT
+signature" }`.
+
+**Cause**: The JWT was signed with a different secret than what is configured in
+`JWT_SECRET`.
+
+**Solution**:
+
+```bash
+# 1. Verify the secret matches on token-issuer side and gateway side
+grep JWT_SECRET .env
+
+# 2. The gateway uses HMAC-SHA256 (HS256) exclusively
+# Make sure your token issuer also uses HS256 with the same secret
+
+# 3. Re-generate the token using the correct secret
+```
+
+### 10d. No Authentication Header Provided
+
+**Symptom**: Request returns `{ "error": "AUTH_FAILED", "message": "Authentication
+required. Provide X-API-Key header or Authorization: Bearer token." }`.
+
+**Cause**: The request has no `X-API-Key` header and no `Authorization: Bearer`
+header, and authentication is enabled (API_KEYS or JWT_SECRET is set).
+
+**Solution**:
+
+```bash
+# Option A: Use API Key
+curl -H "X-API-Key: your-key" http://localhost:3000/health
+
+# Option B: Use JWT Bearer token
+curl -H "Authorization: Bearer your.jwt.token" http://localhost:3000/health
+
+# Option C: Disable auth for development (NOT for production)
+# Remove all values from API_KEYS and JWT_SECRET in .env:
+API_KEYS=
+JWT_SECRET=
+```
+
+### 10e. Insufficient Permissions (Forbidden)
+
+**Symptom**: Request returns HTTP 403 with `{ "error": "FORBIDDEN", "message":
+"Admin role required for this operation" }`.
+
+**Cause**: You are using a `user` role key to perform an admin-only action
+(workflow CRUD).
+
+**Solution**:
+
+```bash
+# 1. Check which role your key has
+grep API_KEYS .env
+# Example: sk-user-key:user,sk-admin-key:admin
+
+# 2. Use the admin key for workflow management
+curl -H "X-API-Key: sk-admin-key" -X POST http://localhost:3000/workflows ...
+
+# 3. User role can: create jobs, read own jobs, view health/capabilities
+# Admin role can: everything the user can + workflow CRUD + view all jobs
+```
+
+---
+
+## Quick Diagnostic Commands
+
+```bash
+# Gateway health
+curl -s http://localhost:3000/health | jq .
+
+# ComfyUI direct connectivity
+curl -s http://127.0.0.1:8188/ | head -5
+
+# Queue status
+curl -s http://localhost:3000/jobs?status=queued -H "X-API-Key: KEY" | jq '.count'
+curl -s http://localhost:3000/jobs?status=running -H "X-API-Key: KEY" | jq '.count'
+
+# GPU memory
+nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader
+
+# Redis connectivity
+redis-cli -u "$REDIS_URL" ping
+
+# SQLite integrity
+sqlite3 ./data/gateway.db "PRAGMA integrity_check;"
+
+# Logs (if using pino-pretty)
+npm run dev 2>&1 | npx pino-pretty
+
+# Check all configured environment variables
+grep -v '^#' .env | grep -v '^$'
+```