@vodailoc/kilo-kit-mcp 1.1.0 → 1.2.0

package/skills/engineering/backend-development/references/backend-performance.md

@@ -1,397 +1,397 @@
-# Backend Performance & Scalability
-
-Performance optimization strategies, caching patterns, and scalability best practices (2025).
-
-## Database Performance
-
-### Query Optimization
-
-#### Indexing Strategies
-
-**Impact:** 30% disk I/O reduction, 10-100x query speedup
-
-```sql
--- Create index on frequently queried columns
-CREATE INDEX idx_users_email ON users(email);
-CREATE INDEX idx_orders_user_id ON orders(user_id);
-
--- Composite index for multi-column queries
-CREATE INDEX idx_orders_user_date ON orders(user_id, created_at DESC);
-
--- Partial index for filtered queries
-CREATE INDEX idx_active_users ON users(email) WHERE active = true;
-
--- Analyze query performance
-EXPLAIN ANALYZE SELECT * FROM orders
-WHERE user_id = 123 AND created_at > '2025-01-01';
-```
-
-**Index Types:**
-- **B-tree** - Default, general-purpose (equality, range queries)
-- **Hash** - Fast equality lookups, no range queries
-- **GIN** - Full-text search, JSONB queries
-- **GiST** - Geospatial queries, range types
-
-**When NOT to Index:**
-- Small tables (<1000 rows)
-- Frequently updated columns
-- Low-cardinality columns (e.g., boolean with 2 values)
-
-### Connection Pooling
-
-**Impact:** 5-10x performance improvement
-
-```typescript
-// PostgreSQL with pg-pool
-import { Pool } from 'pg';
-
-const pool = new Pool({
-  host: process.env.DB_HOST,
-  database: process.env.DB_NAME,
-  user: process.env.DB_USER,
-  password: process.env.DB_PASSWORD,
-  max: 20, // Maximum connections
-  min: 5, // Minimum connections
-  idleTimeoutMillis: 30000, // Close idle connections after 30s
-  connectionTimeoutMillis: 2000, // Error if can't connect in 2s
-});
-
-// Use pool for queries
-const result = await pool.query('SELECT * FROM users WHERE id = $1', [userId]);
-```
-
-**Recommended Pool Sizes:**
-- **Web servers:** `connections = (core_count * 2) + effective_spindle_count`
-- **Typical:** 20-30 connections per app instance
-- **Monitor:** Connection saturation in production
-
-### N+1 Query Problem
-
-**Bad: N+1 queries**
-```typescript
-// Fetches 1 query for posts, then N queries for authors
-const posts = await Post.findAll();
-for (const post of posts) {
-  post.author = await User.findById(post.authorId); // N queries!
-}
-```
-
-**Good: Join or eager loading**
-```typescript
-// Single query with JOIN
-const posts = await Post.findAll({
-  include: [{ model: User, as: 'author' }],
-});
-```
-
-## Caching Strategies
-
-### Redis Caching
-
-**Impact:** 90% DB load reduction, 10-100x faster response
-
-#### Cache-Aside Pattern (Lazy Loading)
-
-```typescript
-async function getUser(userId: string) {
-  // Try cache first
-  const cached = await redis.get(`user:${userId}`);
-  if (cached) return JSON.parse(cached);
-
-  // Cache miss - fetch from DB
-  const user = await db.users.findById(userId);
-
-  // Store in cache (TTL: 1 hour)
-  await redis.setex(`user:${userId}`, 3600, JSON.stringify(user));
-
-  return user;
-}
-```
-
-#### Write-Through Pattern
-
-```typescript
-async function updateUser(userId: string, data: UpdateUserDto) {
-  // Update database
-  const user = await db.users.update(userId, data);
-
-  // Update cache immediately
-  await redis.setex(`user:${userId}`, 3600, JSON.stringify(user));
-
-  return user;
-}
-```
-
-#### Cache Invalidation
-
-```typescript
-// Invalidate on update
-async function deleteUser(userId: string) {
-  await db.users.delete(userId);
-  await redis.del(`user:${userId}`);
-  await redis.del(`user:${userId}:posts`); // Invalidate related caches
-}
-
-// Pattern-based invalidation
-await redis.keys('user:*').then(keys => redis.del(...keys));
-```
-
-### Cache Layers
-
-```
-Client
-  → CDN Cache (static assets, 50%+ latency reduction)
-  → API Gateway Cache (public endpoints)
-  → Application Cache (Redis)
-  → Database Query Cache
-  → Database
-```
-
-### Cache Best Practices
-
-1. **Cache frequently accessed data** - User profiles, config, product catalogs
-2. **Set appropriate TTL** - Balance freshness vs performance
-3. **Invalidate on write** - Keep cache consistent
-4. **Use cache keys wisely** - `resource:id:attribute` pattern
-5. **Monitor hit rates** - Target >80% hit rate
-
-## Load Balancing
-
-### Algorithms
-
-**Round Robin** - Distribute evenly across servers
-```nginx
-upstream backend {
-    server backend1.example.com;
-    server backend2.example.com;
-    server backend3.example.com;
-}
-```
-
-**Least Connections** - Route to server with fewest connections
-```nginx
-upstream backend {
-    least_conn;
-    server backend1.example.com;
-    server backend2.example.com;
-}
-```
-
-**IP Hash** - Same client → same server (session affinity)
-```nginx
-upstream backend {
-    ip_hash;
-    server backend1.example.com;
-    server backend2.example.com;
-}
-```
-
-### Health Checks
-
-```typescript
-// Express health check endpoint
-app.get('/health', async (req, res) => {
-  const checks = {
-    uptime: process.uptime(),
-    timestamp: Date.now(),
-    database: await checkDatabase(),
-    redis: await checkRedis(),
-    memory: process.memoryUsage(),
-  };
-
-  const isHealthy = checks.database && checks.redis;
-  res.status(isHealthy ? 200 : 503).json(checks);
-});
-```
-
-## Asynchronous Processing
-
-### Message Queues for Long-Running Tasks
-
-```typescript
-// Producer - Add job to queue
-import Queue from 'bull';
-
-const emailQueue = new Queue('email', {
-  redis: { host: 'localhost', port: 6379 },
-});
-
-await emailQueue.add('send-welcome', {
-  userId: user.id,
-  email: user.email,
-});
-
-// Consumer - Process jobs
-emailQueue.process('send-welcome', async (job) => {
-  await sendWelcomeEmail(job.data.email);
-});
-```
-
-**Use Cases:**
-- Email sending
-- Image/video processing
-- Report generation
-- Data export
-- Webhook delivery
-
-## CDN (Content Delivery Network)
-
-**Impact:** 50%+ latency reduction for global users
-
-### Configuration
-
-```typescript
-// Cache-Control headers
-res.setHeader('Cache-Control', 'public, max-age=31536000, immutable'); // Static assets
-res.setHeader('Cache-Control', 'public, max-age=3600'); // API responses
-res.setHeader('Cache-Control', 'private, no-cache'); // User-specific data
-```
-
-**CDN Providers:**
-- Cloudflare (generous free tier, global coverage)
-- AWS CloudFront (AWS integration)
-- Fastly (real-time purging)
-
-## Horizontal vs Vertical Scaling
-
-### Horizontal Scaling (Scale Out)
-
-**Pros:**
-- Better fault tolerance
-- Unlimited scaling potential
-- Cost-effective (commodity hardware)
-
-**Cons:**
-- Complex architecture
-- Data consistency challenges
-- Network overhead
-
-**When to use:** High traffic, need redundancy, stateless applications
-
-### Vertical Scaling (Scale Up)
-
-**Pros:**
-- Simple architecture
-- No code changes needed
-- Easier data consistency
-
-**Cons:**
-- Hardware limits
-- Single point of failure
-- Expensive at high end
-
-**When to use:** Monolithic apps, rapid scaling needed, data consistency critical
-
-## Database Scaling Patterns
-
-### Read Replicas
-
-```
-Primary (Write) → Replica 1 (Read)
-               → Replica 2 (Read)
-               → Replica 3 (Read)
-```
-
-**Implementation:**
-```typescript
-// Write to primary
-await primaryDb.users.create(userData);
-
-// Read from replica
-const users = await replicaDb.users.findAll();
-```
-
-**Use Cases:**
-- Read-heavy workloads (90%+ reads)
-- Analytics queries
-- Reporting dashboards
-
-### Database Sharding
-
-**Horizontal Partitioning** - Split data across databases
-
-```typescript
-// Shard by user ID
-function getShardId(userId: string): number {
-  return hashCode(userId) % SHARD_COUNT;
-}
-
-const shardId = getShardId(userId);
-const db = shards[shardId];
-const user = await db.users.findById(userId);
-```
-
-**Sharding Strategies:**
-- **Range-based:** Users 1-1M → Shard 1, 1M-2M → Shard 2
-- **Hash-based:** Hash(userId) % shard_count
-- **Geographic:** EU users → EU shard, US users → US shard
-- **Entity-based:** Users → Shard 1, Orders → Shard 2
-
-## Performance Monitoring
-
-### Key Metrics
-
-**Application:**
-- Response time (p50, p95, p99)
-- Throughput (requests/second)
-- Error rate
-- CPU/memory usage
-
-**Database:**
-- Query execution time
-- Connection pool saturation
-- Cache hit rate
-- Slow query log
-
-**Tools:**
-- Prometheus + Grafana (metrics)
-- New Relic / Datadog (APM)
-- Sentry (error tracking)
-- OpenTelemetry (distributed tracing)
-
-## Performance Optimization Checklist
-
-### Database
-- [ ] Indexes on frequently queried columns
-- [ ] Connection pooling configured
-- [ ] N+1 queries eliminated
-- [ ] Slow query log monitored
-- [ ] Query execution plans analyzed
-
-### Caching
-- [ ] Redis cache for hot data
-- [ ] Cache TTL configured appropriately
-- [ ] Cache invalidation on writes
-- [ ] CDN for static assets
-- [ ] >80% cache hit rate achieved
-
-### Application
-- [ ] Async processing for long tasks
-- [ ] Response compression enabled (gzip)
-- [ ] Load balancing configured
-- [ ] Health checks implemented
-- [ ] Resource limits set (CPU, memory)
-
-### Monitoring
-- [ ] APM tool configured (New Relic/Datadog)
-- [ ] Error tracking (Sentry)
-- [ ] Performance dashboards (Grafana)
-- [ ] Alerting on key metrics
-- [ ] Distributed tracing for microservices
-
-## Common Performance Pitfalls
-
-1. **No caching** - Repeatedly querying same data
-2. **Missing indexes** - Full table scans
-3. **N+1 queries** - Fetching related data in loops
-4. **Synchronous processing** - Blocking on long tasks
-5. **No connection pooling** - Creating new connections per request
-6. **Unbounded queries** - No LIMIT on large tables
-7. **No CDN** - Serving static assets from origin
-
-## Resources
-
-- **PostgreSQL Performance:** https://www.postgresql.org/docs/current/performance-tips.html
-- **Redis Best Practices:** https://redis.io/docs/management/optimization/
-- **Web Performance:** https://web.dev/performance/
-- **Database Indexing:** https://use-the-index-luke.com/
+# Backend Performance & Scalability
+
+Performance optimization strategies, caching patterns, and scalability best practices (2025).
+
+## Database Performance
+
+### Query Optimization
+
+#### Indexing Strategies
+
+**Impact:** 30% disk I/O reduction, 10-100x query speedup
+
+```sql
+-- Create index on frequently queried columns
+CREATE INDEX idx_users_email ON users(email);
+CREATE INDEX idx_orders_user_id ON orders(user_id);
+
+-- Composite index for multi-column queries
+CREATE INDEX idx_orders_user_date ON orders(user_id, created_at DESC);
+
+-- Partial index for filtered queries
+CREATE INDEX idx_active_users ON users(email) WHERE active = true;
+
+-- Analyze query performance
+EXPLAIN ANALYZE SELECT * FROM orders
+WHERE user_id = 123 AND created_at > '2025-01-01';
+```
+
+**Index Types:**
+- **B-tree** - Default, general-purpose (equality, range queries)
+- **Hash** - Fast equality lookups, no range queries
+- **GIN** - Full-text search, JSONB queries
+- **GiST** - Geospatial queries, range types
+
+**When NOT to Index:**
+- Small tables (<1000 rows)
+- Frequently updated columns
+- Low-cardinality columns (e.g., boolean with 2 values)
+
+### Connection Pooling
+
+**Impact:** 5-10x performance improvement
+
+```typescript
+// PostgreSQL with pg-pool
+import { Pool } from 'pg';
+
+const pool = new Pool({
+  host: process.env.DB_HOST,
+  database: process.env.DB_NAME,
+  user: process.env.DB_USER,
+  password: process.env.DB_PASSWORD,
+  max: 20, // Maximum connections
+  min: 5, // Minimum connections
+  idleTimeoutMillis: 30000, // Close idle connections after 30s
+  connectionTimeoutMillis: 2000, // Error if can't connect in 2s
+});
+
+// Use pool for queries
+const result = await pool.query('SELECT * FROM users WHERE id = $1', [userId]);
+```
+
+**Recommended Pool Sizes:**
+- **Web servers:** `connections = (core_count * 2) + effective_spindle_count`
+- **Typical:** 20-30 connections per app instance
+- **Monitor:** Connection saturation in production
+
+### N+1 Query Problem
+
+**Bad: N+1 queries**
+```typescript
+// Fetches 1 query for posts, then N queries for authors
+const posts = await Post.findAll();
+for (const post of posts) {
+  post.author = await User.findById(post.authorId); // N queries!
+}
+```
+
+**Good: Join or eager loading**
+```typescript
+// Single query with JOIN
+const posts = await Post.findAll({
+  include: [{ model: User, as: 'author' }],
+});
+```
+
+## Caching Strategies
+
+### Redis Caching
+
+**Impact:** 90% DB load reduction, 10-100x faster response
+
+#### Cache-Aside Pattern (Lazy Loading)
+
+```typescript
+async function getUser(userId: string) {
+  // Try cache first
+  const cached = await redis.get(`user:${userId}`);
+  if (cached) return JSON.parse(cached);
+
+  // Cache miss - fetch from DB
+  const user = await db.users.findById(userId);
+
+  // Store in cache (TTL: 1 hour)
+  await redis.setex(`user:${userId}`, 3600, JSON.stringify(user));
+
+  return user;
+}
+```
+
+#### Write-Through Pattern
+
+```typescript
+async function updateUser(userId: string, data: UpdateUserDto) {
+  // Update database
+  const user = await db.users.update(userId, data);
+
+  // Update cache immediately
+  await redis.setex(`user:${userId}`, 3600, JSON.stringify(user));
+
+  return user;
+}
+```
+
+#### Cache Invalidation
+
+```typescript
+// Invalidate on update
+async function deleteUser(userId: string) {
+  await db.users.delete(userId);
+  await redis.del(`user:${userId}`);
+  await redis.del(`user:${userId}:posts`); // Invalidate related caches
+}
+
+// Pattern-based invalidation
+await redis.keys('user:*').then(keys => redis.del(...keys));
+```
+
+### Cache Layers
+
+```
+Client
+  → CDN Cache (static assets, 50%+ latency reduction)
+  → API Gateway Cache (public endpoints)
+  → Application Cache (Redis)
+  → Database Query Cache
+  → Database
+```
+
+### Cache Best Practices
+
+1. **Cache frequently accessed data** - User profiles, config, product catalogs
+2. **Set appropriate TTL** - Balance freshness vs performance
+3. **Invalidate on write** - Keep cache consistent
+4. **Use cache keys wisely** - `resource:id:attribute` pattern
+5. **Monitor hit rates** - Target >80% hit rate
+
+## Load Balancing
+
+### Algorithms
+
+**Round Robin** - Distribute evenly across servers
+```nginx
+upstream backend {
+    server backend1.example.com;
+    server backend2.example.com;
+    server backend3.example.com;
+}
+```
+
+**Least Connections** - Route to server with fewest connections
+```nginx
+upstream backend {
+    least_conn;
+    server backend1.example.com;
+    server backend2.example.com;
+}
+```
+
+**IP Hash** - Same client → same server (session affinity)
+```nginx
+upstream backend {
+    ip_hash;
+    server backend1.example.com;
+    server backend2.example.com;
+}
+```
+
+### Health Checks
+
+```typescript
+// Express health check endpoint
+app.get('/health', async (req, res) => {
+  const checks = {
+    uptime: process.uptime(),
+    timestamp: Date.now(),
+    database: await checkDatabase(),
+    redis: await checkRedis(),
+    memory: process.memoryUsage(),
+  };
+
+  const isHealthy = checks.database && checks.redis;
+  res.status(isHealthy ? 200 : 503).json(checks);
+});
+```
+
+## Asynchronous Processing
+
+### Message Queues for Long-Running Tasks
+
+```typescript
+// Producer - Add job to queue
+import Queue from 'bull';
+
+const emailQueue = new Queue('email', {
+  redis: { host: 'localhost', port: 6379 },
+});
+
+await emailQueue.add('send-welcome', {
+  userId: user.id,
+  email: user.email,
+});
+
+// Consumer - Process jobs
+emailQueue.process('send-welcome', async (job) => {
+  await sendWelcomeEmail(job.data.email);
+});
+```
+
+**Use Cases:**
+- Email sending
+- Image/video processing
+- Report generation
+- Data export
+- Webhook delivery
+
+## CDN (Content Delivery Network)
+
+**Impact:** 50%+ latency reduction for global users
+
+### Configuration
+
+```typescript
+// Cache-Control headers
+res.setHeader('Cache-Control', 'public, max-age=31536000, immutable'); // Static assets
+res.setHeader('Cache-Control', 'public, max-age=3600'); // API responses
+res.setHeader('Cache-Control', 'private, no-cache'); // User-specific data
+```
+
+**CDN Providers:**
+- Cloudflare (generous free tier, global coverage)
+- AWS CloudFront (AWS integration)
+- Fastly (real-time purging)
+
+## Horizontal vs Vertical Scaling
+
+### Horizontal Scaling (Scale Out)
+
+**Pros:**
+- Better fault tolerance
+- Unlimited scaling potential
+- Cost-effective (commodity hardware)
+
+**Cons:**
+- Complex architecture
+- Data consistency challenges
+- Network overhead
+
+**When to use:** High traffic, need redundancy, stateless applications
+
+### Vertical Scaling (Scale Up)
+
+**Pros:**
+- Simple architecture
+- No code changes needed
+- Easier data consistency
+
+**Cons:**
+- Hardware limits
+- Single point of failure
+- Expensive at high end
+
+**When to use:** Monolithic apps, rapid scaling needed, data consistency critical
+
+## Database Scaling Patterns
+
+### Read Replicas
+
+```
+Primary (Write) → Replica 1 (Read)
+               → Replica 2 (Read)
+               → Replica 3 (Read)
+```
+
+**Implementation:**
+```typescript
+// Write to primary
+await primaryDb.users.create(userData);
+
+// Read from replica
+const users = await replicaDb.users.findAll();
+```
+
+**Use Cases:**
+- Read-heavy workloads (90%+ reads)
+- Analytics queries
+- Reporting dashboards
+
+### Database Sharding
+
+**Horizontal Partitioning** - Split data across databases
+
+```typescript
+// Shard by user ID
+function getShardId(userId: string): number {
+  return hashCode(userId) % SHARD_COUNT;
+}
+
+const shardId = getShardId(userId);
+const db = shards[shardId];
+const user = await db.users.findById(userId);
+```
+
+**Sharding Strategies:**
+- **Range-based:** Users 1-1M → Shard 1, 1M-2M → Shard 2
+- **Hash-based:** Hash(userId) % shard_count
+- **Geographic:** EU users → EU shard, US users → US shard
+- **Entity-based:** Users → Shard 1, Orders → Shard 2
+
+## Performance Monitoring
+
+### Key Metrics
+
+**Application:**
+- Response time (p50, p95, p99)
+- Throughput (requests/second)
+- Error rate
+- CPU/memory usage
+
+**Database:**
+- Query execution time
+- Connection pool saturation
+- Cache hit rate
+- Slow query log
+
+**Tools:**
+- Prometheus + Grafana (metrics)
+- New Relic / Datadog (APM)
+- Sentry (error tracking)
+- OpenTelemetry (distributed tracing)
+
+## Performance Optimization Checklist
+
+### Database
+- [ ] Indexes on frequently queried columns
+- [ ] Connection pooling configured
+- [ ] N+1 queries eliminated
+- [ ] Slow query log monitored
+- [ ] Query execution plans analyzed
+
+### Caching
+- [ ] Redis cache for hot data
+- [ ] Cache TTL configured appropriately
+- [ ] Cache invalidation on writes
+- [ ] CDN for static assets
+- [ ] >80% cache hit rate achieved
+
+### Application
+- [ ] Async processing for long tasks
+- [ ] Response compression enabled (gzip)
+- [ ] Load balancing configured
+- [ ] Health checks implemented
+- [ ] Resource limits set (CPU, memory)
+
+### Monitoring
+- [ ] APM tool configured (New Relic/Datadog)
+- [ ] Error tracking (Sentry)
+- [ ] Performance dashboards (Grafana)
+- [ ] Alerting on key metrics
+- [ ] Distributed tracing for microservices
+
+## Common Performance Pitfalls
+
+1. **No caching** - Repeatedly querying same data
+2. **Missing indexes** - Full table scans
+3. **N+1 queries** - Fetching related data in loops
+4. **Synchronous processing** - Blocking on long tasks
+5. **No connection pooling** - Creating new connections per request
+6. **Unbounded queries** - No LIMIT on large tables
+7. **No CDN** - Serving static assets from origin
+
+## Resources
+
+- **PostgreSQL Performance:** https://www.postgresql.org/docs/current/performance-tips.html
+- **Redis Best Practices:** https://redis.io/docs/management/optimization/
+- **Web Performance:** https://web.dev/performance/
+- **Database Indexing:** https://use-the-index-luke.com/