proagents 1.0.0

package/proagents/examples/backend-nodejs/complete-conversation.md

@@ -0,0 +1,601 @@
+# Complete Conversation Example: Node.js API
+
+An example of building a backend feature using ProAgents workflow.
+
+---
+
+## Starting Point
+
+**User has:** Express + TypeScript API with Prisma ORM and JWT auth.
+
+**Goal:** Add a rate limiting system with per-user quotas.
+
+---
+
+## The Conversation
+
+### User Initiates
+
+```
+User: Add rate limiting to our API. I want different limits for different
+user tiers (free, pro, enterprise) and per-endpoint limits for expensive
+operations.
+```
+
+### AI Response: Analysis & Understanding
+
+```
+AI: I'll help you implement a tiered rate limiting system. Let me analyze
+your codebase first.
+
+[Running Analysis...]
+
+──────────────────────────────────────────────────────────
+Codebase Analysis Complete
+──────────────────────────────────────────────────────────
+
+Tech Stack:
+• Node.js 20.x + TypeScript 5.2
+• Express 4.18
+• Prisma ORM 5.x (PostgreSQL)
+• JWT authentication
+• Redis connected (for caching)
+
+Middleware Structure:
+src/middleware/
+├── auth.ts           # JWT validation, sets req.user
+├── validate.ts       # Zod schema validation
+├── errorHandler.ts   # Global error handler
+└── cors.ts           # CORS configuration
+
+Existing User Model:
+model User {
+  id        String   @id @default(uuid())
+  email     String   @unique
+  tier      UserTier @default(FREE)
+  // ... other fields
+}
+
+enum UserTier {
+  FREE
+  PRO
+  ENTERPRISE
+}
+
+I see you already have:
+• User tier in the database
+• Redis available for rate limit storage
+• Auth middleware that sets req.user
+
+This is a good foundation for tiered rate limiting. Let me gather the
+specific requirements.
+```
+
+### AI: Requirements Clarification
+
+```
+AI: I have some questions to define the rate limits:
+
+1. What are the default rate limits per tier?
+
+   My suggestion based on common patterns:
+   ┌────────────────────────────────────────────────────┐
+   │ Tier        │ Requests/min │ Requests/day │ Burst  │
+   ├────────────────────────────────────────────────────┤
+   │ FREE        │ 60           │ 1,000        │ 10     │
+   │ PRO         │ 300          │ 10,000       │ 50     │
+   │ ENTERPRISE  │ 1,000        │ 100,000      │ 200    │
+   └────────────────────────────────────────────────────┘
+
+2. Which endpoints need special (lower) limits?
+
+   Common expensive operations:
+   [ ] File uploads
+   [ ] Report generation
+   [ ] Bulk operations
+   [ ] Search with complex queries
+   [ ] Export endpoints
+
+3. What should happen when rate limited?
+   [ ] Return 429 with retry-after header (standard)
+   [ ] Return 429 with upgrade prompt
+   [ ] Gradually degrade (slower responses)
+
+4. Should rate limits reset at midnight or rolling window?
+   [ ] Rolling window (recommended)
+   [ ] Fixed daily reset
+```
+
+### User Provides Details
+
+```
+User:
+1. Your suggestions look good, use those numbers
+2. File uploads, report generation, and bulk operations need lower limits
+3. Return 429 with retry-after and include upgrade info for free users
+4. Rolling window
+```
+
+### AI: Implementation Plan
+
+```
+AI: Got it. Here's the implementation plan:
+
+──────────────────────────────────────────────────────────
+Implementation Plan: Rate Limiting System
+──────────────────────────────────────────────────────────
+
+Architecture:
+┌─────────────────────────────────────────────────────────┐
+│                    Request Flow                         │
+├─────────────────────────────────────────────────────────┤
+│                                                         │
+│  Request → Auth Middleware → Rate Limiter → Route       │
+│                    │                                    │
+│                    ↓                                    │
+│              ┌─────────────┐                           │
+│              │   Redis     │                           │
+│              │ (Counters)  │                           │
+│              └─────────────┘                           │
+│                                                         │
+└─────────────────────────────────────────────────────────┘
+
+Files to Create:
+
+src/middleware/
+├── rateLimit/
+│   ├── index.ts              # Main middleware export
+│   ├── rateLimiter.ts        # Core rate limiting logic
+│   ├── tierLimits.ts         # Tier configuration
+│   ├── endpointLimits.ts     # Per-endpoint overrides
+│   └── types.ts              # TypeScript interfaces
+
+src/services/
+└── rateLimitStore.ts         # Redis operations
+
+src/utils/
+└── rateLimitResponse.ts      # 429 response formatting
+
+Files to Modify:
+
+• src/app.ts                  # Apply global rate limiter
+• src/routes/files.ts         # Add stricter limits
+• src/routes/reports.ts       # Add stricter limits
+• src/routes/bulk.ts          # Add stricter limits
+• src/types/express.d.ts      # Extend Request type
+
+Database Changes:
+
+• Add RateLimitOverride table (for custom limits per user)
+• Migration: 20240120_add_rate_limit_overrides
+
+Shall I proceed with implementation?
+```
+
+### User Confirms
+
+```
+User: Yes, proceed.
+```
+
+### AI: Implementation
+
+```
+AI: Starting implementation...
+
+✓ Created types and configuration
+──────────────────────────────────────────────────────────
+// src/middleware/rateLimit/tierLimits.ts
+
+import { UserTier } from '@prisma/client';
+
+export interface TierLimit {
+  requestsPerMinute: number;
+  requestsPerDay: number;
+  burstLimit: number;
+}
+
+export const TIER_LIMITS: Record<UserTier, TierLimit> = {
+  FREE: {
+    requestsPerMinute: 60,
+    requestsPerDay: 1_000,
+    burstLimit: 10,
+  },
+  PRO: {
+    requestsPerMinute: 300,
+    requestsPerDay: 10_000,
+    burstLimit: 50,
+  },
+  ENTERPRISE: {
+    requestsPerMinute: 1_000,
+    requestsPerDay: 100_000,
+    burstLimit: 200,
+  },
+};
+
+// Expensive endpoint multipliers (lower = stricter)
+export const ENDPOINT_MULTIPLIERS: Record<string, number> = {
+  'POST:/api/files/upload': 0.1,      // 10% of normal limit
+  'POST:/api/reports/generate': 0.05,  // 5% of normal limit
+  'POST:/api/bulk/*': 0.02,            // 2% of normal limit
+};
+──────────────────────────────────────────────────────────
+
+✓ Created Redis store
+──────────────────────────────────────────────────────────
+// src/services/rateLimitStore.ts
+
+import { redis } from './redis';
+
+export class RateLimitStore {
+  private prefix = 'ratelimit:';
+
+  async increment(
+    userId: string,
+    window: 'minute' | 'day',
+    endpoint?: string
+  ): Promise<{ count: number; ttl: number }> {
+    const key = this.getKey(userId, window, endpoint);
+    const ttl = window === 'minute' ? 60 : 86400;
+
+    const multi = redis.multi();
+    multi.incr(key);
+    multi.expire(key, ttl);
+    multi.ttl(key);
+
+    const [count, , remainingTtl] = await multi.exec();
+
+    return {
+      count: count as number,
+      ttl: remainingTtl as number,
+    };
+  }
+
+  async getCount(
+    userId: string,
+    window: 'minute' | 'day',
+    endpoint?: string
+  ): Promise<number> {
+    const key = this.getKey(userId, window, endpoint);
+    const count = await redis.get(key);
+    return parseInt(count || '0', 10);
+  }
+
+  private getKey(
+    userId: string,
+    window: string,
+    endpoint?: string
+  ): string {
+    const base = `${this.prefix}${userId}:${window}`;
+    return endpoint ? `${base}:${endpoint}` : base;
+  }
+}
+
+export const rateLimitStore = new RateLimitStore();
+──────────────────────────────────────────────────────────
+
+✓ Created main rate limiter middleware
+──────────────────────────────────────────────────────────
+// src/middleware/rateLimit/rateLimiter.ts
+
+import { Request, Response, NextFunction } from 'express';
+import { rateLimitStore } from '@/services/rateLimitStore';
+import { TIER_LIMITS, ENDPOINT_MULTIPLIERS } from './tierLimits';
+import { RateLimitError } from '@/errors';
+
+interface RateLimitOptions {
+  windowMs?: number;
+  endpointKey?: string;
+}
+
+export function createRateLimiter(options: RateLimitOptions = {}) {
+  return async (req: Request, res: Response, next: NextFunction) => {
+    // Skip if no authenticated user (handled by auth)
+    if (!req.user) {
+      return next();
+    }
+
+    const userId = req.user.id;
+    const userTier = req.user.tier;
+    const limits = TIER_LIMITS[userTier];
+
+    // Get endpoint-specific multiplier
+    const endpointKey = options.endpointKey ||
+      `${req.method}:${req.baseUrl}${req.path}`;
+    const multiplier = getMultiplier(endpointKey);
+
+    // Calculate effective limits
+    const effectiveMinuteLimit = Math.floor(
+      limits.requestsPerMinute * multiplier
+    );
+    const effectiveDayLimit = Math.floor(
+      limits.requestsPerDay * multiplier
+    );
+
+    // Check minute window
+    const minuteResult = await rateLimitStore.increment(
+      userId,
+      'minute',
+      multiplier < 1 ? endpointKey : undefined
+    );
+
+    if (minuteResult.count > effectiveMinuteLimit) {
+      return sendRateLimitResponse(res, {
+        retryAfter: minuteResult.ttl,
+        limit: effectiveMinuteLimit,
+        window: 'minute',
+        userTier,
+      });
+    }
+
+    // Check daily window
+    const dayResult = await rateLimitStore.increment(userId, 'day');
+
+    if (dayResult.count > effectiveDayLimit) {
+      return sendRateLimitResponse(res, {
+        retryAfter: dayResult.ttl,
+        limit: effectiveDayLimit,
+        window: 'day',
+        userTier,
+      });
+    }
+
+    // Add rate limit headers
+    res.setHeader('X-RateLimit-Limit', effectiveMinuteLimit);
+    res.setHeader('X-RateLimit-Remaining',
+      Math.max(0, effectiveMinuteLimit - minuteResult.count)
+    );
+    res.setHeader('X-RateLimit-Reset',
+      Math.floor(Date.now() / 1000) + minuteResult.ttl
+    );
+
+    next();
+  };
+}
+
+function getMultiplier(endpointKey: string): number {
+  // Check exact match first
+  if (ENDPOINT_MULTIPLIERS[endpointKey]) {
+    return ENDPOINT_MULTIPLIERS[endpointKey];
+  }
+
+  // Check wildcard patterns
+  for (const [pattern, mult] of Object.entries(ENDPOINT_MULTIPLIERS)) {
+    if (pattern.endsWith('/*')) {
+      const prefix = pattern.slice(0, -1);
+      if (endpointKey.startsWith(prefix)) {
+        return mult;
+      }
+    }
+  }
+
+  return 1;
+}
+
+function sendRateLimitResponse(
+  res: Response,
+  info: {
+    retryAfter: number;
+    limit: number;
+    window: string;
+    userTier: string;
+  }
+) {
+  res.setHeader('Retry-After', info.retryAfter);
+
+  const response: any = {
+    error: 'Too Many Requests',
+    message: `Rate limit exceeded. Try again in ${info.retryAfter} seconds.`,
+    retryAfter: info.retryAfter,
+  };
+
+  // Add upgrade prompt for free users
+  if (info.userTier === 'FREE') {
+    response.upgrade = {
+      message: 'Upgrade to Pro for higher rate limits',
+      link: '/pricing',
+      proLimit: TIER_LIMITS.PRO.requestsPerMinute,
+    };
+  }
+
+  return res.status(429).json(response);
+}
+
+export const rateLimit = createRateLimiter();
+──────────────────────────────────────────────────────────
+
+✓ Applied to app.ts
+──────────────────────────────────────────────────────────
+// In src/app.ts - After auth middleware
+
+import { rateLimit } from '@/middleware/rateLimit';
+
+// Apply global rate limiter
+app.use(rateLimit);
+──────────────────────────────────────────────────────────
+
+✓ Added stricter limits to expensive routes
+──────────────────────────────────────────────────────────
+// In src/routes/files.ts
+
+import { createRateLimiter } from '@/middleware/rateLimit';
+
+const uploadLimiter = createRateLimiter({
+  endpointKey: 'POST:/api/files/upload',
+});
+
+router.post('/upload', uploadLimiter, async (req, res) => {
+  // ... upload logic
+});
+──────────────────────────────────────────────────────────
+
+✓ Created database migration
+──────────────────────────────────────────────────────────
+-- Migration: 20240120_add_rate_limit_overrides
+
+CREATE TABLE rate_limit_overrides (
+  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+  requests_per_minute INT,
+  requests_per_day INT,
+  endpoint_pattern VARCHAR(255),
+  created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
+  expires_at TIMESTAMP WITH TIME ZONE,
+  reason TEXT,
+  UNIQUE(user_id, endpoint_pattern)
+);
+
+CREATE INDEX idx_rate_limit_overrides_user
+  ON rate_limit_overrides(user_id);
+──────────────────────────────────────────────────────────
+
+Implementation complete. Running tests...
+```
+
+### AI: Testing
+
+```
+AI: Running tests...
+
+──────────────────────────────────────────────────────────
+Test Results
+──────────────────────────────────────────────────────────
+
+Unit Tests:
+✓ rateLimitStore.increment increments counter (15ms)
+✓ rateLimitStore.increment sets TTL correctly (12ms)
+✓ createRateLimiter allows requests under limit (28ms)
+✓ createRateLimiter blocks requests over minute limit (32ms)
+✓ createRateLimiter blocks requests over daily limit (35ms)
+✓ createRateLimiter applies endpoint multipliers (25ms)
+✓ createRateLimiter sets correct headers (22ms)
+✓ FREE tier gets upgrade prompt in 429 response (30ms)
+✓ PRO tier does not get upgrade prompt (28ms)
+
+Integration Tests:
+✓ Rate limit persists across requests (145ms)
+✓ Rate limit resets after window (2015ms)
+✓ Different users have separate limits (95ms)
+✓ Expensive endpoints have lower limits (88ms)
+
+Load Tests:
+✓ Handles 1000 concurrent rate checks (234ms)
+
+Coverage: 96%
+
+All 14 tests passed.
+```
+
+### AI: Summary
+
+```
+AI: Rate limiting implementation complete!
+
+──────────────────────────────────────────────────────────
+Summary: Rate Limiting System
+──────────────────────────────────────────────────────────
+
+Features Implemented:
+✓ Tiered rate limits (FREE/PRO/ENTERPRISE)
+✓ Per-minute and per-day windows
+✓ Rolling window implementation
+✓ Per-endpoint overrides for expensive operations
+✓ Standard rate limit headers
+✓ 429 responses with retry-after
+✓ Upgrade prompts for free users
+✓ Redis-based distributed counting
+✓ Custom override support via database
+
+Rate Limits:
+┌────────────────────────────────────────────────────┐
+│ Tier        │ /min  │ /day    │ Upload │ Report  │
+├────────────────────────────────────────────────────┤
+│ FREE        │ 60    │ 1,000   │ 6      │ 3       │
+│ PRO         │ 300   │ 10,000  │ 30     │ 15      │
+│ ENTERPRISE  │ 1,000 │ 100,000 │ 100    │ 50      │
+└────────────────────────────────────────────────────┘
+
+Files Created: 6
+Files Modified: 4
+Test Coverage: 96%
+
+Ready to commit?
+```
+
+### User Commits
+
+```
+User: Yes, commit it.
+```
+
+### AI: Commit
+
+```
+AI: Committing changes...
+
+$ git add .
+$ git commit -m "feat(api): add tiered rate limiting system
+
+- Add rate limiting middleware with Redis storage
+- Implement FREE/PRO/ENTERPRISE tier limits
+- Add per-endpoint multipliers for expensive operations
+- Include rate limit headers on all responses
+- Return 429 with retry-after and upgrade prompts
+- Add rate limit override table for custom limits
+- Add comprehensive tests (96% coverage)
+
+Rate limits:
+- FREE: 60/min, 1000/day
+- PRO: 300/min, 10000/day
+- ENTERPRISE: 1000/min, 100000/day
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
+
+Committed: abc123f
+Branch: feature/rate-limiting
+```
+
+---
+
+## API Documentation Generated
+
+```
+AI: I've also generated API documentation for the rate limiting:
+
+──────────────────────────────────────────────────────────
+Rate Limiting Documentation
+──────────────────────────────────────────────────────────
+
+All API requests include rate limit headers:
+
+Response Headers:
+• X-RateLimit-Limit: Maximum requests per minute
+• X-RateLimit-Remaining: Requests remaining in window
+• X-RateLimit-Reset: Unix timestamp when window resets
+
+429 Response:
+{
+  "error": "Too Many Requests",
+  "message": "Rate limit exceeded. Try again in 45 seconds.",
+  "retryAfter": 45,
+  "upgrade": {
+    "message": "Upgrade to Pro for higher rate limits",
+    "link": "/pricing",
+    "proLimit": 300
+  }
+}
+
+For custom rate limits, contact support.
+──────────────────────────────────────────────────────────
+```
+
+---
+
+## Key Backend Patterns Shown
+
+1. **Middleware composition** - Rate limiter as Express middleware
+2. **Redis integration** - For distributed rate limit counting
+3. **Configuration-driven** - Tier limits in separate config file
+4. **Header standards** - Proper rate limit headers
+5. **Database flexibility** - Override table for exceptions
+6. **Testability** - High test coverage with unit and integration tests