mdcontext 0.1.0 → 0.2.0

package/research/llm-summarization/provider-switching-patterns-2026.md

@@ -0,0 +1,2153 @@
+# LLM Provider Switching and Fallback Patterns - 2026 Research
+
+**Date**: January 26, 2026
+**Author**: Research Team
+**Purpose**: Document production-ready patterns for multi-provider LLM architectures with automatic fallback, cost optimization, and reliability
+
+---
+
+## Executive Summary
+
+This document synthesizes current best practices for building robust multi-provider LLM systems in 2026. Based on analysis of production implementations from Vercel AI SDK, LiteLLM, LangChain, and real-world deployments, the following patterns emerge as essential:
+
+**Key Takeaways:**
+1. **Multi-provider architecture is now standard** - Single-provider dependency is considered a production anti-pattern
+2. **Cost optimization through smart routing** - Teams see 30-50% cost reduction with intelligent provider selection
+3. **Caching delivers 80%+ savings** - Combined prompt caching and semantic caching provide the highest ROI
+4. **Circuit breakers prevent cascading failures** - Essential for production stability
+5. **Provider abstraction layers are mature** - TypeScript solutions with strong type safety available
+
+**Cost Reduction Potential:**
+- Smart routing: 30-50% reduction
+- Prompt caching: 45-80% reduction
+- Semantic caching: 15-30% additional reduction
+- Combined strategies: 80-95% total reduction possible
+
+---
+
+## 1. Multi-Provider Architecture Patterns
+
+### 1.1 Primary + Fallback Strategy
+
+The most common production pattern uses a primary provider with automatic fallback to secondary providers when errors occur.
+
+**Architecture:**
+```typescript
+interface ProviderConfig {
+  name: string;
+  priority: number;
+  models: {
+    cheap: string;    // For simple tasks
+    standard: string; // For typical workloads
+    premium: string;  // For complex reasoning
+  };
+  healthCheck: () => Promise<boolean>;
+  circuitBreaker: CircuitBreakerConfig;
+}
+
+const providerChain: ProviderConfig[] = [
+  {
+    name: 'anthropic',
+    priority: 1,
+    models: {
+      cheap: 'claude-3-haiku-20240307',
+      standard: 'claude-3-5-sonnet-20241022',
+      premium: 'claude-opus-4-5-20251101'
+    },
+    healthCheck: () => checkAnthropicHealth(),
+    circuitBreaker: {
+      failureThreshold: 10,
+      successThreshold: 3,
+      timeout: 60000,
+    }
+  },
+  {
+    name: 'openai',
+    priority: 2,
+    models: {
+      cheap: 'gpt-4o-mini',
+      standard: 'gpt-4o',
+      premium: 'o1'
+    },
+    healthCheck: () => checkOpenAIHealth(),
+    circuitBreaker: {
+      failureThreshold: 10,
+      successThreshold: 3,
+      timeout: 60000,
+    }
+  },
+  {
+    name: 'google',
+    priority: 3,
+    models: {
+      cheap: 'gemini-2.5-flash',
+      standard: 'gemini-2.5-pro',
+      premium: 'gemini-2.5-pro'
+    },
+    healthCheck: () => checkGoogleHealth(),
+    circuitBreaker: {
+      failureThreshold: 10,
+      successThreshold: 3,
+      timeout: 60000,
+    }
+  }
+];
+```
+
+**Key Features:**
+- Ordered fallback chain with priority levels
+- Per-provider circuit breakers to prevent cascading failures
+- Health checks to proactively detect provider issues
+- Quality tiers (cheap/standard/premium) for cost optimization
+
+**Sources:**
+- [Vercel AI Gateway Model Fallbacks](https://vercel.com/changelog/model-fallbacks-now-available-in-vercel-ai-gateway)
+- [LiteLLM Router Architecture](https://docs.litellm.ai/docs/router_architecture)
+
+### 1.2 Quality-Based Routing
+
+Route requests to appropriate model tiers based on task complexity, achieving 30-50% cost reduction.
+
+**Implementation Pattern:**
+```typescript
+interface TaskClassification {
+  complexity: 'simple' | 'standard' | 'complex';
+  estimatedTokens: number;
+  requiresReasoning: boolean;
+  requiresToolUse: boolean;
+}
+
+function selectModel(task: TaskClassification): ModelConfig {
+  // Route 70% of queries to cheap models, escalate only when needed
+  if (task.complexity === 'simple' && !task.requiresReasoning) {
+    return {
+      provider: 'google',
+      model: 'gemini-2.5-flash',
+      costPer1M: { input: 0.30, output: 2.50 }
+    };
+  }
+
+  if (task.complexity === 'standard' || task.requiresToolUse) {
+    return {
+      provider: 'anthropic',
+      model: 'claude-3-5-sonnet-20241022',
+      costPer1M: { input: 3.00, output: 15.00 }
+    };
+  }
+
+  // Complex reasoning tasks
+  return {
+    provider: 'anthropic',
+    model: 'claude-opus-4-5-20251101',
+    costPer1M: { input: 15.00, output: 75.00 }
+  };
+}
+```
+
+**Routing Strategies:**
+1. **Simple Classification** - Use cheap models (GPT-4o-mini, Gemini Flash, Claude Haiku)
+2. **Standard Q&A** - Use mid-tier models (GPT-4o, Claude Sonnet, Gemini Pro)
+3. **Complex Reasoning** - Use premium models (Claude Opus, o1, Gemini Pro with extended context)
+4. **Internal Processing** - Use local open-source models when possible
+
+**Production Impact:**
+- 70% of queries routed to cheap models
+- 25% to standard models
+- 5% to premium models
+- Typical cost reduction: 30-50%
+
+**Sources:**
+- [LLM Cost Optimization Strategies](https://medium.com/@ajayverma23/taming-the-beast-cost-optimization-strategies-for-llm-api-calls-in-production-11f16dbe2c39)
+- [Smart Routing with Load Balancing](https://www.kosmoy.com/post/llm-cost-management-stop-burning-money-on-tokens)
+
+### 1.3 Cost-Optimized Provider Selection
+
+Dynamically route to the cheapest provider for equivalent capability.
+
+**2026 Cost Comparison (per 1M tokens):**
+
+| Provider | Cheap Model | Input | Output | Best For |
+|----------|-------------|-------|--------|----------|
+| **DeepSeek** | deepseek-v3 | $0.27 | $1.10 | Massive cost savings (95% vs OpenAI) |
+| **Google** | gemini-2.5-flash | $0.30 | $2.50 | Speed + cost balance |
+| **OpenAI** | gpt-4o-mini | $0.15 | $0.60 | Simple tasks, fast responses |
+| **Anthropic** | claude-3-haiku | $0.25 | $1.25 | Structured output, tool use |
+| **Mistral** | mistral-small | $0.20 | $0.60 | EU data residency |
+
+| Provider | Standard Model | Input | Output | Best For |
+|----------|---------------|-------|--------|----------|
+| **DeepSeek** | deepseek-r1 | $0.55 | $2.19 | Cost-effective reasoning |
+| **Google** | gemini-2.5-pro | $1.25 | $10.00 | Large context windows |
+| **OpenAI** | gpt-4o | $2.50 | $10.00 | General-purpose, reliable |
+| **Anthropic** | claude-3.5-sonnet | $3.00 | $15.00 | Code generation, analysis |
+| **Mistral** | devstral-2 | $0.62 | $2.46 | Code-specific tasks |
+
+**Dynamic Routing Logic:**
+```typescript
+interface ProviderCost {
+  provider: string;
+  model: string;
+  inputCostPer1M: number;
+  outputCostPer1M: number;
+  estimatedLatencyMs: number;
+}
+
+function selectCheapestProvider(
+  estimatedInputTokens: number,
+  estimatedOutputTokens: number,
+  maxLatency?: number
+): ProviderCost {
+  const options: ProviderCost[] = [
+    {
+      provider: 'deepseek',
+      model: 'deepseek-v3',
+      inputCostPer1M: 0.27,
+      outputCostPer1M: 1.10,
+      estimatedLatencyMs: 2000
+    },
+    {
+      provider: 'google',
+      model: 'gemini-2.5-flash',
+      inputCostPer1M: 0.30,
+      outputCostPer1M: 2.50,
+      estimatedLatencyMs: 800
+    },
+    {
+      provider: 'openai',
+      model: 'gpt-4o-mini',
+      inputCostPer1M: 0.15,
+      outputCostPer1M: 0.60,
+      estimatedLatencyMs: 600
+    }
+  ];
+
+  // Filter by latency requirement
+  const validOptions = maxLatency
+    ? options.filter(o => o.estimatedLatencyMs <= maxLatency)
+    : options;
+
+  // Calculate total cost and select cheapest
+  return validOptions.reduce((cheapest, current) => {
+    const currentCost =
+      (estimatedInputTokens / 1_000_000) * current.inputCostPer1M +
+      (estimatedOutputTokens / 1_000_000) * current.outputCostPer1M;
+
+    const cheapestCost =
+      (estimatedInputTokens / 1_000_000) * cheapest.inputCostPer1M +
+      (estimatedOutputTokens / 1_000_000) * cheapest.outputCostPer1M;
+
+    return currentCost < cheapestCost ? current : cheapest;
+  });
+}
+```
+
+**Key Insight:** Pricing varies 10x across providers for similar capability - smart routing prevents overpaying.
+
+**Sources:**
+- [LLM Cost Optimization 2026](https://byteiota.com/llm-cost-optimization-stop-overpaying-5-10x-in-2026/)
+- [LLM API Pricing Comparison](https://pricepertoken.com/)
+
+---
+
+## 2. Automatic Failover and Error Handling
+
+### 2.1 Circuit Breaker Pattern
+
+Circuit breakers prevent cascading failures by temporarily blocking requests to unhealthy providers.
+
+**States:**
+1. **Closed** - Normal operation, requests pass through
+2. **Open** - Too many failures, requests blocked for timeout period
+3. **Half-Open** - Testing recovery, limited requests allowed
+
+**Implementation:**
+```typescript
+class CircuitBreaker {
+  private state: 'closed' | 'open' | 'half-open' = 'closed';
+  private failures = 0;
+  private successes = 0;
+  private lastFailureTime: number | null = null;
+
+  constructor(
+    private config: {
+      failureThreshold: number;     // e.g., 10 failures
+      successThreshold: number;     // e.g., 3 successes to close
+      timeout: number;              // e.g., 60000ms (60s)
+      jitter: number;               // e.g., 5000ms random variation
+    }
+  ) {}
+
+  async execute<T>(fn: () => Promise<T>): Promise<T> {
+    if (this.state === 'open') {
+      const timeoutWithJitter =
+        this.config.timeout + Math.random() * this.config.jitter;
+
+      if (Date.now() - this.lastFailureTime! < timeoutWithJitter) {
+        throw new Error('Circuit breaker is OPEN');
+      }
+
+      // Transition to half-open for testing
+      this.state = 'half-open';
+      this.successes = 0;
+    }
+
+    try {
+      const result = await fn();
+      this.onSuccess();
+      return result;
+    } catch (error) {
+      this.onFailure();
+      throw error;
+    }
+  }
+
+  private onSuccess(): void {
+    this.failures = 0;
+
+    if (this.state === 'half-open') {
+      this.successes++;
+
+      if (this.successes >= this.config.successThreshold) {
+        this.state = 'closed';
+        console.log('Circuit breaker closed - provider recovered');
+      }
+    }
+  }
+
+  private onFailure(): void {
+    this.failures++;
+    this.lastFailureTime = Date.now();
+
+    if (this.failures >= this.config.failureThreshold) {
+      this.state = 'open';
+      console.log('Circuit breaker opened - provider unhealthy');
+    }
+  }
+
+  getState(): { state: string; failures: number; successes: number } {
+    return {
+      state: this.state,
+      failures: this.failures,
+      successes: this.successes
+    };
+  }
+}
+```
+
+**Benefits:**
+- Prevents overwhelming unhealthy providers
+- Gives systems time to recover
+- Provides clear health signals for monitoring
+- Reduces cascading failures
+
+**Sources:**
+- [Circuit Breakers in LLM Apps](https://portkey.ai/blog/retries-fallbacks-and-circuit-breakers-in-llm-apps/)
+- [Azure API Management LLM Resiliency](https://techcommunity.microsoft.com/blog/azuredevcommunityblog/improve-llm-backend-resiliency-with-load-balancer-and-circuit-breaker-rules-in-a/4394502)
+
+### 2.2 Retry Logic with Exponential Backoff
+
+All major LLM providers recommend exponential backoff with jitter for handling rate limits and transient failures.
+
+**Implementation:**
+```typescript
+interface RetryConfig {
+  maxRetries: number;
+  baseDelay: number;      // e.g., 1000ms
+  maxDelay: number;       // e.g., 60000ms
+  exponentialBase: number; // e.g., 2
+  jitter: boolean;
+}
+
+async function retryWithExponentialBackoff<T>(
+  fn: () => Promise<T>,
+  config: RetryConfig,
+  onRetry?: (attempt: number, delay: number, error: any) => void
+): Promise<T> {
+  let lastError: any;
+
+  for (let attempt = 0; attempt <= config.maxRetries; attempt++) {
+    try {
+      return await fn();
+    } catch (error: any) {
+      lastError = error;
+
+      // Don't retry on non-retryable errors
+      if (!isRetryableError(error)) {
+        throw error;
+      }
+
+      // Check if we've exhausted retries
+      if (attempt === config.maxRetries) {
+        break;
+      }
+
+      // Calculate delay with exponential backoff
+      let delay = Math.min(
+        config.baseDelay * Math.pow(config.exponentialBase, attempt),
+        config.maxDelay
+      );
+
+      // Add jitter to prevent thundering herd
+      if (config.jitter) {
+        delay = delay * (0.5 + Math.random() * 0.5);
+      }
+
+      // Check for retry-after header
+      const retryAfter = parseRetryAfterHeader(error);
+      if (retryAfter) {
+        delay = Math.max(delay, retryAfter * 1000);
+      }
+
+      onRetry?.(attempt + 1, delay, error);
+
+      await sleep(delay);
+    }
+  }
+
+  throw lastError;
+}
+
+function isRetryableError(error: any): boolean {
+  // Rate limiting (429)
+  if (error.status === 429) return true;
+
+  // Server errors (500-599)
+  if (error.status >= 500 && error.status < 600) return true;
+
+  // Network errors
+  if (error.code === 'ECONNRESET' ||
+      error.code === 'ETIMEDOUT' ||
+      error.code === 'ENOTFOUND') return true;
+
+  // Provider-specific timeout errors
+  if (error.message?.includes('timeout')) return true;
+
+  return false;
+}
+
+function parseRetryAfterHeader(error: any): number | null {
+  const retryAfter = error.response?.headers?.['retry-after'];
+  if (!retryAfter) return null;
+
+  // Handle both seconds and date formats
+  const seconds = parseInt(retryAfter, 10);
+  return isNaN(seconds) ? null : seconds;
+}
+
+function sleep(ms: number): Promise<void> {
+  return new Promise(resolve => setTimeout(resolve, ms));
+}
+```
+
+**Provider-Specific Retry Guidelines (2026):**
+
+**OpenAI:**
+- Automatically retries with exponential backoff
+- Respects retry-after headers
+- Recommended: 3-5 retries with base delay 1s
+
+**Anthropic (Claude):**
+- Returns 429 for RPM, ITPM, OTPM limits
+- Provides retry-after header for precise wait times
+- Recommended: Combine retry-after with exponential backoff fallback
+
+**Google (Gemini):**
+- Uses token bucket algorithm
+- 429 quota-exceeded on any dimension
+- Recommended: Exponential backoff with jitter (gold standard)
+
+**Common Pattern:**
+```typescript
+const retryConfig: RetryConfig = {
+  maxRetries: 5,
+  baseDelay: 1000,      // 1s
+  maxDelay: 60000,      // 60s max
+  exponentialBase: 2,   // Double each time: 1s, 2s, 4s, 8s, 16s
+  jitter: true          // Prevent thundering herd
+};
+```
+
+**Sources:**
+- [OpenAI Rate Limits Guide](https://platform.openai.com/docs/guides/rate-limits)
+- [Claude API 429 Error Fix](https://www.aifreeapi.com/en/posts/fix-claude-api-429-rate-limit-error)
+- [Gemini API Rate Limits 2026](https://www.aifreeapi.com/en/posts/gemini-api-rate-limit-explained)
+
+### 2.3 Fallback Chain Implementation
+
+Combine retries with provider fallbacks for maximum reliability.
+
+**Complete Pattern:**
+```typescript
+interface FallbackConfig {
+  providers: ProviderConfig[];
+  retryConfig: RetryConfig;
+  circuitBreakers: Map<string, CircuitBreaker>;
+}
+
+async function executeWithFallback<T>(
+  task: LLMTask,
+  config: FallbackConfig
+): Promise<T> {
+  const errors: Array<{ provider: string; error: any }> = [];
+
+  for (const provider of config.providers) {
+    const breaker = config.circuitBreakers.get(provider.name);
+
+    // Skip if circuit breaker is open
+    if (breaker?.getState().state === 'open') {
+      console.log(`Skipping ${provider.name} - circuit breaker open`);
+      continue;
+    }
+
+    try {
+      // Attempt with retries
+      const result = await breaker!.execute(() =>
+        retryWithExponentialBackoff(
+          () => callProvider(provider, task),
+          config.retryConfig,
+          (attempt, delay, error) => {
+            console.log(
+              `Retry ${attempt} for ${provider.name} after ${delay}ms`,
+              error.message
+            );
+          }
+        )
+      );
+
+      console.log(`Success with ${provider.name}`);
+      return result;
+
+    } catch (error: any) {
+      errors.push({ provider: provider.name, error });
+      console.error(`Failed with ${provider.name}:`, error.message);
+
+      // Continue to next provider in chain
+      continue;
+    }
+  }
+
+  // All providers failed
+  throw new Error(
+    `All providers failed:\n${errors.map(e =>
+      `  ${e.provider}: ${e.error.message}`
+    ).join('\n')}`
+  );
+}
+```
+
+**LiteLLM Router Flow:**
+1. Request sent to `function_with_fallbacks`
+2. Wraps request in try-catch for fallback handling
+3. Passes to `function_with_retries` for retry logic
+4. Calls base LiteLLM unified function
+5. If model is cooling down (rate limited), skip to next
+6. After `num_retries`, fallback to next model group
+7. Fallbacks typically go from one model_name to another
+
+**Benefits:**
+- Maximum reliability across provider outages
+- Automatic recovery from rate limits
+- Graceful degradation under load
+- Clear error trails for debugging
+
+**Sources:**
+- [LiteLLM Reliability Features](https://docs.litellm.ai/docs/completion/reliable_completions)
+- [LangChain Fallbacks](https://python.langchain.com/v0.1/docs/guides/productionization/fallbacks/)
+
+---
+
+## 3. Caching Strategies
+
+Caching provides the highest ROI for cost reduction (80-95% savings possible).
+
+### 3.1 Prompt Caching (Provider-Level)
+
+Provider-native prefix caching delivers 50-90% cost reduction with minimal implementation effort.
+
+**2026 Provider Support:**
+
+| Provider | Feature | Cost Reduction | Latency Improvement | TTL |
+|----------|---------|---------------|-------------------|-----|
+| **Anthropic** | Prompt Caching | Up to 90% | Up to 85% | 5 min |
+| **OpenAI** | Automatic Caching | 50% (cached tokens) | Moderate | Automatic |
+| **Google** | Context Caching | 90% (cached reads) | Significant | Variable |
+
+**Anthropic Implementation:**
+```typescript
+const message = await anthropic.messages.create({
+  model: 'claude-3-5-sonnet-20241022',
+  max_tokens: 1024,
+  system: [
+    {
+      type: 'text',
+      text: 'You are an AI assistant analyzing code repositories.',
+    },
+    {
+      type: 'text',
+      text: largeCodebaseContext, // This gets cached
+      cache_control: { type: 'ephemeral' }
+    }
+  ],
+  messages: [
+    { role: 'user', content: 'Summarize this file: ...' }
+  ]
+});
+
+// Usage breakdown:
+// - Input tokens: 1000
+// - Cache creation: 10000 (write at 1.25x cost)
+// - Cache read: 10000 (read at 0.1x cost = 90% savings)
+// - Output tokens: 500
+```
+
+**OpenAI Implementation:**
+```typescript
+const completion = await openai.chat.completions.create({
+  model: 'gpt-4o',
+  messages: [
+    {
+      role: 'system',
+      content: largeCodebaseContext // Automatically cached if >1024 tokens
+    },
+    {
+      role: 'user',
+      content: 'Summarize this file: ...'
+    }
+  ]
+});
+
+// OpenAI automatically caches prefixes >1024 tokens
+// Cached input tokens: 50% discount
+// No explicit cache control needed
+```
+
+**Best Practices:**
+1. **Place static content first** - System prompts, codebase context, documentation
+2. **Put dynamic content last** - User queries, file-specific questions
+3. **Avoid caching tool definitions** - They change frequently, reducing cache hits
+4. **Strategic block control** - Exclude dynamic sections from cache scope
+
+**Recent Research (2026):**
+- Strategic prompt cache block control provides more consistent benefits than naive full-context caching
+- Reduces API costs by 45-80%
+- Improves time to first token by 13-31%
+- Chat apps with stable system prompts can cache 70%+ of input tokens
+
+**Sources:**
+- [Prompt Caching Infrastructure Guide](https://introl.com/blog/prompt-caching-infrastructure-llm-cost-latency-reduction-guide-2025)
+- [Don't Break the Cache Research](https://arxiv.org/abs/2601.06007)
+
+### 3.2 Semantic Caching
+
+Semantic caching eliminates API calls entirely for semantically similar queries.
+
+**Architecture:**
+```typescript
+interface SemanticCache {
+  // L1: Exact match cache (in-memory)
+  exactCache: Map<string, CachedResponse>;
+
+  // L2: Semantic match cache (vector DB)
+  vectorStore: VectorStore;
+  similarityThreshold: number; // e.g., 0.85
+}
+
+interface CachedResponse {
+  query: string;
+  embedding: number[];
+  response: string;
+  timestamp: number;
+  ttl: number;
+  metadata: {
+    provider: string;
+    model: string;
+    tokens: number;
+    cost: number;
+  };
+}
+
+async function getOrGenerateResponse(
+  query: string,
+  cache: SemanticCache,
+  generateFn: () => Promise<string>
+): Promise<{ response: string; cacheHit: boolean }> {
+  // L1: Check exact match
+  const exactMatch = cache.exactCache.get(query);
+  if (exactMatch && !isExpired(exactMatch)) {
+    return { response: exactMatch.response, cacheHit: true };
+  }
+
+  // L2: Check semantic similarity
+  const queryEmbedding = await generateEmbedding(query);
+  const similarQueries = await cache.vectorStore.similaritySearch(
+    queryEmbedding,
+    cache.similarityThreshold
+  );
+
+  if (similarQueries.length > 0) {
+    const bestMatch = similarQueries[0];
+    console.log(
+      `Semantic cache hit (${bestMatch.similarity.toFixed(3)}) for: "${query}"`
+    );
+    return { response: bestMatch.response, cacheHit: true };
+  }
+
+  // Cache miss - generate new response
+  const response = await generateFn();
+
+  // Store in both caches
+  const cached: CachedResponse = {
+    query,
+    embedding: queryEmbedding,
+    response,
+    timestamp: Date.now(),
+    ttl: 3600000, // 1 hour
+    metadata: {
+      provider: 'anthropic',
+      model: 'claude-3-5-sonnet',
+      tokens: response.length / 4, // Rough estimate
+      cost: calculateCost(response)
+    }
+  };
+
+  cache.exactCache.set(query, cached);
+  await cache.vectorStore.upsert(queryEmbedding, cached);
+
+  return { response, cacheHit: false };
+}
+```
+
+**Multi-Tier Performance:**
+- **Microsecond-level** responses for exact matches (L1)
+- **Sub-second** responses for semantic matches (L2)
+- **Full API latency** only for novel queries
+
+**Production Impact:**
+- 31% of LLM queries exhibit semantic similarity to previous requests
+- Chat apps with repetitive questions: 30% cache hit rate
+- Combined with prompt caching: 80%+ total savings
+
+**Implementation Options:**
+- **GPTCache** - Open-source semantic cache with LangChain integration
+- **Redis** - Semantic caching with vector similarity search
+- **Qdrant** - Vector database optimized for semantic search
+- **LiteLLM** - Built-in caching support
+
+**Sources:**
+- [Redis Semantic Caching](https://redis.io/blog/what-is-semantic-caching/)
+- [GPTCache GitHub](https://github.com/zilliztech/GPTCache)
+- [LLM Caching Implementation Guide](https://reintech.io/blog/how-to-implement-llm-caching-strategies-for-faster-response-times)
+
+### 3.3 Hybrid Caching Strategy
+
+Combine prompt caching and semantic caching for maximum savings.
+
+**Example Configuration:**
+```typescript
+interface HybridCacheConfig {
+  // Semantic cache for query deduplication
+  semanticCache: {
+    enabled: true,
+    provider: 'redis', // or 'qdrant', 'gptcache'
+    similarityThreshold: 0.85,
+    ttl: 3600000, // 1 hour
+  },
+
+  // Prompt cache for large context windows
+  promptCache: {
+    enabled: true,
+    strategy: 'provider-native', // Use Anthropic/OpenAI/Google native caching
+    systemPromptCaching: true,
+    codebaseContextCaching: true,
+    toolDefinitionCaching: false, // Avoid caching dynamic tools
+  },
+
+  // Cache invalidation
+  invalidation: {
+    onCodebaseChange: true,
+    maxAge: 86400000, // 24 hours
+  }
+}
+```
+
+**Combined Savings Example:**
+```
+Scenario: Code summarization service
+- 1000 requests/day
+- Average input: 15,000 tokens (10k context + 5k query)
+- Average output: 500 tokens
+
+Without caching:
+- Cost: 1000 * ((15000/1M * $3) + (500/1M * $15)) = $52.50/day
+
+With prompt caching (70% of input tokens cached):
+- Cost: 1000 * ((4500/1M * $3) + (10500/1M * $0.30) + (500/1M * $15)) = $24/day
+- Savings: 54%
+
+With semantic caching (30% query deduplication):
+- Actual LLM calls: 700
+- Cost: 700 * calculation above = $16.80/day
+- Additional savings: 30%
+
+Total savings: 68%
+
+With both + optimized routing:
+- Use cheaper models for 50% of queries
+- Final cost: ~$8/day
+- Total savings: 85%
+```
+
+**Sources:**
+- [Ultimate Guide to LLM Caching](https://latitude-blog.ghost.io/blog/ultimate-guide-to-llm-caching-for-low-latency-ai/)
+- [Effective LLM Caching](https://www.helicone.ai/blog/effective-llm-caching)
+
+---
+
+## 4. Provider Abstraction Layer
+
+### 4.1 Design Principles
+
+Modern provider abstraction layers follow these principles:
+
+1. **Unified Interface** - Single API for all providers
+2. **Type Safety** - Strong TypeScript types prevent runtime errors
+3. **Provider-Specific Features** - Gracefully handle unique capabilities
+4. **Streaming Support** - First-class support for streaming responses
+5. **Tool/Function Calling** - Normalized tool use across providers
+6. **Observability** - Built-in logging, metrics, and tracing
+
+**Core Interface:**
+```typescript
+interface LLMProvider {
+  name: string;
+
+  // Chat completion
+  complete(params: CompletionParams): Promise<CompletionResponse>;
+
+  // Streaming
+  stream(params: CompletionParams): AsyncIterator<StreamChunk>;
+
+  // Embeddings
+  embed(text: string): Promise<number[]>;
+
+  // Health check
+  healthCheck(): Promise<HealthStatus>;
+
+  // Cost estimation
+  estimateCost(params: CompletionParams): CostEstimate;
+}
+
+interface CompletionParams {
+  model: string;
+  messages: Message[];
+  temperature?: number;
+  maxTokens?: number;
+  tools?: Tool[];
+
+  // Provider-specific options
+  providerOptions?: {
+    anthropic?: {
+      cacheControl?: CacheControl[];
+    };
+    openai?: {
+      responseFormat?: { type: 'json_object' };
+    };
+    google?: {
+      safetySettings?: SafetySetting[];
+    };
+  };
+}
+
+interface CompletionResponse {
+  content: string;
+  role: 'assistant';
+  finishReason: 'stop' | 'length' | 'tool_calls' | 'content_filter';
+  usage: {
+    inputTokens: number;
+    outputTokens: number;
+    cachedTokens?: number;
+  };
+  metadata: {
+    provider: string;
+    model: string;
+    latencyMs: number;
+    cost: number;
+  };
+  toolCalls?: ToolCall[];
+}
+```
+
+**Sources:**
+- [Multi-Provider LLM Orchestration 2026](https://dev.to/ash_dubai/multi-provider-llm-orchestration-in-production-a-2026-guide-1g10)
+- [TypeScript & LLMs: Lessons from Production](https://johnchildseddy.medium.com/typescript-llms-lessons-learned-from-9-months-in-production-4910485e3272)
+
+### 4.2 Production Implementations
+
+**Vercel AI SDK:**
+```typescript
+import { generateText } from 'ai';
+import { anthropic } from '@ai-sdk/anthropic';
+import { openai } from '@ai-sdk/openai';
+import { google } from '@ai-sdk/google';
+
+// Unified interface across providers
+const result = await generateText({
+  model: anthropic('claude-3-5-sonnet-20241022'),
+  messages: [
+    { role: 'user', content: 'Summarize this code...' }
+  ],
+  // Automatic fallback via AI Gateway
+  providerOptions: {
+    fallbacks: [
+      { model: openai('gpt-4o') },
+      { model: google('gemini-2.5-flash') }
+    ]
+  }
+});
+```
+
+**LiteLLM:**
+```typescript
+import litellm from 'litellm';
+
+// Unified OpenAI-compatible interface
+const response = await litellm.completion({
+  model: 'claude-3-5-sonnet-20241022',
+  messages: [{ role: 'user', content: 'Hello' }],
+
+  // Router config for fallbacks
+  fallbacks: ['gpt-4o', 'gemini-2.5-flash'],
+  num_retries: 3,
+  timeout: 30
+});
+
+// Automatic cost tracking
+console.log(response._hidden_params.response_cost);
+```
+
+**Custom TypeScript Abstraction:**
+```typescript
+class UnifiedLLMClient {
+  private providers: Map<string, LLMProvider>;
+  private router: ProviderRouter;
+  private cache: SemanticCache;
+  private metrics: MetricsCollector;
+
+  constructor(config: UnifiedLLMConfig) {
+    this.providers = new Map([
+      ['anthropic', new AnthropicProvider(config.anthropic)],
+      ['openai', new OpenAIProvider(config.openai)],
+      ['google', new GoogleProvider(config.google)]
+    ]);
+
+    this.router = new ProviderRouter(config.routing);
+    this.cache = new SemanticCache(config.caching);
+    this.metrics = new MetricsCollector();
+  }
+
+  async complete(
+    params: CompletionParams,
+    options?: {
+      preferredProvider?: string;
+      enableFallback?: boolean;
+      enableCaching?: boolean;
+    }
+  ): Promise<CompletionResponse> {
+    const startTime = Date.now();
+
+    // Check semantic cache first
+    if (options?.enableCaching) {
+      const cached = await this.cache.get(params.messages);
+      if (cached) {
+        this.metrics.recordCacheHit();
+        return cached;
+      }
+    }
+
+    // Select provider via routing logic
+    const provider = options?.preferredProvider
+      ? this.providers.get(options.preferredProvider)!
+      : await this.router.selectProvider(params);
+
+    try {
+      // Execute with fallback
+      const response = await this.executeWithFallback(
+        provider,
+        params,
+        options?.enableFallback ?? true
+      );
+
+      // Record metrics
+      const latency = Date.now() - startTime;
+      this.metrics.record({
+        provider: response.metadata.provider,
+        latency,
+        tokens: response.usage.inputTokens + response.usage.outputTokens,
+        cost: response.metadata.cost,
+        cached: false
+      });
+
+      // Update cache
+      if (options?.enableCaching) {
+        await this.cache.set(params.messages, response);
+      }
+
+      return response;
+
+    } catch (error) {
+      this.metrics.recordError(provider.name, error);
+      throw error;
+    }
+  }
+
+  private async executeWithFallback(
+    primary: LLMProvider,
+    params: CompletionParams,
+    enableFallback: boolean
+  ): Promise<CompletionResponse> {
+    // Implement fallback chain logic
+    // (See section 2.3 for full implementation)
+  }
+}
+```
+
+**Key Libraries (2026):**
+- **Vercel AI SDK** - Production-ready, TypeScript-first, excellent DX
+- **LiteLLM** - Python-focused, 100+ provider support, proxy mode
+- **AnyLLM** - TypeScript abstraction layer for seamless provider switching
+- **ModelFusion** - Vercel's abstraction for AI model integration
+
+**Sources:**
+- [Vercel AI SDK Docs](https://ai-sdk.dev/)
+- [AnyLLM GitHub](https://github.com/fkesheh/any-llm)
+- [ModelFusion GitHub](https://github.com/vercel/modelfusion)
+
+### 4.3 Handling Provider-Specific Features
+
+Not all providers support all features - graceful degradation is essential.
+
+**Feature Matrix (2026):**
+
+| Feature | Anthropic | OpenAI | Google | Handling Strategy |
+|---------|-----------|--------|--------|-------------------|
+| Tool Calling | ✅ Native | ✅ Native | ✅ Native | Universal support |
+| JSON Mode | ✅ Native | ✅ Native | ✅ Native | Universal support |
+| Vision | ✅ Claude 3+ | ✅ GPT-4o | ✅ Gemini | Universal support |
+| Prompt Caching | ✅ Explicit | ✅ Automatic | ✅ Context cache | Abstract with unified API |
+| Streaming | ✅ SSE | ✅ SSE | ✅ SSE | Universal support |
+| System Messages | ✅ Separate | ✅ In messages | ✅ Separate | Normalize in abstraction |
+| Max Output Tokens | ✅ 8192 | ✅ 16384 | ✅ 8192 | Enforce limits per provider |
+
+**Graceful Degradation Pattern:**
+```typescript
+async function executeWithFeatureDetection(
+  provider: LLMProvider,
+  params: CompletionParams
+): Promise<CompletionResponse> {
+  const capabilities = provider.getCapabilities();
+
+  // Adapt parameters to provider capabilities
+  const adaptedParams = { ...params };
+
+  // Handle tool calling
+  if (params.tools && !capabilities.toolCalling) {
+    console.warn(
+      `Provider ${provider.name} doesn't support tool calling, ` +
+      `using function injection instead`
+    );
+    adaptedParams.messages = injectToolsAsContext(
+      params.messages,
+      params.tools
+    );
+    delete adaptedParams.tools;
+  }
+
+  // Handle JSON mode
+  if (params.responseFormat?.type === 'json_object') {
+    if (capabilities.jsonMode) {
+      // Use native JSON mode
+      adaptedParams.providerOptions = {
+        ...adaptedParams.providerOptions,
+        responseFormat: { type: 'json_object' }
+      };
+    } else {
+      // Fallback to prompt engineering
+      console.warn(
+        `Provider ${provider.name} doesn't support native JSON mode, ` +
+        `using prompt-based approach`
+      );
+      adaptedParams.messages[0].content +=
+        '\n\nIMPORTANT: Respond with valid JSON only.';
+    }
+  }
+
+  // Handle prompt caching
+  if (params.enableCaching && !capabilities.promptCaching) {
+    console.warn(
+      `Provider ${provider.name} doesn't support prompt caching`
+    );
+    // Fall back to semantic caching layer
+  }
+
+  return provider.complete(adaptedParams);
+}
+```
+
+**Sources:**
+- [Building Bridges to LLMs: Moving Beyond Over Abstraction](https://hatchworks.com/blog/gen-ai/llm-projects-production-abstraction/)
+- [AI SDK Provider Management](https://ai-sdk.dev/docs/ai-sdk-core/provider-management)
+
+---
+
+## 5. Monitoring and Observability
+
+### 5.1 Key Metrics to Track
+
+**Essential Metrics:**
+```typescript
+interface LLMMetrics {
+  // Performance
+  latency: {
+    p50: number;
+    p95: number;
+    p99: number;
+  };
+
+  // Cost
+  cost: {
+    total: number;
+    perRequest: number;
+    perProvider: Map<string, number>;
+    perModel: Map<string, number>;
+    perUser?: Map<string, number>;
+  };
+
+  // Usage
+  tokens: {
+    input: number;
+    output: number;
+    cached: number;
+    total: number;
+  };
+
+  // Reliability
+  reliability: {
+    successRate: number;
+    errorRate: number;
+    retryRate: number;
+    fallbackRate: number;
+  };
+
+  // Caching
+  cache: {
+    hitRate: number;
+    semanticHitRate: number;
+    promptCacheHitRate: number;
+    savings: number; // Dollar amount saved
+  };
+
+  // Provider Health
+  providerHealth: Map<string, {
+    availability: number;
+    avgLatency: number;
+    errorRate: number;
+    circuitBreakerState: string;
+  }>;
+}
+```
+
+**Critical Metrics (2026):**
+1. **Tokens per request** - Normalize usage patterns
+2. **Cost per user/team/feature** - Attribution for chargeback
+3. **Cache hit ratio** - Reveal spend savings potential
+4. **Provider availability** - Track SLA compliance
+5. **Error rate by provider** - Identify stability issues
+6. **Latency percentiles** - User experience monitoring
+
+**Sources:**
+- [Langfuse Token and Cost Tracking](https://langfuse.com/docs/observability/features/token-and-cost-tracking)
+- [Tracking LLM Token Usage](https://www.traceloop.com/blog/from-bills-to-budgets-how-to-track-llm-token-usage-and-cost-per-user)
+
+### 5.2 Observability Platforms
+
+**Top Platforms (2026):**
+
+**1. Langfuse (Open Source)**
+```typescript
+import { Langfuse } from 'langfuse';
+
+const langfuse = new Langfuse({
+  publicKey: process.env.LANGFUSE_PUBLIC_KEY,
+  secretKey: process.env.LANGFUSE_SECRET_KEY
+});
+
+const trace = langfuse.trace({
+  name: 'code-summarization',
+  userId: 'user-123',
+  metadata: { repository: 'mdcontext' }
+});
+
+const generation = trace.generation({
+  name: 'summarize-file',
+  model: 'claude-3-5-sonnet-20241022',
+  modelParameters: {
+    temperature: 0.7,
+    maxTokens: 1024
+  },
+  input: messages
+});
+
+// Make LLM call
+const response = await anthropic.messages.create(...);
+
+// Log result
+generation.end({
+  output: response.content,
+  usage: {
+    promptTokens: response.usage.input_tokens,
+    completionTokens: response.usage.output_tokens
+  },
+  metadata: {
+    cost: calculateCost(response.usage),
+    cached: response.usage.cache_read_input_tokens > 0
+  }
+});
+```
+
+**Features:**
+- Automatic cost tracking for 100+ models
+- User/session-level attribution
+- Trace-level debugging
+- Open-source and self-hostable
+
+**2. Datadog LLM Observability**
+```typescript
+import { datadogLLM } from '@datadog/llm-observability';
+
+// Automatic tracing
+const traced = datadogLLM.trace(anthropic.messages.create);
+
+const response = await traced({
+  model: 'claude-3-5-sonnet-20241022',
+  messages: [...]
+});
+
+// Automatic metrics:
+// - llm.request.duration
+// - llm.request.tokens.input
+// - llm.request.tokens.output
+// - llm.request.cost
+```
+
+**Features:**
+- End-to-end AI agent tracing
+- Cloud cost integration
+- Anomaly detection
+- Enterprise-grade dashboards
+
+**3. Traceloop/OpenLLMetry**
+```typescript
+import { Traceloop } from '@traceloop/node-server-sdk';
+
+Traceloop.initialize({
+  apiKey: process.env.TRACELOOP_API_KEY,
+  baseUrl: 'https://api.traceloop.com'
+});
+
+// Automatic instrumentation via OpenTelemetry
+const response = await anthropic.messages.create({
+  model: 'claude-3-5-sonnet-20241022',
+  messages: [...],
+  metadata: {
+    user_id: 'user-123', // Automatic attribution
+    feature: 'code-summarization'
+  }
+});
+```
+
+**Features:**
+- OpenTelemetry-based (industry standard)
+- Automatic user attribution
+- Rich contextual data
+- Integration with existing observability stack
+
+**4. Portkey**
+```typescript
+import Portkey from 'portkey-ai';
+
+const portkey = new Portkey({
+  apiKey: process.env.PORTKEY_API_KEY,
+  virtualKey: process.env.ANTHROPIC_VIRTUAL_KEY
+});
+
+const response = await portkey.chatCompletions.create({
+  model: 'claude-3-5-sonnet-20241022',
+  messages: [...]
+});
+
+// Dashboard shows:
+// - Usage by model, query, token
+// - Cost tracking and forecasts
+// - Near-real-time monitoring
+// - Automatic intervention on limits
+```
+
+**Features:**
+- Finance/leadership dashboards
+- Usage breakdowns and trends
+- Spend forecasting
+- Automatic budget controls
+
+**Sources:**
+- [Langfuse Documentation](https://langfuse.com/docs/observability/features/token-and-cost-tracking)
+- [Datadog LLM Observability](https://www.datadoghq.com/product/llm-observability/)
+- [Traceloop Blog](https://www.traceloop.com/blog/granular-llm-monitoring-for-tracking-token-usage-and-latency-per-user-and-feature)
+- [Portkey Token Tracking](https://portkey.ai/blog/tracking-llm-token-usage-across-providers-teams-and-workloads/)
+
+### 5.3 Monitoring Implementation
+
+**Complete Monitoring Setup:**
+```typescript
+import { Langfuse } from 'langfuse';
+import { CloudWatch } from '@aws-sdk/client-cloudwatch';
+
+class LLMMonitor {
+  private langfuse: Langfuse;
+  private cloudwatch: CloudWatch;
+  private metrics: Map<string, MetricBuffer>;
+
+  constructor() {
+    this.langfuse = new Langfuse({
+      publicKey: process.env.LANGFUSE_PUBLIC_KEY,
+      secretKey: process.env.LANGFUSE_SECRET_KEY
+    });
+
+    this.cloudwatch = new CloudWatch({ region: 'us-east-1' });
+    this.metrics = new Map();
+  }
+
+  async trackRequest(
+    provider: string,
+    model: string,
+    params: CompletionParams,
+    response: CompletionResponse,
+    metadata: {
+      userId?: string;
+      feature?: string;
+      cacheHit?: boolean;
+    }
+  ): Promise<void> {
+    const trace = this.langfuse.trace({
+      name: metadata.feature || 'llm-request',
+      userId: metadata.userId,
+      metadata: {
+        provider,
+        model,
+        cacheHit: metadata.cacheHit || false
+      }
+    });
+
+    const generation = trace.generation({
+      name: 'completion',
+      model,
+      input: params.messages,
+      output: response.content,
+      usage: {
+        promptTokens: response.usage.inputTokens,
+        completionTokens: response.usage.outputTokens,
+        totalTokens:
+          response.usage.inputTokens + response.usage.outputTokens
+      },
+      metadata: {
+        latencyMs: response.metadata.latencyMs,
+        cost: response.metadata.cost,
+        cachedTokens: response.usage.cachedTokens || 0,
+        finishReason: response.finishReason
+      }
+    });
+
+    generation.end();
+
+    // Send metrics to CloudWatch
+    await this.cloudwatch.putMetricData({
+      Namespace: 'LLM/Production',
+      MetricData: [
+        {
+          MetricName: 'Latency',
+          Value: response.metadata.latencyMs,
+          Unit: 'Milliseconds',
+          Dimensions: [
+            { Name: 'Provider', Value: provider },
+            { Name: 'Model', Value: model }
+          ]
+        },
+        {
+          MetricName: 'Cost',
+          Value: response.metadata.cost,
+          Unit: 'None',
+          Dimensions: [
+            { Name: 'Provider', Value: provider },
+            { Name: 'Feature', Value: metadata.feature || 'unknown' }
+          ]
+        },
+        {
+          MetricName: 'TokensUsed',
+          Value: response.usage.inputTokens + response.usage.outputTokens,
+          Unit: 'Count',
+          Dimensions: [
+            { Name: 'Provider', Value: provider }
+          ]
+        },
+        {
+          MetricName: 'CacheHit',
+          Value: metadata.cacheHit ? 1 : 0,
+          Unit: 'Count'
+        }
+      ]
+    });
+  }
+
+  async trackError(
+    provider: string,
+    model: string,
+    error: any,
+    metadata: {
+      retryAttempt?: number;
+      fallbackTriggered?: boolean;
+    }
+  ): Promise<void> {
+    // Log to Langfuse
+    this.langfuse.trace({
+      name: 'llm-error',
+      metadata: {
+        provider,
+        model,
+        error: error.message,
+        statusCode: error.status,
+        retryAttempt: metadata.retryAttempt,
+        fallbackTriggered: metadata.fallbackTriggered
+      }
+    });
+
+    // Send error metrics
+    await this.cloudwatch.putMetricData({
+      Namespace: 'LLM/Production',
+      MetricData: [
+        {
+          MetricName: 'Errors',
+          Value: 1,
+          Unit: 'Count',
+          Dimensions: [
+            { Name: 'Provider', Value: provider },
+            { Name: 'ErrorType', Value: error.status || 'unknown' }
+          ]
+        }
+      ]
+    });
+  }
+
+  async getProviderHealth(): Promise<Map<string, ProviderHealth>> {
+    // Query CloudWatch metrics
+    const endTime = new Date();
+    const startTime = new Date(endTime.getTime() - 3600000); // Last hour
+
+    const healthMap = new Map<string, ProviderHealth>();
+
+    for (const provider of ['anthropic', 'openai', 'google']) {
+      const errorRate = await this.getErrorRate(provider, startTime, endTime);
+      const avgLatency = await this.getAvgLatency(provider, startTime, endTime);
+
+      healthMap.set(provider, {
+        availability: 1 - errorRate,
+        avgLatency,
+        errorRate,
+        status: errorRate > 0.05 ? 'degraded' : 'healthy'
+      });
+    }
+
+    return healthMap;
+  }
+}
+```
+
+**Dashboard Recommendations:**
+1. **Real-time costs** by provider, model, user, feature
+2. **Latency percentiles** (p50, p95, p99) per provider
+3. **Cache hit rates** for cost optimization tracking
+4. **Error rates and circuit breaker states** for reliability
+5. **Token usage trends** for capacity planning
+6. **Provider health scores** for SLA monitoring
+
+**Sources:**
+- [LLM Observability Tools 2026](https://lakefs.io/blog/llm-observability-tools/)
+- [Best LLM Observability Tools](https://www.firecrawl.dev/blog/best-llm-observability-tools)
+
+---
+
+## 6. Configuration Best Practices
+
+### 6.1 Environment-Based Configuration
+
+**Production Configuration Pattern:**
+```typescript
+interface LLMConfig {
+  providers: {
+    anthropic: {
+      apiKey: string;
+      baseURL?: string;
+      timeout: number;
+      maxRetries: number;
+    };
+    openai: {
+      apiKey: string;
+      organization?: string;
+      timeout: number;
+      maxRetries: number;
+    };
+    google: {
+      apiKey: string;
+      timeout: number;
+      maxRetries: number;
+    };
+  };
+
+  routing: {
+    strategy: 'cost' | 'quality' | 'latency' | 'custom';
+    fallbackEnabled: boolean;
+    circuitBreaker: {
+      enabled: boolean;
+      failureThreshold: number;
+      successThreshold: number;
+      timeout: number;
+    };
+  };
+
+  caching: {
+    semantic: {
+      enabled: boolean;
+      provider: 'redis' | 'qdrant' | 'memory';
+      similarityThreshold: number;
+      ttl: number;
+    };
+    prompt: {
+      enabled: boolean;
+      strategy: 'provider-native' | 'custom';
+    };
+  };
+
+  observability: {
+    enabled: boolean;
+    provider: 'langfuse' | 'datadog' | 'custom';
+    sampleRate: number; // 0.0 - 1.0
+    logLevel: 'debug' | 'info' | 'warn' | 'error';
+  };
+
+  limits: {
+    maxTokensPerRequest: number;
+    maxCostPerRequest: number;
+    rateLimitPerUser?: number;
+  };
+}
+```
+
+**Environment-Specific Configs:**
+```typescript
+// config/production.ts
+export const productionConfig: LLMConfig = {
+  providers: {
+    anthropic: {
+      apiKey: process.env.ANTHROPIC_API_KEY!,
+      timeout: 30000,
+      maxRetries: 3
+    },
+    openai: {
+      apiKey: process.env.OPENAI_API_KEY!,
+      organization: process.env.OPENAI_ORG,
+      timeout: 30000,
+      maxRetries: 3
+    },
+    google: {
+      apiKey: process.env.GOOGLE_API_KEY!,
+      timeout: 30000,
+      maxRetries: 3
+    }
+  },
+
+  routing: {
+    strategy: 'cost',
+    fallbackEnabled: true,
+    circuitBreaker: {
+      enabled: true,
+      failureThreshold: 10,
+      successThreshold: 3,
+      timeout: 60000
+    }
+  },
+
+  caching: {
+    semantic: {
+      enabled: true,
+      provider: 'redis',
+      similarityThreshold: 0.85,
+      ttl: 3600000 // 1 hour
+    },
+    prompt: {
+      enabled: true,
+      strategy: 'provider-native'
+    }
+  },
+
+  observability: {
+    enabled: true,
+    provider: 'langfuse',
+    sampleRate: 1.0, // Log everything in prod
+    logLevel: 'info'
+  },
+
+  limits: {
+    maxTokensPerRequest: 100000,
+    maxCostPerRequest: 1.00, // $1 max per request
+    rateLimitPerUser: 100 // per hour
+  }
+};
+
+// config/development.ts
+export const developmentConfig: LLMConfig = {
+  ...productionConfig,
+
+  routing: {
+    ...productionConfig.routing,
+    strategy: 'latency' // Prefer fast responses in dev
+  },
+
+  caching: {
+    semantic: {
+      enabled: false, // Disable caching in dev
+      provider: 'memory',
+      similarityThreshold: 0.85,
+      ttl: 600000
+    },
+    prompt: {
+      enabled: false
+    }
+  },
+
+  observability: {
+    enabled: true,
+    provider: 'langfuse',
+    sampleRate: 0.1, // Sample 10% in dev
+    logLevel: 'debug'
+  }
+};
+```
+
+### 6.2 Feature Flags for Gradual Rollout
+
+```typescript
+interface FeatureFlags {
+  enableSemanticCache: boolean;
+  enablePromptCache: boolean;
+  enableMultiProvider: boolean;
+  enableCircuitBreaker: boolean;
+  newProviders: string[]; // e.g., ['deepseek', 'mistral']
+  costOptimizationLevel: 'conservative' | 'balanced' | 'aggressive';
+}
+
+// Use LaunchDarkly, Flagsmith, or similar
+const flags = await featureFlagClient.getFlags('llm-system');
+
+if (flags.enableSemanticCache) {
+  // Enable semantic caching
+}
+
+if (flags.newProviders.includes('deepseek')) {
+  // Add DeepSeek to provider chain
+  providerChain.push({
+    name: 'deepseek',
+    priority: 4,
+    models: {
+      cheap: 'deepseek-v3',
+      standard: 'deepseek-r1',
+      premium: 'deepseek-r1'
+    }
+  });
+}
+```
+
+### 6.3 Model Selection Matrix
+
+**Production Model Selection Guide:**
+
+| Use Case | Recommended Model | Provider | Cost (1M tokens) | Notes |
+|----------|------------------|----------|------------------|-------|
+| **Simple Classification** | gpt-4o-mini | OpenAI | $0.15/$0.60 | Fastest, cheapest |
+| **Standard Q&A** | gemini-2.5-flash | Google | $0.30/$2.50 | Best value |
+| **Code Generation** | claude-3-5-sonnet | Anthropic | $3.00/$15.00 | Industry leading |
+| **Complex Reasoning** | claude-opus-4.5 | Anthropic | $15.00/$75.00 | Best quality |
+| **Cost-Optimized** | deepseek-v3 | DeepSeek | $0.27/$1.10 | 95% cheaper |
+| **Large Context** | gemini-2.5-pro | Google | $1.25/$10.00 | 1M tokens |
+| **Local Deployment** | qwen-2.5-coder-32b | Local | $0 | Self-hosted |
+
+**Sources:**
+- [LLM Pricing 2026](https://research.aimultiple.com/llm-pricing/)
+- [Alternative Providers Research](./alternative-providers-2026.md)
+
+---
+
+## 7. Implementation Roadmap for mdcontext
+
+### 7.1 Phase 1: Basic Multi-Provider Support (Week 1)
+
+**Goals:**
+- Add support for Anthropic, OpenAI, Google providers
+- Implement basic fallback chain
+- Add configuration system
+
+**Tasks:**
+1. Create provider abstraction layer
+2. Implement unified interface
+3. Add environment-based configuration
+4. Basic error handling with retries
+
+**Code Changes:**
+```typescript
+// src/llm/providers/base.ts
+export interface LLMProvider {
+  complete(params: CompletionParams): Promise<CompletionResponse>;
+  healthCheck(): Promise<boolean>;
+}
+
+// src/llm/providers/anthropic.ts
+export class AnthropicProvider implements LLMProvider {
+  async complete(params: CompletionParams): Promise<CompletionResponse> {
+    // Implementation
+  }
+}
+
+// src/llm/router.ts
+export class ProviderRouter {
+  async complete(params: CompletionParams): Promise<CompletionResponse> {
+    // Try primary provider with fallback
+  }
+}
+```
+
+### 7.2 Phase 2: Cost Optimization (Week 2)
+
+**Goals:**
+- Implement prompt caching
+- Add cost-based routing
+- Track token usage and costs
+
+**Tasks:**
+1. Enable Anthropic prompt caching for codebase context
+2. Implement cost estimation per provider
+3. Add Langfuse for cost tracking
+4. Create cost optimization routing logic
+
+**Expected Savings:**
+- 50-70% cost reduction from prompt caching
+- 20-30% additional from smart routing
+
+### 7.3 Phase 3: Reliability (Week 3)
+
+**Goals:**
+- Add circuit breakers
+- Implement exponential backoff
+- Health monitoring
+
+**Tasks:**
+1. Implement circuit breaker per provider
+2. Add exponential backoff with jitter
+3. Health check endpoints
+4. Provider health dashboard
+
+### 7.4 Phase 4: Advanced Caching (Week 4)
+
+**Goals:**
+- Semantic caching layer
+- Combined caching strategy
+- Cache invalidation logic
+
+**Tasks:**
+1. Set up Redis/Qdrant for semantic cache
+2. Implement similarity search
+3. Add cache hit rate tracking
+4. Configure TTL and invalidation
+
+**Expected Savings:**
+- Additional 20-30% from semantic caching
+- Total: 70-90% cost reduction
+
+### 7.5 Phase 5: Observability (Ongoing)
+
+**Goals:**
+- Comprehensive monitoring
+- Cost attribution
+- Performance tracking
+
+**Tasks:**
+1. Langfuse integration complete
+2. CloudWatch metrics
+3. Cost per feature/user tracking
+4. Alerting on budget thresholds
+
+---
+
+## 8. Testing Strategy
+
+### 8.1 Provider Resilience Testing
+
+```typescript
+describe('Provider Fallback', () => {
+  it('should fallback to secondary provider on primary failure', async () => {
+    // Mock primary provider failure
+    mockAnthropicProvider.complete.mockRejectedValue(
+      new Error('Rate limit exceeded')
+    );
+
+    mockOpenAIProvider.complete.mockResolvedValue({
+      content: 'Success',
+      usage: { inputTokens: 100, outputTokens: 50 }
+    });
+
+    const result = await router.complete({
+      messages: [{ role: 'user', content: 'Test' }]
+    });
+
+    expect(result.metadata.provider).toBe('openai');
+    expect(mockAnthropicProvider.complete).toHaveBeenCalledTimes(1);
+    expect(mockOpenAIProvider.complete).toHaveBeenCalledTimes(1);
+  });
+
+  it('should respect circuit breaker state', async () => {
+    // Trigger circuit breaker
+    for (let i = 0; i < 10; i++) {
+      try {
+        await router.complete({ messages: [...] });
+      } catch {}
+    }
+
+    const health = circuitBreaker.getState();
+    expect(health.state).toBe('open');
+
+    // Should skip provider with open circuit
+    const result = await router.complete({ messages: [...] });
+    expect(result.metadata.provider).not.toBe('anthropic');
+  });
+});
+```
+
+### 8.2 Cost Validation Testing
+
+```typescript
+describe('Cost Optimization', () => {
+  it('should route simple tasks to cheap models', () => {
+    const model = selectModel({
+      complexity: 'simple',
+      estimatedTokens: 1000,
+      requiresReasoning: false,
+      requiresToolUse: false
+    });
+
+    expect(model.provider).toBe('google');
+    expect(model.model).toBe('gemini-2.5-flash');
+  });
+
+  it('should use prompt caching for large contexts', async () => {
+    const params = {
+      messages: [
+        { role: 'system', content: largeCodebaseContext },
+        { role: 'user', content: 'Summarize this file' }
+      ],
+      enableCaching: true
+    };
+
+    const result = await provider.complete(params);
+
+    expect(result.usage.cachedTokens).toBeGreaterThan(0);
+    expect(result.metadata.cost).toBeLessThan(
+      calculateCostWithoutCaching(result.usage)
+    );
+  });
+});
+```
+
+### 8.3 Load Testing
+
+```bash
+# Artillery config for load testing
+artillery run tests/load/provider-fallback.yml
+```
+
+```yaml
+# tests/load/provider-fallback.yml
+config:
+  target: 'http://localhost:3000'
+  phases:
+    - duration: 60
+      arrivalRate: 10
+      name: 'Warm up'
+    - duration: 120
+      arrivalRate: 50
+      name: 'Ramp up'
+    - duration: 300
+      arrivalRate: 100
+      name: 'Sustained load'
+
+scenarios:
+  - name: 'Code Summarization'
+    flow:
+      - post:
+          url: '/api/summarize'
+          json:
+            code: '{{ $randomString() }}'
+            model: 'claude-3-5-sonnet-20241022'
+          capture:
+            - json: '$.cost'
+              as: 'requestCost'
+            - json: '$.provider'
+              as: 'usedProvider'
+```
+
+---
+
+## 9. Key Recommendations for mdcontext
+
+### 9.1 Immediate Actions (This Week)
+
+1. **Add Multi-Provider Support**
+   - Implement Anthropic, OpenAI, Google providers
+   - Basic fallback chain: Claude Sonnet → GPT-4o → Gemini Pro
+   - Environment-based configuration
+
+2. **Enable Prompt Caching**
+   - Use Anthropic's cache_control for codebase context
+   - Expected: 50-70% cost reduction immediately
+
+3. **Add Basic Monitoring**
+   - Integrate Langfuse for cost tracking
+   - Track provider success/failure rates
+
+### 9.2 Short-Term (Next 2 Weeks)
+
+1. **Implement Circuit Breakers**
+   - Prevent cascading failures
+   - Health monitoring per provider
+
+2. **Cost-Based Routing**
+   - Route simple tasks to Gemini Flash
+   - Complex reasoning to Claude Sonnet
+   - Expected: 30-50% additional savings
+
+3. **Retry Logic**
+   - Exponential backoff with jitter
+   - Respect retry-after headers
+
+### 9.3 Medium-Term (Next Month)
+
+1. **Semantic Caching**
+   - Set up Redis for cache storage
+   - Implement similarity search
+   - Expected: 20-30% additional savings
+
+2. **Advanced Routing**
+   - Task complexity classification
+   - Dynamic provider selection based on:
+     - Current cost
+     - Latency requirements
+     - Provider health
+
+3. **Complete Observability**
+   - Cost per feature attribution
+   - User-level tracking
+   - Budget alerts
+
+### 9.4 Architecture Recommendation
+
+```
+┌─────────────────────────────────────────────────────────┐
+│                     Client Request                       │
+└───────────────────────────┬─────────────────────────────┘
+                            │
+                            ▼
+┌─────────────────────────────────────────────────────────┐
+│                  Semantic Cache Layer                    │
+│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │
+│  │ Exact Match  │  │  Similarity  │  │  TTL Check   │  │
+│  │  (Memory)    │  │   (Redis)    │  │              │  │
+│  └──────────────┘  └──────────────┘  └──────────────┘  │
+└───────────────────────────┬─────────────────────────────┘
+                            │ Cache Miss
+                            ▼
+┌─────────────────────────────────────────────────────────┐
+│                    Provider Router                       │
+│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │
+│  │   Classify   │  │    Route     │  │   Estimate   │  │
+│  │     Task     │  │   Provider   │  │     Cost     │  │
+│  └──────────────┘  └──────────────┘  └──────────────┘  │
+└───────────────────────────┬─────────────────────────────┘
+                            │
+                            ▼
+┌─────────────────────────────────────────────────────────┐
+│              Provider Chain with Fallback                │
+│                                                           │
+│  ┌─────────────────────────────────────────────────┐   │
+│  │ Primary: Anthropic Claude                        │   │
+│  │ ┌──────────────┐  ┌──────────────┐             │   │
+│  │ │Circuit Breaker│ │Retry w/Backoff│             │   │
+│  │ └──────────────┘  └──────────────┘             │   │
+│  │ Prompt Caching: ✅                              │   │
+│  └─────────────────────────────────────────────────┘   │
+│                            │ Fails                      │
+│                            ▼                            │
+│  ┌─────────────────────────────────────────────────┐   │
+│  │ Secondary: OpenAI GPT-4o                         │   │
+│  │ ┌──────────────┐  ┌──────────────┐             │   │
+│  │ │Circuit Breaker│ │Retry w/Backoff│             │   │
+│  │ └──────────────┘  └──────────────┘             │   │
+│  │ Prompt Caching: ✅ (Automatic)                  │   │
+│  └─────────────────────────────────────────────────┘   │
+│                            │ Fails                      │
+│                            ▼                            │
+│  ┌─────────────────────────────────────────────────┐   │
+│  │ Tertiary: Google Gemini                          │   │
+│  │ ┌──────────────┐  ┌──────────────┐             │   │
+│  │ │Circuit Breaker│ │Retry w/Backoff│             │   │
+│  │ └──────────────┘  └──────────────┘             │   │
+│  │ Context Caching: ✅                             │   │
+│  └─────────────────────────────────────────────────┘   │
+└───────────────────────────┬─────────────────────────────┘
+                            │
+                            ▼
+┌─────────────────────────────────────────────────────────┐
+│                 Observability Layer                      │
+│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │
+│  │   Langfuse   │  │  CloudWatch  │  │    Alerts    │  │
+│  │ Cost Tracking│  │   Metrics    │  │   Budgets    │  │
+│  └──────────────┘  └──────────────┘  └──────────────┘  │
+└─────────────────────────────────────────────────────────┘
+```
+
+### 9.5 Expected Outcomes
+
+**Cost Reduction:**
+- Phase 1 (Prompt Caching): 50-70%
+- Phase 2 (Smart Routing): 20-30%
+- Phase 3 (Semantic Caching): 10-20%
+- **Total: 70-90% reduction**
+
+**Reliability Improvement:**
+- 99.9%+ uptime with multi-provider fallback
+- Circuit breakers prevent cascading failures
+- Automatic recovery from rate limits
+
+**Performance:**
+- <100ms for cache hits (semantic)
+- <1s for cached prompts
+- Minimal latency overhead from routing layer
+
+---
+
+## 10. Additional Resources
+
+### 10.1 Documentation
+
+- [Vercel AI SDK Documentation](https://ai-sdk.dev/)
+- [LiteLLM Documentation](https://docs.litellm.ai/)
+- [Anthropic Prompt Caching Guide](https://docs.anthropic.com/claude/docs/prompt-caching)
+- [OpenAI Rate Limits Guide](https://platform.openai.com/docs/guides/rate-limits)
+- [Google Gemini API Documentation](https://ai.google.dev/docs)
+
+### 10.2 Tools and Libraries
+
+**Provider SDKs:**
+- [@anthropic-ai/sdk](https://www.npmjs.com/package/@anthropic-ai/sdk)
+- [openai](https://www.npmjs.com/package/openai)
+- [@google/generative-ai](https://www.npmjs.com/package/@google/generative-ai)
+
+**Abstraction Layers:**
+- [ai (Vercel AI SDK)](https://www.npmjs.com/package/ai)
+- [litellm](https://pypi.org/project/litellm/)
+- [any-llm](https://www.npmjs.com/package/any-llm)
+
+**Observability:**
+- [langfuse](https://www.npmjs.com/package/langfuse)
+- [@datadog/llm-observability](https://docs.datadoghq.com/llm_observability/)
+- [@traceloop/node-server-sdk](https://www.npmjs.com/package/@traceloop/node-server-sdk)
+
+**Caching:**
+- [redis](https://www.npmjs.com/package/redis)
+- [qdrant-js](https://www.npmjs.com/package/@qdrant/js-client-rest)
+- [gptcache](https://github.com/zilliztech/GPTCache)
+
+### 10.3 Related Research Documents
+
+- [Alternative LLM Providers 2026](./alternative-providers-2026.md) - Comprehensive provider comparison
+- [Anthropic 2026](./anthropic-2026.md) - Claude-specific features and pricing
+- [OpenAI 2026](./openai-2026.md) - GPT models and capabilities
+- [Prompt Engineering 2026](./prompt-engineering-2026.md) - Optimization techniques
+
+---
+
+## Sources
+
+This research synthesizes information from the following sources:
+
+### Multi-Provider Architecture
+- [Vercel AI Gateway Model Fallbacks](https://vercel.com/changelog/model-fallbacks-now-available-in-vercel-ai-gateway)
+- [Vercel AI SDK Provider Management](https://ai-sdk.dev/docs/ai-sdk-core/provider-management)
+- [LiteLLM Router Architecture](https://docs.litellm.ai/docs/router_architecture)
+- [LiteLLM Fallbacks & Reliability](https://docs.litellm.ai/docs/proxy/reliability)
+- [LangChain Fallbacks](https://python.langchain.com/v0.1/docs/guides/productionization/fallbacks/)
+- [Dynamic Failover with LangChain](https://medium.com/@andrewnguonly/dynamic-failover-and-load-balancing-llms-with-langchain-e930a094be61)
+
+### Cost Optimization
+- [LLM Cost Optimization Strategies](https://medium.com/@ajayverma23/taming-the-beast-cost-optimization-strategies-for-llm-api-calls-in-production-11f16dbe2c39)
+- [Stop Overpaying 5-10x in 2026](https://byteiota.com/llm-cost-optimization-stop-overpaying-5-10x-in-2026/)
+- [Smart Routing for Cost Management](https://www.kosmoy.com/post/llm-cost-management-stop-burning-money-on-tokens)
+- [Monitor and Optimize LLM Costs](https://www.helicone.ai/blog/monitor-and-optimize-llm-costs)
+- [LLM API Pricing 2026](https://pricepertoken.com/)
+
+### Caching Strategies
+- [Redis Semantic Caching](https://redis.io/blog/what-is-semantic-caching/)
+- [GPTCache GitHub](https://github.com/zilliztech/GPTCache)
+- [Prompt Caching: 10x Cheaper Tokens](https://ngrok.com/blog/prompt-caching/)
+- [Prompt Caching Infrastructure Guide](https://introl.com/blog/prompt-caching-infrastructure-llm-cost-latency-reduction-guide-2025)
+- [Don't Break the Cache Research](https://arxiv.org/abs/2601.06007)
+- [LiteLLM Caching](https://docs.litellm.ai/docs/proxy/caching)
+- [Ultimate Guide to LLM Caching](https://latitude-blog.ghost.io/blog/ultimate-guide-to-llm-caching-for-low-latency-ai/)
+- [Effective LLM Caching](https://www.helicone.ai/blog/effective-llm-caching)
+
+### Circuit Breakers & Reliability
+- [Circuit Breakers in LLM Apps](https://portkey.ai/blog/retries-fallbacks-and-circuit-breakers-in-llm-apps/)
+- [Azure API Management LLM Resiliency](https://techcommunity.microsoft.com/blog/azuredevcommunityblog/improve-llm-backend-resiliency-with-load-balancer-and-circuit-breaker-rules-in-a/4394502)
+- [Apigee Circuit Breaker Pattern](https://github.com/GoogleCloudPlatform/apigee-samples/blob/main/llm-circuit-breaking/llm_circuit_breaking.ipynb)
+
+### Rate Limiting & Retry Logic
+- [OpenAI Rate Limits](https://platform.openai.com/docs/guides/rate-limits)
+- [OpenAI Exponential Backoff](https://platform.openai.com/docs/guides/rate-limits/retrying-with-exponential-backoff)
+- [Claude API 429 Error Fix](https://www.aifreeapi.com/en/posts/fix-claude-api-429-rate-limit-error)
+- [Gemini API Rate Limits 2026](https://www.aifreeapi.com/en/posts/gemini-api-rate-limit-explained)
+
+### Provider Abstraction
+- [Multi-Provider LLM Orchestration 2026](https://dev.to/ash_dubai/multi-provider-llm-orchestration-in-production-a-2026-guide-1g10)
+- [AnyLLM GitHub](https://github.com/fkesheh/any-llm)
+- [TypeScript & LLMs: Production Lessons](https://johnchildseddy.medium.com/typescript-llms-lessons-learned-from-9-months-in-production-4910485e3272)
+- [ModelFusion GitHub](https://github.com/vercel/modelfusion)
+- [Building Bridges to LLMs](https://hatchworks.com/blog/gen-ai/llm-projects-production-abstraction/)
+
+### Observability
+- [Langfuse Token and Cost Tracking](https://langfuse.com/docs/observability/features/token-and-cost-tracking)
+- [Datadog LLM Observability](https://www.datadoghq.com/product/llm-observability/)
+- [Traceloop: Track Token Usage Per User](https://www.traceloop.com/blog/from-bills-to-budgets-how-to-track-llm-token-usage-and-cost-per-user)
+- [Portkey Token Tracking](https://portkey.ai/blog/tracking-llm-token-usage-across-providers-teams-and-workloads/)
+- [Elastic LLM Observability](https://www.elastic.co/observability/llm-monitoring)
+- [LLM Observability Tools 2026](https://lakefs.io/blog/llm-observability-tools/)
+- [Best LLM Observability Tools](https://www.firecrawl.dev/blog/best-llm-observability-tools)
+
+---
+
+**Document Version**: 1.0
+**Last Updated**: January 26, 2026
+**Next Review**: February 26, 2026