qa360 2.1.7 → 2.2.0

package/docs/llm-providers.md

@@ -0,0 +1,512 @@
+# LLM Providers - QA360 AI Architecture
+
+QA360 supports multiple LLM providers with automatic fallback. This ensures your AI-powered test generation always works, even if one provider is unavailable.
+
+## Overview
+
+The AI module uses a **fallback chain** to automatically select the best available provider:
+
+```
+DeepSeek → Ollama (local) → OpenAI → Anthropic → Mock (testing)
+```
+
+### Provider Priority
+
+| Priority | Provider | Type | Cost | Latency | Notes |
+|----------|----------|------|------|----------|-------|
+| 1st | **DeepSeek** | Cloud | $ | Low | GPT-4 level at 95% lower cost |
+| 2nd | **Ollama** | Local | Free | Medium | Privacy, offline, no API cost |
+| 3rd | OpenAI | Cloud | $$$ | Low | GPT-4, reliable |
+| 4th | Anthropic | Cloud | $$$ | Low | Claude models |
+| 5th | Mock | Local | Free | Instant | For testing only |
+
+> **DeepSeek** is the recommended default provider for production use. It offers GPT-4 level performance at $0.14/M input tokens (vs ~$30/M for GPT-4).
+
+## Quick Start
+
+### Automatic Provider Selection
+
+```typescript
+import { createLLMProvider } from '@qa360/core';
+
+// Automatically selects the first available provider
+const provider = await createLLMProvider();
+
+// Generate code
+const response = await provider.generate({
+  prompt: 'Write a test for login API',
+  systemPrompt: 'You are a test generation expert',
+  maxTokens: 2000
+});
+
+console.log(response.content);
+```
+
+### Force a Specific Provider
+
+```typescript
+import { createLLMProvider } from '@qa360/core';
+
+// Prefer OpenAI, fall back to others if unavailable
+const provider = await createLLMProvider({
+  preferred: 'openai'
+});
+```
+
+### Skip a Provider
+
+```typescript
+import { createLLMProvider } from '@qa360/core';
+
+// Skip Ollama, go directly to cloud providers
+const provider = await createLLMProvider({
+  skip: ['ollama']
+});
+```
+
+## Provider Configuration
+
+### DeepSeek (Cloud - Recommended)
+
+DeepSeek offers GPT-4 level performance at a fraction of the cost. It's the recommended provider for production use.
+
+**Environment Variables:**
+```bash
+export DEEPSEEK_API_KEY="sk-..."  # Required
+export DEEPSEEK_BASE_URL="https://api.deepseek.com"  # Optional
+export DEEPSEEK_MODEL="deepseek-chat"  # Optional
+```
+
+**Pricing (as of 2025):**
+- Input: $0.14 per million tokens
+- Output: $0.28 per million tokens
+- **~99.5% cheaper than GPT-4**
+
+**Supported Models:**
+- `deepseek-chat` - General purpose, GPT-4 level
+- `deepseek-coder` - Specialized for code tasks
+- `deepseek-reasoner` - Enhanced reasoning capabilities
+
+**Setup:**
+```bash
+# Get API key
+# Visit: https://platform.deepseek.com/api_keys
+
+# Set environment variable
+export DEEPSEEK_API_KEY="your_api_key_here"
+
+# Or use QA360 secrets
+qa360 secrets add DEEPSEEK_API_KEY
+```
+
+**Code:**
+```typescript
+import { createDeepSeekProvider } from '@qa360/core';
+
+const provider = createDeepSeekProvider({
+  apiKey: 'sk-...',
+  model: 'deepseek-chat'
+});
+
+const available = await provider.isAvailable();
+console.log('DeepSeek available:', available);
+
+// Estimate cost
+const cost = provider.estimateCost(1000, 500);
+console.log(`Estimated cost: $${cost.toFixed(6)}`);
+```
+
+### Ollama (Local - Free)
+
+Ollama is the preferred provider for local-first, free AI inference.
+
+**Environment Variables:**
+```bash
+export OLLAMA_BASE_URL="http://localhost:11434"  # Optional
+export OLLAMA_MODEL="deepseek-coder"              # Optional
+```
+
+**Setup:**
+```bash
+# Install Ollama
+curl -fsSL https://ollama.com/install.sh | sh
+
+# Pull a model
+ollama pull deepseek-coder
+
+# Start Ollama server
+ollama serve
+```
+
+**Code:**
+```typescript
+import { createOllamaProvider } from '@qa360/core';
+
+const provider = createOllamaProvider({
+  baseUrl: 'http://localhost:11434',
+  model: 'deepseek-coder'
+});
+
+const available = await provider.isAvailable();
+console.log('Ollama available:', available);
+```
+
+### OpenAI (Cloud)
+
+OpenAI provides GPT-4 and other powerful models via API.
+
+**Environment Variables:**
+```bash
+export OPENAI_API_KEY="sk-..."  # Required
+export OPENAI_BASE_URL="https://api.openai.com/v1"  # Optional
+export OPENAI_MODEL="gpt-4"  # Optional
+```
+
+**Supported Models:**
+- `gpt-4` - Most capable
+- `gpt-4-turbo` - Faster, cheaper
+- `gpt-4o` - Latest GPT-4 Omni
+- `gpt-4o-mini` - Smaller, faster
+- `gpt-3.5-turbo` - Budget option
+
+**Code:**
+```typescript
+import { createOpenAIProvider } from '@qa360/core';
+
+const provider = createOpenAIProvider({
+  apiKey: 'sk-...',
+  model: 'gpt-4'
+});
+```
+
+### Anthropic (Cloud)
+
+Anthropic provides Claude models with strong coding capabilities.
+
+**Environment Variables:**
+```bash
+export ANTHROPIC_API_KEY="sk-ant-..."  # Required
+export ANTHROPIC_BASE_URL="https://api.anthropic.com"  # Optional
+export ANTHROPIC_MODEL="claude-3-5-sonnet-20241022"  # Optional
+```
+
+**Supported Models:**
+- `claude-3-5-sonnet-20241022` - Best for code
+- `claude-3-5-sonnet-20240620` - Previous Sonnet
+- `claude-3-opus-20240229` - Most capable
+- `claude-3-haiku-20240307` - Fastest
+
+**Code:**
+```typescript
+import { createAnthropicProvider } from '@qa360/core';
+
+const provider = createAnthropicProvider({
+  apiKey: 'sk-ant-...',
+  model: 'claude-3-5-sonnet-20241022'
+});
+```
+
+### Mock (Testing)
+
+The Mock provider returns predefined responses without making network calls. Useful for testing and development.
+
+```typescript
+import { createMockProvider } from '@qa360/core';
+
+const provider = createMockProvider({
+  delay: 50,  // Simulate latency in ms
+  responses: {
+    'test': 'Mock response for test'
+  }
+});
+```
+
+## Advanced Usage
+
+### Check Provider Availability
+
+```typescript
+import { getProviderStatus, detectBestProvider } from '@qa360/core';
+
+// Check all providers
+const status = await getProviderStatus();
+console.log(status);
+// { deepseek: true, ollama: false, openai: false, anthropic: false, mock: true }
+
+// Detect best available
+const best = await detectBestProvider();
+console.log('Best provider:', best);
+// 'deepseek'
+```
+
+### Custom Fallback Chain
+
+```typescript
+import { createLLMProvider } from '@qa360/core';
+
+// Custom order: DeepSeek → Ollama → Mock (skip cloud providers)
+const provider = await createLLMProvider({
+  chain: ['deepseek', 'ollama', 'mock']
+});
+```
+
+### Streaming Responses
+
+```typescript
+const provider = await createLLMProvider();
+
+// Stream tokens as they arrive
+for await (const chunk of provider.stream({
+  prompt: 'Write a test function',
+  maxTokens: 500
+})) {
+  process.stdout.write(chunk);
+}
+```
+
+### Count Tokens
+
+```typescript
+const provider = await createLLMProvider();
+
+const text = 'Your prompt here...';
+const tokens = provider.countTokens(text);
+
+console.log(`Approximate tokens: ${tokens}`);
+```
+
+## Error Handling
+
+All providers throw specific errors that you can catch:
+
+```typescript
+import { DeepSeekError, OpenAIError, AnthropicError, OllamaError } from '@qa360/core';
+
+try {
+  const response = await provider.generate({ prompt: 'test' });
+} catch (error) {
+  if (error instanceof DeepSeekError) {
+    console.error('DeepSeek error:', error.message);
+    console.error('Details:', error.details);
+  } else if (error instanceof OpenAIError) {
+    console.error('OpenAI error:', error.message);
+    console.error('Details:', error.details);
+  } else if (error instanceof OllamaError) {
+    console.error('Ollama error:', error.message);
+  } else if (error instanceof AnthropicError) {
+    console.error('Anthropic error:', error.message);
+  } else {
+    console.error('Unknown error:', error);
+  }
+}
+```
+
+## CLI Integration
+
+The AI module is integrated into QA360 CLI commands:
+
+```bash
+# List all AI providers and their status
+qa360 ai list
+
+# Test a specific provider
+qa360 ai use deepseek --verify
+qa360 ai use ollama
+
+# Generate tests using AI
+qa360 ai generate "Write tests for login API" --type api
+
+# Benchmark all providers
+qa360 ai benchmark
+
+# Show AI configuration
+qa360 ai config
+```
+
+## Best Practices
+
+### 1. Use DeepSeek for Production
+
+DeepSeek offers the best value for production use:
+- GPT-4 level performance at 95% lower cost
+- Reliable API with excellent uptime
+- Specialized code model (`deepseek-coder`)
+
+### 2. Use Ollama for Local Development
+
+Ollama provides free, local inference. It's ideal for:
+- Development environments
+- Offline work
+- Sensitive data (no external API calls)
+
+### 3. Set Fallback API Keys for Production
+
+Configure cloud API keys as fallback:
+```bash
+# .env - Priority order
+DEEPSEEK_API_KEY=sk-...  # Primary (best value)
+OPENAI_API_KEY=sk-...    # Fallback
+ANTHROPIC_API_KEY=sk-ant-...  # Fallback
+```
+
+### 4. Use Mock for Testing
+
+The Mock provider ensures deterministic tests:
+```typescript
+const mockProvider = createMockProvider({
+  responses: {
+    'generate login test': 'it("should login", () => { ... })'
+  }
+});
+```
+
+### 4. Handle Provider Failures Gracefully
+
+```typescript
+const provider = await createLLMProvider();
+const available = await provider.isAvailable();
+
+if (!available) {
+  console.warn('Primary provider unavailable, using fallback');
+}
+```
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────┐
+│                   QA360 AI Module                        │
+├─────────────────────────────────────────────────────────┤
+│                                                         │
+│  ┌──────────────┐       ┌──────────────┐              │
+│  │   Provider   │       │   Provider   │              │
+│  │   Factory    │───────│   Fallback   │              │
+│  └──────────────┘       └──────────────┘              │
+│         │                       │                       │
+│         ▼                       ▼                       │
+│  ┌──────────────┐       ┌──────────────┐              │
+│  │  DeepSeek    │───────│   Ollama     │              │
+│  │  Provider    │       │  Provider    │              │
+│  └──────────────┘       └──────────────┘              │
+│         │                       │                       │
+│         ▼                       ▼                       │
+│  ┌──────────────┐       ┌──────────────┐              │
+│  │   OpenAI     │───────│  Anthropic   │              │
+│  │  Provider    │       │  Provider    │              │
+│  └──────────────┘       └──────────────┘              │
+│         │                       │                       │
+│         ▼                       ▼                       │
+│  ┌──────────────────────────────────────────┐         │
+│  │            Mock Provider                  │         │
+│  └──────────────────────────────────────────┘         │
+│                                                         │
+└─────────────────────────────────────────────────────────┘
+```
+
+## Reference
+
+### LLMProvider Interface
+
+```typescript
+interface LLMProvider {
+  name: string;                    // Provider name
+  models: string[];                // Supported models
+
+  generate(request: GenerationRequest): Promise<GenerationResponse>;
+  stream?(request: GenerationRequest): AsyncIterable<string>;
+  countTokens(text: string): number;
+  isAvailable(): Promise<boolean>;
+}
+
+interface GenerationRequest {
+  prompt: string;
+  systemPrompt?: string;
+  maxTokens?: number;
+  temperature?: number;
+}
+
+interface GenerationResponse {
+  content: string;
+  usage: {
+    promptTokens: number;
+    completionTokens: number;
+    totalTokens: number;
+  };
+  model: string;
+  finishReason: 'stop' | 'length' | 'content_filter';
+}
+```
+
+### ProviderFactoryConfig
+
+```typescript
+interface ProviderFactoryConfig {
+  preferred?: 'deepseek' | 'ollama' | 'openai' | 'anthropic' | 'mock';
+  chain?: Array<'deepseek' | 'ollama' | 'openai' | 'anthropic' | 'mock'>;
+  skip?: Array<'deepseek' | 'ollama' | 'openai' | 'anthropic' | 'mock'>;
+
+  deepseek?: { apiKey?: string; baseURL?: string; model?: string; timeout?: number };
+  ollama?: { baseUrl?: string; model?: string; timeout?: number };
+  openai?: { apiKey?: string; baseURL?: string; model?: string; timeout?: number };
+  anthropic?: { apiKey?: string; baseURL?: string; model?: string; timeout?: number };
+  mock?: { delay?: number; responses?: Record<string, string> };
+}
+```
+
+## Troubleshooting
+
+### DeepSeek Not Available
+
+```bash
+# Verify API key
+echo $DEEPSEEK_API_KEY
+
+# Test API key
+curl https://api.deepseek.com/v1/models \
+  -H "Authorization: Bearer $DEEPSEEK_API_KEY"
+
+# Set API key
+export DEEPSEEK_API_KEY="your_api_key_here"
+```
+
+### Ollama Not Available
+
+```bash
+# Check Ollama is running
+curl http://localhost:11434/api/tags
+
+# Start Ollama
+ollama serve
+
+# Pull a model
+ollama pull deepseek-coder
+```
+
+### OpenAI/Anthropic API Errors
+
+```bash
+# Verify API key
+echo $OPENAI_API_KEY
+
+# Test API key
+curl https://api.openai.com/v1/models \
+  -H "Authorization: Bearer $OPENAI_API_KEY"
+```
+
+### Provider Selection Issues
+
+```typescript
+// Debug provider selection
+import { getProviderStatus } from '@qa360/core';
+
+const status = await getProviderStatus();
+console.table(status);
+```
+
+## See Also
+
+- [DeepSeek Platform](https://platform.deepseek.com)
+- [DeepSeek API Docs](https://platform.deepseek.com/docs)
+- [Ollama Documentation](https://ollama.com/docs)
+- [OpenAI API Reference](https://platform.openai.com/docs/api-reference)
+- [Anthropic API Reference](https://docs.anthropic.com/claude/reference)