qa360 2.2.1 → 2.2.13

package/docs/llm-providers.md

@@ -1,512 +0,0 @@
-# LLM Providers - QA360 AI Architecture
-
-QA360 supports multiple LLM providers with automatic fallback. This ensures your AI-powered test generation always works, even if one provider is unavailable.
-
-## Overview
-
-The AI module uses a **fallback chain** to automatically select the best available provider:
-
-```
-DeepSeek → Ollama (local) → OpenAI → Anthropic → Mock (testing)
-```
-
-### Provider Priority
-
-| Priority | Provider | Type | Cost | Latency | Notes |
-|----------|----------|------|------|----------|-------|
-| 1st | **DeepSeek** | Cloud | $ | Low | GPT-4 level at 95% lower cost |
-| 2nd | **Ollama** | Local | Free | Medium | Privacy, offline, no API cost |
-| 3rd | OpenAI | Cloud | $$$ | Low | GPT-4, reliable |
-| 4th | Anthropic | Cloud | $$$ | Low | Claude models |
-| 5th | Mock | Local | Free | Instant | For testing only |
-
-> **DeepSeek** is the recommended default provider for production use. It offers GPT-4 level performance at $0.14/M input tokens (vs ~$30/M for GPT-4).
-
-## Quick Start
-
-### Automatic Provider Selection
-
-```typescript
-import { createLLMProvider } from '@qa360/core';
-
-// Automatically selects the first available provider
-const provider = await createLLMProvider();
-
-// Generate code
-const response = await provider.generate({
-  prompt: 'Write a test for login API',
-  systemPrompt: 'You are a test generation expert',
-  maxTokens: 2000
-});
-
-console.log(response.content);
-```
-
-### Force a Specific Provider
-
-```typescript
-import { createLLMProvider } from '@qa360/core';
-
-// Prefer OpenAI, fall back to others if unavailable
-const provider = await createLLMProvider({
-  preferred: 'openai'
-});
-```
-
-### Skip a Provider
-
-```typescript
-import { createLLMProvider } from '@qa360/core';
-
-// Skip Ollama, go directly to cloud providers
-const provider = await createLLMProvider({
-  skip: ['ollama']
-});
-```
-
-## Provider Configuration
-
-### DeepSeek (Cloud - Recommended)
-
-DeepSeek offers GPT-4 level performance at a fraction of the cost. It's the recommended provider for production use.
-
-**Environment Variables:**
-```bash
-export DEEPSEEK_API_KEY="sk-..."  # Required
-export DEEPSEEK_BASE_URL="https://api.deepseek.com"  # Optional
-export DEEPSEEK_MODEL="deepseek-chat"  # Optional
-```
-
-**Pricing (as of 2025):**
-- Input: $0.14 per million tokens
-- Output: $0.28 per million tokens
-- **~99.5% cheaper than GPT-4**
-
-**Supported Models:**
-- `deepseek-chat` - General purpose, GPT-4 level
-- `deepseek-coder` - Specialized for code tasks
-- `deepseek-reasoner` - Enhanced reasoning capabilities
-
-**Setup:**
-```bash
-# Get API key
-# Visit: https://platform.deepseek.com/api_keys
-
-# Set environment variable
-export DEEPSEEK_API_KEY="your_api_key_here"
-
-# Or use QA360 secrets
-qa360 secrets add DEEPSEEK_API_KEY
-```
-
-**Code:**
-```typescript
-import { createDeepSeekProvider } from '@qa360/core';
-
-const provider = createDeepSeekProvider({
-  apiKey: 'sk-...',
-  model: 'deepseek-chat'
-});
-
-const available = await provider.isAvailable();
-console.log('DeepSeek available:', available);
-
-// Estimate cost
-const cost = provider.estimateCost(1000, 500);
-console.log(`Estimated cost: $${cost.toFixed(6)}`);
-```
-
-### Ollama (Local - Free)
-
-Ollama is the preferred provider for local-first, free AI inference.
-
-**Environment Variables:**
-```bash
-export OLLAMA_BASE_URL="http://localhost:11434"  # Optional
-export OLLAMA_MODEL="deepseek-coder"              # Optional
-```
-
-**Setup:**
-```bash
-# Install Ollama
-curl -fsSL https://ollama.com/install.sh | sh
-
-# Pull a model
-ollama pull deepseek-coder
-
-# Start Ollama server
-ollama serve
-```
-
-**Code:**
-```typescript
-import { createOllamaProvider } from '@qa360/core';
-
-const provider = createOllamaProvider({
-  baseUrl: 'http://localhost:11434',
-  model: 'deepseek-coder'
-});
-
-const available = await provider.isAvailable();
-console.log('Ollama available:', available);
-```
-
-### OpenAI (Cloud)
-
-OpenAI provides GPT-4 and other powerful models via API.
-
-**Environment Variables:**
-```bash
-export OPENAI_API_KEY="sk-..."  # Required
-export OPENAI_BASE_URL="https://api.openai.com/v1"  # Optional
-export OPENAI_MODEL="gpt-4"  # Optional
-```
-
-**Supported Models:**
-- `gpt-4` - Most capable
-- `gpt-4-turbo` - Faster, cheaper
-- `gpt-4o` - Latest GPT-4 Omni
-- `gpt-4o-mini` - Smaller, faster
-- `gpt-3.5-turbo` - Budget option
-
-**Code:**
-```typescript
-import { createOpenAIProvider } from '@qa360/core';
-
-const provider = createOpenAIProvider({
-  apiKey: 'sk-...',
-  model: 'gpt-4'
-});
-```
-
-### Anthropic (Cloud)
-
-Anthropic provides Claude models with strong coding capabilities.
-
-**Environment Variables:**
-```bash
-export ANTHROPIC_API_KEY="sk-ant-..."  # Required
-export ANTHROPIC_BASE_URL="https://api.anthropic.com"  # Optional
-export ANTHROPIC_MODEL="claude-3-5-sonnet-20241022"  # Optional
-```
-
-**Supported Models:**
-- `claude-3-5-sonnet-20241022` - Best for code
-- `claude-3-5-sonnet-20240620` - Previous Sonnet
-- `claude-3-opus-20240229` - Most capable
-- `claude-3-haiku-20240307` - Fastest
-
-**Code:**
-```typescript
-import { createAnthropicProvider } from '@qa360/core';
-
-const provider = createAnthropicProvider({
-  apiKey: 'sk-ant-...',
-  model: 'claude-3-5-sonnet-20241022'
-});
-```
-
-### Mock (Testing)
-
-The Mock provider returns predefined responses without making network calls. Useful for testing and development.
-
-```typescript
-import { createMockProvider } from '@qa360/core';
-
-const provider = createMockProvider({
-  delay: 50,  // Simulate latency in ms
-  responses: {
-    'test': 'Mock response for test'
-  }
-});
-```
-
-## Advanced Usage
-
-### Check Provider Availability
-
-```typescript
-import { getProviderStatus, detectBestProvider } from '@qa360/core';
-
-// Check all providers
-const status = await getProviderStatus();
-console.log(status);
-// { deepseek: true, ollama: false, openai: false, anthropic: false, mock: true }
-
-// Detect best available
-const best = await detectBestProvider();
-console.log('Best provider:', best);
-// 'deepseek'
-```
-
-### Custom Fallback Chain
-
-```typescript
-import { createLLMProvider } from '@qa360/core';
-
-// Custom order: DeepSeek → Ollama → Mock (skip cloud providers)
-const provider = await createLLMProvider({
-  chain: ['deepseek', 'ollama', 'mock']
-});
-```
-
-### Streaming Responses
-
-```typescript
-const provider = await createLLMProvider();
-
-// Stream tokens as they arrive
-for await (const chunk of provider.stream({
-  prompt: 'Write a test function',
-  maxTokens: 500
-})) {
-  process.stdout.write(chunk);
-}
-```
-
-### Count Tokens
-
-```typescript
-const provider = await createLLMProvider();
-
-const text = 'Your prompt here...';
-const tokens = provider.countTokens(text);
-
-console.log(`Approximate tokens: ${tokens}`);
-```
-
-## Error Handling
-
-All providers throw specific errors that you can catch:
-
-```typescript
-import { DeepSeekError, OpenAIError, AnthropicError, OllamaError } from '@qa360/core';
-
-try {
-  const response = await provider.generate({ prompt: 'test' });
-} catch (error) {
-  if (error instanceof DeepSeekError) {
-    console.error('DeepSeek error:', error.message);
-    console.error('Details:', error.details);
-  } else if (error instanceof OpenAIError) {
-    console.error('OpenAI error:', error.message);
-    console.error('Details:', error.details);
-  } else if (error instanceof OllamaError) {
-    console.error('Ollama error:', error.message);
-  } else if (error instanceof AnthropicError) {
-    console.error('Anthropic error:', error.message);
-  } else {
-    console.error('Unknown error:', error);
-  }
-}
-```
-
-## CLI Integration
-
-The AI module is integrated into QA360 CLI commands:
-
-```bash
-# List all AI providers and their status
-qa360 ai list
-
-# Test a specific provider
-qa360 ai use deepseek --verify
-qa360 ai use ollama
-
-# Generate tests using AI
-qa360 ai generate "Write tests for login API" --type api
-
-# Benchmark all providers
-qa360 ai benchmark
-
-# Show AI configuration
-qa360 ai config
-```
-
-## Best Practices
-
-### 1. Use DeepSeek for Production
-
-DeepSeek offers the best value for production use:
-- GPT-4 level performance at 95% lower cost
-- Reliable API with excellent uptime
-- Specialized code model (`deepseek-coder`)
-
-### 2. Use Ollama for Local Development
-
-Ollama provides free, local inference. It's ideal for:
-- Development environments
-- Offline work
-- Sensitive data (no external API calls)
-
-### 3. Set Fallback API Keys for Production
-
-Configure cloud API keys as fallback:
-```bash
-# .env - Priority order
-DEEPSEEK_API_KEY=sk-...  # Primary (best value)
-OPENAI_API_KEY=sk-...    # Fallback
-ANTHROPIC_API_KEY=sk-ant-...  # Fallback
-```
-
-### 4. Use Mock for Testing
-
-The Mock provider ensures deterministic tests:
-```typescript
-const mockProvider = createMockProvider({
-  responses: {
-    'generate login test': 'it("should login", () => { ... })'
-  }
-});
-```
-
-### 4. Handle Provider Failures Gracefully
-
-```typescript
-const provider = await createLLMProvider();
-const available = await provider.isAvailable();
-
-if (!available) {
-  console.warn('Primary provider unavailable, using fallback');
-}
-```
-
-## Architecture
-
-```
-┌─────────────────────────────────────────────────────────┐
-│                   QA360 AI Module                        │
-├─────────────────────────────────────────────────────────┤
-│                                                         │
-│  ┌──────────────┐       ┌──────────────┐              │
-│  │   Provider   │       │   Provider   │              │
-│  │   Factory    │───────│   Fallback   │              │
-│  └──────────────┘       └──────────────┘              │
-│         │                       │                       │
-│         ▼                       ▼                       │
-│  ┌──────────────┐       ┌──────────────┐              │
-│  │  DeepSeek    │───────│   Ollama     │              │
-│  │  Provider    │       │  Provider    │              │
-│  └──────────────┘       └──────────────┘              │
-│         │                       │                       │
-│         ▼                       ▼                       │
-│  ┌──────────────┐       ┌──────────────┐              │
-│  │   OpenAI     │───────│  Anthropic   │              │
-│  │  Provider    │       │  Provider    │              │
-│  └──────────────┘       └──────────────┘              │
-│         │                       │                       │
-│         ▼                       ▼                       │
-│  ┌──────────────────────────────────────────┐         │
-│  │            Mock Provider                  │         │
-│  └──────────────────────────────────────────┘         │
-│                                                         │
-└─────────────────────────────────────────────────────────┘
-```
-
-## Reference
-
-### LLMProvider Interface
-
-```typescript
-interface LLMProvider {
-  name: string;                    // Provider name
-  models: string[];                // Supported models
-
-  generate(request: GenerationRequest): Promise<GenerationResponse>;
-  stream?(request: GenerationRequest): AsyncIterable<string>;
-  countTokens(text: string): number;
-  isAvailable(): Promise<boolean>;
-}
-
-interface GenerationRequest {
-  prompt: string;
-  systemPrompt?: string;
-  maxTokens?: number;
-  temperature?: number;
-}
-
-interface GenerationResponse {
-  content: string;
-  usage: {
-    promptTokens: number;
-    completionTokens: number;
-    totalTokens: number;
-  };
-  model: string;
-  finishReason: 'stop' | 'length' | 'content_filter';
-}
-```
-
-### ProviderFactoryConfig
-
-```typescript
-interface ProviderFactoryConfig {
-  preferred?: 'deepseek' | 'ollama' | 'openai' | 'anthropic' | 'mock';
-  chain?: Array<'deepseek' | 'ollama' | 'openai' | 'anthropic' | 'mock'>;
-  skip?: Array<'deepseek' | 'ollama' | 'openai' | 'anthropic' | 'mock'>;
-
-  deepseek?: { apiKey?: string; baseURL?: string; model?: string; timeout?: number };
-  ollama?: { baseUrl?: string; model?: string; timeout?: number };
-  openai?: { apiKey?: string; baseURL?: string; model?: string; timeout?: number };
-  anthropic?: { apiKey?: string; baseURL?: string; model?: string; timeout?: number };
-  mock?: { delay?: number; responses?: Record<string, string> };
-}
-```
-
-## Troubleshooting
-
-### DeepSeek Not Available
-
-```bash
-# Verify API key
-echo $DEEPSEEK_API_KEY
-
-# Test API key
-curl https://api.deepseek.com/v1/models \
-  -H "Authorization: Bearer $DEEPSEEK_API_KEY"
-
-# Set API key
-export DEEPSEEK_API_KEY="your_api_key_here"
-```
-
-### Ollama Not Available
-
-```bash
-# Check Ollama is running
-curl http://localhost:11434/api/tags
-
-# Start Ollama
-ollama serve
-
-# Pull a model
-ollama pull deepseek-coder
-```
-
-### OpenAI/Anthropic API Errors
-
-```bash
-# Verify API key
-echo $OPENAI_API_KEY
-
-# Test API key
-curl https://api.openai.com/v1/models \
-  -H "Authorization: Bearer $OPENAI_API_KEY"
-```
-
-### Provider Selection Issues
-
-```typescript
-// Debug provider selection
-import { getProviderStatus } from '@qa360/core';
-
-const status = await getProviderStatus();
-console.table(status);
-```
-
-## See Also
-
-- [DeepSeek Platform](https://platform.deepseek.com)
-- [DeepSeek API Docs](https://platform.deepseek.com/docs)
-- [Ollama Documentation](https://ollama.com/docs)
-- [OpenAI API Reference](https://platform.openai.com/docs/api-reference)
-- [Anthropic API Reference](https://docs.anthropic.com/claude/reference)