@od-oneapp/ai-platform 0.1.0

package/src/agents/evaluator-optimizer/README.md

@@ -0,0 +1,612 @@
+# Evaluator-Optimizer Feedback Loop Pattern
+
+Comprehensive implementation of the iterative quality refinement pattern from **VERCEL_AI_SDK_V6_PATTERNS.md section
+4.2**.
+
+This module implements an automated feedback loop that iteratively evaluates and optimizes content until it meets a
+defined quality threshold.
+
+## Pattern Overview
+
+### 5-Step Workflow
+
+```
+STEP 1: Generate Initial Content
+         ↓
+STEP 2: Evaluate Quality
+         ↓
+STEP 3: Quality Check
+    ├─ Yes (≥ threshold) → DELIVER
+    └─ No (< threshold) → Optimize
+         ↓
+STEP 4: Optimize Content
+         ↓
+STEP 5: Re-evaluate
+    └─ Repeat Steps 3-5 (max cycles)
+```
+
+### 9-Dimensional Evaluation
+
+Content is assessed across nine dimensions:
+
+1. **Quality Score** (1-10): Overall quality rating
+2. **Tone Preservation**: Is original tone maintained?
+3. **Nuance Preservation**: Are subtle meanings preserved?
+4. **Cultural Accuracy**: Is content culturally appropriate?
+5. **Technical Correctness**: Are facts/claims accurate?
+6. **Specific Issues**: Concrete problems identified
+7. **Improvement Suggestions**: Actionable feedback
+8. **Strengths**: Aspects to preserve
+9. **Meets Threshold**: Binary pass/fail at target score
+
+## Installation
+
+```bash
+npm install ai zod
+```
+
+## Quick Start
+
+### Basic Usage
+
+```typescript
+import { createEvaluatorOptimizer, createLLMEvaluator, createPromptOptimizer } from "./evaluator-optimizer";
+
+const loop = createEvaluatorOptimizer({
+  maxIterations: 5,
+  targetScore: 8,
+  evaluator: createLLMEvaluator(),
+  optimizer: createPromptOptimizer(),
+  enableLogging: true
+});
+
+const result = await loop.runFeedbackLoop("Initial content", "Original user request");
+
+console.log(`Final Score: ${result.finalEvaluation.qualityScore}/10`);
+console.log(`Optimized Content:\n${result.finalContent}`);
+```
+
+### Using Helper Function
+
+```typescript
+import { runFeedbackLoop } from "./evaluator-optimizer";
+
+const result = await runFeedbackLoop("Initial content", "Original request", {
+  maxIterations: 5,
+  targetScore: 8,
+  enableLogging: true
+});
+```
+
+## Components
+
+### 1. Evaluators
+
+Three evaluation strategies:
+
+#### LLMEvaluator (LLM-as-Judge)
+
+Uses language model for nuanced assessment across all 9 dimensions.
+
+```typescript
+import { createLLMEvaluator } from "./evaluator";
+
+const evaluator = createLLMEvaluator("openai/gpt-4o");
+```
+
+**Pros:**
+
+- Nuanced evaluation
+- Handles complex criteria
+- Provides detailed feedback
+
+**Cons:**
+
+- Higher latency
+- Higher cost
+- Can be inconsistent
+
+#### RubricEvaluator
+
+Uses predefined scoring rubric for consistent, deterministic evaluation.
+
+```typescript
+import { createRubricEvaluator } from "./evaluator";
+
+const evaluator = createRubricEvaluator();
+```
+
+**Pros:**
+
+- Fast and consistent
+- Deterministic
+- Low cost
+
+**Cons:**
+
+- Less nuanced
+- Requires custom rubric definition
+- Limited to defined criteria
+
+#### CompositeEvaluator
+
+Combines multiple evaluators with weighted averaging.
+
+```typescript
+import { createCompositeEvaluator, createLLMEvaluator, createRubricEvaluator } from "./evaluator";
+
+const composite = createCompositeEvaluator([
+  { evaluator: createLLMEvaluator(), weight: 0.7 },
+  { evaluator: createRubricEvaluator(), weight: 0.3 }
+]);
+```
+
+**Pros:**
+
+- Balanced assessment
+- Combines strengths
+- More robust
+
+**Cons:**
+
+- Higher latency (runs in parallel)
+- Complexity
+
+### 2. Optimizers
+
+Three optimization strategies:
+
+#### PromptOptimizer
+
+Comprehensive optimization addressing all feedback issues.
+
+```typescript
+import { createPromptOptimizer } from "./optimizer";
+
+const optimizer = createPromptOptimizer("anthropic/claude-3-5-sonnet-20241022");
+```
+
+**Use When:**
+
+- Need comprehensive optimization
+- All issues must be addressed
+- Quality must improve significantly
+
+#### OutputOptimizer
+
+Strategy-based optimization (expand, refine, restructure).
+
+```typescript
+import { createOutputOptimizer } from "./optimizer";
+
+const optimizer = createOutputOptimizer(
+  "anthropic/claude-3-5-sonnet-20241022",
+  "expand" // Strategy: 'expand' | 'refine' | 'restructure'
+);
+```
+
+**Strategies:**
+
+- `expand`: Add detail and examples
+- `refine`: Polish and clarify
+- `restructure`: Improve organization
+
+#### HybridOptimizer
+
+Adapts optimization strategy based on iteration number.
+
+```typescript
+import { createHybridOptimizer } from "./optimizer";
+
+const optimizer = createHybridOptimizer();
+```
+
+**Strategy:**
+
+- Iterations 1-2: Output optimization (fast)
+- Iterations 3-4: Prompt optimization (comprehensive)
+- Iteration 5+: Output optimization (final polish)
+
+## Configuration
+
+### OptimizerConfig
+
+```typescript
+interface OptimizerConfig {
+  /**
+   * Maximum iterations for feedback loop (1-10)
+   * @default 5
+   */
+  maxIterations: number;
+
+  /**
+   * Target quality score (1-10)
+   * Content must reach this score to pass
+   * @default 8
+   */
+  targetScore: number;
+
+  /**
+   * Evaluator instance
+   */
+  evaluator: Evaluator;
+
+  /**
+   * Optimizer instance
+   */
+  optimizer: Optimizer;
+
+  /**
+   * Enable detailed logging
+   * @default false
+   */
+  enableLogging?: boolean;
+
+  /**
+   * Stop on first threshold achievement
+   * @default true
+   */
+  stopOnThreshold?: boolean;
+
+  /**
+   * Detect diminishing returns after N cycles
+   * @default 2
+   */
+  diminishingReturnThreshold?: number;
+}
+```
+
+### FeedbackLoopConfig
+
+Per-run configuration:
+
+```typescript
+interface FeedbackLoopConfig {
+  maxIterations?: number; // Override default
+  targetScore?: number; // Override default
+  enableLogging?: boolean;
+  stopOnThreshold?: boolean;
+  diminishingReturnThreshold?: number;
+}
+```
+
+## Result Structure
+
+```typescript
+interface OptimizedResult {
+  // Final output
+  finalContent: string;
+  finalEvaluation: EvaluationResult;
+
+  // Process metrics
+  cyclesCompleted: number;
+  meetsThreshold: boolean;
+  scoreHistory: number[]; // Scores at each cycle
+  contentHistory: string[]; // Content versions
+
+  // Loop termination
+  terminationReason: "threshold_met" | "max_iterations" | "diminishing_returns" | "error";
+  error?: string;
+
+  // Timing metadata
+  metadata?: {
+    startTime: string;
+    endTime: string;
+    processingTimeMs: number;
+    averageScoreImprovement: number;
+  };
+}
+```
+
+## Use Cases
+
+### 1. Content Quality Assurance
+
+Marketing copy, technical documentation requiring iterative refinement.
+
+```typescript
+const loop = createEvaluatorOptimizer({
+  maxIterations: 5,
+  targetScore: 8,
+  evaluator: createLLMEvaluator(),
+  optimizer: createPromptOptimizer()
+});
+
+const result = await loop.runFeedbackLoop(marketingCopy, "Write compelling marketing copy");
+```
+
+### 2. Translation Quality
+
+Multi-language content with tone/nuance preservation.
+
+```typescript
+const loop = createEvaluatorOptimizer({
+  maxIterations: 7,
+  targetScore: 9,
+  evaluator: createCompositeEvaluator([
+    { evaluator: createLLMEvaluator("openai/gpt-4o"), weight: 0.6 },
+    { evaluator: createRubricEvaluator(), weight: 0.4 }
+  ]),
+  optimizer: createHybridOptimizer()
+});
+
+const result = await loop.runFeedbackLoop(translatedText, "Translate while preserving tone and nuance");
+```
+
+### 3. Code Review Automation
+
+Evaluate code quality, suggest improvements, re-evaluate.
+
+```typescript
+const loop = createEvaluatorOptimizer({
+  maxIterations: 4,
+  targetScore: 8,
+  evaluator: createLLMEvaluator("openai/gpt-4o"),
+  optimizer: createOutputOptimizer(undefined, "refine")
+});
+
+const result = await loop.runFeedbackLoop(codeContent, "Review and improve code quality");
+```
+
+### 4. Creative Writing
+
+Iterate on stories, articles, proposals.
+
+```typescript
+const loop = createEvaluatorOptimizer({
+  maxIterations: 6,
+  targetScore: 8.5,
+  evaluator: createLLMEvaluator(),
+  optimizer: createHybridOptimizer(),
+  enableLogging: true
+});
+
+const result = await loop.runFeedbackLoop(draftStory, "Write engaging narrative with strong character development");
+```
+
+### 5. Compliance Verification
+
+Legal/medical content requiring accuracy checks.
+
+```typescript
+const loop = createEvaluatorOptimizer({
+  maxIterations: 5,
+  targetScore: 9, // Strict threshold
+  evaluator: createCompositeEvaluator([
+    { evaluator: createLLMEvaluator("openai/gpt-4o"), weight: 0.8 },
+    { evaluator: createRubricEvaluator(), weight: 0.2 }
+  ]),
+  optimizer: createPromptOptimizer("anthropic/claude-3-5-sonnet-20241022")
+});
+
+const result = await loop.runFeedbackLoop(legalText, "Ensure compliance and accuracy for legal document");
+```
+
+## Best Practices
+
+### 1. Set Realistic Thresholds
+
+- **8/10**: Standard content (marketing, blog posts)
+- **8.5/10**: Important content (product descriptions, key docs)
+- **9/10**: Critical content (legal, medical, compliance)
+
+### 2. Limit Iterations
+
+- Typical: 3-5 cycles
+- Complex content: 5-7 cycles
+- Max: 10 cycles (diminishing returns)
+
+### 3. Choose Right Evaluator
+
+- **LLM**: Nuanced content, complex criteria
+- **Rubric**: Speed, consistency, cost
+- **Composite**: Balance accuracy and performance
+
+### 4. Choose Right Optimizer
+
+- **Prompt**: Comprehensive improvement
+- **Output**: Fast, targeted optimization
+- **Hybrid**: Adaptive optimization across iterations
+
+### 5. Monitor Performance
+
+```typescript
+const result = await loop.runFeedbackLoop(content, request);
+
+console.log(`Score improvement: ${result.scoreHistory[result.scoreHistory.length - 1] - result.scoreHistory[0]}/10`);
+
+console.log(`Cycles needed: ${result.cyclesCompleted}/${config.maxIterations}`);
+
+console.log(`Time per cycle: ${(result.metadata?.processingTimeMs || 0) / result.cyclesCompleted}ms`);
+```
+
+### 6. Handle Errors Gracefully
+
+```typescript
+const result = await loop.runFeedbackLoop(content, request);
+
+if (result.terminationReason === "error") {
+  console.error(`Optimization failed: ${result.error}`);
+  // Use original content
+  return content;
+}
+
+if (result.terminationReason === "diminishing_returns") {
+  console.warn("Stopped due to diminishing returns");
+}
+
+return result.finalContent;
+```
+
+## Advanced Patterns
+
+### 1. Batch Processing
+
+Optimize multiple items efficiently:
+
+```typescript
+const items = [...];
+const results = await Promise.all(
+  items.map(item =>
+    loop.runFeedbackLoop(item.content, item.request)
+  )
+);
+```
+
+### 2. Cost Optimization
+
+Use fast evaluator/optimizer until near threshold:
+
+```typescript
+// Initial pass with fast models
+let result = await fastLoop.runFeedbackLoop(content, request);
+
+// Final polish with high-quality models if needed
+if (result.finalEvaluation.qualityScore < 8) {
+  result = await qualityLoop.runFeedbackLoop(result.finalContent, request);
+}
+```
+
+### 3. Conditional Optimization
+
+Skip optimization for already-high-quality content:
+
+```typescript
+const result = await loop.runFeedbackLoop(content, request);
+
+// Additional polishing only if needed
+if (result.cyclesCompleted > 3) {
+  // Many cycles = struggling to reach threshold
+  // Consider manual review
+  console.warn("Content required excessive optimization");
+}
+```
+
+### 4. Feedback Preservation
+
+Maintain score improvements across iterations:
+
+```typescript
+// Track score trend
+const trend = result.scoreHistory.slice(-3).map((score, i) => (i > 0 ? score - result.scoreHistory[i - 1] : 0));
+
+const isImproving = trend.some((delta) => delta > 0);
+const isStagnant = trend.every((delta) => delta === 0);
+```
+
+## Performance Considerations
+
+### Latency
+
+- **LLMEvaluator**: 2-5s per evaluation
+- **RubricEvaluator**: <100ms per evaluation
+- **CompositeEvaluator**: 2-5s (parallel)
+
+### Cost
+
+- **LLM evaluations**: 0.01-0.05 USD each
+- **LLM optimizations**: 0.02-0.10 USD each
+- **Full loop (5 cycles)**: 0.15-0.75 USD
+
+### Optimization
+
+```typescript
+// Use fast model for multiple iterations
+const fastEval = createLLMEvaluator("openai/gpt-4o-mini");
+const qualityOpt = createPromptOptimizer("anthropic/claude-3-5-sonnet-20241022");
+
+// Process many items
+for (const item of items) {
+  const result = await loop.runFeedbackLoop(item.content, item.request);
+  // ~10-20s total per item
+}
+```
+
+## Troubleshooting
+
+### Content Not Reaching Threshold
+
+```typescript
+// Check what's failing
+result.finalEvaluation.specificIssues; // Array of problems
+
+// Try with different optimizer
+const betterOptimizer = createPromptOptimizer("anthropic/claude-3-5-sonnet-20241022");
+
+// Try with higher quality evaluator
+const betterEvaluator = createLLMEvaluator("openai/gpt-4o");
+```
+
+### Stuck in Loops
+
+```typescript
+// Enable diminishing returns detection
+if (result.terminationReason === "diminishing_returns") {
+  // Content improved initially but hit plateau
+  // Return best version so far
+  return result.contentHistory[result.scoreHistory.lastIndexOf(Math.max(...result.scoreHistory))];
+}
+```
+
+### High Costs
+
+```typescript
+// Use cheaper models
+const cheapEval = createLLMEvaluator("openai/gpt-4o-mini");
+const cheapOpt = createPromptOptimizer("openai/gpt-4o-mini");
+
+// Or use rubric evaluator
+const rubricEval = createRubricEvaluator();
+
+// Or reduce max iterations
+loop.runFeedbackLoop(content, request, { maxIterations: 3 });
+```
+
+## API Reference
+
+### Main Functions
+
+- `createEvaluatorOptimizer(config): EvaluatorOptimizer`
+- `runFeedbackLoop(content, request, config): Promise<OptimizedResult>`
+
+### Evaluator Factory Functions
+
+- `createLLMEvaluator(modelId?): Evaluator`
+- `createRubricEvaluator(rubric?): Evaluator`
+- `createCompositeEvaluator(evaluators?): Evaluator`
+
+### Optimizer Factory Functions
+
+- `createPromptOptimizer(modelId?): Optimizer`
+- `createOutputOptimizer(modelId?, strategy?): Optimizer`
+- `createHybridOptimizer(promptModel?, outputModel?): Optimizer`
+
+### Types
+
+- `EvaluationResult`: Evaluation output
+- `OptimizedResult`: Final optimization result
+- `OptimizerConfig`: Configuration interface
+- `Evaluator`: Evaluator interface
+- `Optimizer`: Optimizer interface
+
+## Examples
+
+See `usage-example.ts` for 10 complete examples:
+
+1. Basic Content Optimization
+2. Technical Documentation Refinement
+3. Multi-Evaluator Approach
+4. Strategy-Based Optimization
+5. Quality Score Tracking
+6. Error Handling and Recovery
+7. Quick Setup with Defaults
+8. Batch Content Optimization
+9. Custom Evaluation Criteria
+10. Production-Ready Implementation
+
+## References
+
+- VERCEL_AI_SDK_V6_PATTERNS.md section 4.2
+- AI SDK Documentation: https://sdk.vercel.ai
+- Anthropic Claude: https://claude.ai
+- OpenAI GPT: https://openai.com