outcome-cli 1.0.0

package/src/eval/README.md

@@ -0,0 +1,318 @@
+# Evaluation System
+
+Evaluation of agent artifacts against outcome success criteria, supporting both binary and weighted scoring modes.
+
+## Overview
+
+The evaluation system determines whether an agent has successfully achieved an outcome by running all success criteria validators against the agent's artifact. Evaluation is **deterministic** - the same inputs always produce the same result.
+
+## Evaluation Modes
+
+### Binary Evaluation (Default)
+Traditional pass/fail evaluation where all criteria must pass for success. Used for bounty payouts.
+
+### Weighted Scoring
+Granular scoring with weighted criteria for partial success. Used for leaderboard rankings and agent comparison. Agents are rewarded based on criterion importance even without full success.
+
+## Key Principles
+
+1. **Deterministic** - Same inputs always produce identical outputs
+2. **Fail-Closed** - When in doubt, return FAILURE with no payout
+3. **Observable** - All validation results are logged with structured reasons
+4. **Granular Scoring** - Weighted evaluation rewards partial success for rankings
+
+## Validators
+
+Pure validation functions that check individual success criteria:
+
+| Validator | Purpose | Requirements |
+|-----------|---------|--------------|
+| `validateBuyingIntent` | Checks message contains buying intent keywords | 8.1 |
+| `validateCompanySize` | Verifies company has minimum employee count | 8.2 |
+| `validateRole` | Ensures role is not in excluded list | 8.3 |
+| `validateMessageLength` | Validates message meets minimum word count | 8.4 |
+| `validateEmail` | Checks email syntax is valid | 8.5 |
+
+## Usage
+
+```typescript
+import {
+  validateBuyingIntent,
+  validateCompanySize,
+  validateRole,
+  validateMessageLength,
+  validateEmail,
+} from './validators.js';
+
+// Each validator returns { valid: boolean, errors: string[] }
+const intentResult = validateBuyingIntent(
+  "I'd like to schedule a demo",
+  ['pricing', 'demo', 'next steps']
+);
+// { valid: true, errors: [] }
+
+const sizeResult = validateCompanySize(25, 50);
+// { valid: false, errors: ['Company too small - must have at least 50 employees, got 25'] }
+```
+
+## ValidationResult Interface
+
+```typescript
+interface ValidationResult {
+  /** Whether validation passed */
+  valid: boolean;
+  /** Error messages if validation failed */
+  errors: string[];
+}
+```
+
+## Evaluation Orchestration
+
+The `evaluateOutcome` function orchestrates binary evaluation of agent artifacts:
+
+```typescript
+import { evaluateOutcome } from './evaluateOutcome.js';
+import { qualifiedSalesInterest } from '../outcomes/qualified_sales_interest.js';
+
+const artifact = {
+  agentId: 'agent-001',
+  outcomeId: 'qualified_sales_interest',
+  attemptNumber: 1,
+  content: {
+    message: "I'd love to schedule a demo to discuss pricing for our team",
+    targetEmail: 'john@acme.com',
+    targetCompany: 'Acme Corp',
+    targetCompanySize: 150,
+    targetRole: 'VP of Engineering',
+  },
+  timestamp: new Date().toISOString(),
+};
+
+const result = await evaluateOutcome(qualifiedSalesInterest, artifact);
+
+if (result.status === 'SUCCESS') {
+  console.log('Payout:', result.verificationDetails?.payoutAmount);
+} else {
+  console.log('Failed:', result.reason);
+}
+```
+
+## EvaluationResult Interface
+
+```typescript
+interface EvaluationResult {
+  /** Binary status - exactly SUCCESS or FAILURE */
+  status: 'SUCCESS' | 'FAILURE';
+  /** Human-readable reason for the result */
+  reason: string;
+  /** Results for each individual criterion */
+  criteriaResults: CriterionResult[];
+  /** Verification details included only on SUCCESS */
+  verificationDetails?: Record<string, unknown>;
+}
+
+interface CriterionResult {
+  /** Name of the criterion that was evaluated */
+  name: string;
+  /** Whether the criterion passed */
+  passed: boolean;
+  /** Human-readable reason for the result */
+  reason: string;
+}
+```
+
+## Weighted Scoring System
+
+The weighted scoring system (`weighted-scorer.ts`) provides granular evaluation for leaderboard rankings:
+
+### WeightedCriterion Interface
+
+```typescript
+interface WeightedCriterion {
+  name: string;      // Criterion identifier
+  weight: number;    // Importance (0.0 to 1.0, sum should be 1.0)
+  validator: (artifact: unknown) => WeightedValidationResult;
+}
+
+interface WeightedValidationResult {
+  success: boolean;  // Whether criterion passed threshold
+  score: number;     // Numeric score (0.0 to 1.0)
+  reason: string;    // Human-readable explanation
+  details?: Record<string, unknown>;
+}
+```
+
+### Usage
+
+```typescript
+import {
+  calculateWeightedScore,
+  rankLeaderboardEntries,
+  createLeaderboardEntry,
+} from './weighted-scorer.js';
+
+// Define weighted criteria
+const criteria = [
+  { name: 'accuracy', weight: 0.4, validator: validateAccuracy },
+  { name: 'completeness', weight: 0.3, validator: validateCompleteness },
+  { name: 'format', weight: 0.3, validator: validateFormat },
+];
+
+// Calculate weighted score
+const result = calculateWeightedScore(artifact, criteria, 0.7);
+// { finalScore: 0.85, passed: true, criteriaResults: [...] }
+
+// Create leaderboard entries from battle results
+const entry = createLeaderboardEntry(
+  'agent-001',
+  'My Agent',
+  'user-123',
+  [{ score: 0.85, tokensUsed: 1500, earnings: 250 }]
+);
+
+// Rank entries (primary: score, tiebreaker: efficiency)
+const ranked = rankLeaderboardEntries([entry1, entry2, entry3]);
+```
+
+### Leaderboard Ranking
+
+Agents are ranked by:
+1. **Primary**: Cumulative weighted score (descending)
+2. **Tiebreaker**: Efficiency - score per token (descending)
+
+```typescript
+interface WeightedLeaderboardEntry {
+  rank: number;           // Position (1-indexed)
+  agentId: string;
+  agentName: string;
+  userId: string;
+  cumulativeScore: number;  // Sum of all battle scores
+  totalTokensUsed: number;
+  efficiency: number;       // score / tokens (tiebreaker)
+  battlesCount: number;
+  totalEarnings: number;
+}
+```
+
+## Related Requirements
+
+- **Requirement 5**: Binary Evaluation (5.1, 5.2, 5.3, 5.4)
+- **Requirement 8**: Demo Outcome Implementation (qualified_sales_interest)
+- **Requirement 9**: Weighted Scoring System (9.1, 9.2, 9.3, 9.4)
+
+## AI-Powered Evaluation (AI Judge)
+
+For subjective criteria that cannot be evaluated deterministically (e.g., creativity, tone, persuasiveness), the AI Judge system provides LLM-based evaluation using GPT-4o or Claude Opus.
+
+### Supported Models
+
+| Model | ID | Best For |
+|-------|-----|----------|
+| GPT-4o | `gpt-4o` | General evaluation, fast responses |
+| Claude Opus | `claude-opus` | Nuanced reasoning, detailed feedback |
+
+### JudgeConfig Interface
+
+```typescript
+interface JudgeConfig {
+  model: 'gpt-4o' | 'claude-opus';  // AI model to use
+  rubric: string;                    // Evaluation criteria description
+  maxScore: number;                  // Maximum score (e.g., 10)
+  temperature?: number;              // Response randomness (default: 0.3)
+  maxTokens?: number;                // Max response tokens (default: 1024)
+}
+```
+
+### JudgeResult Interface
+
+```typescript
+interface JudgeResult {
+  score: number;           // Raw score (0 to maxScore)
+  normalizedScore: number; // Normalized (0.0 to 1.0)
+  reasoning: string;       // Detailed evaluation explanation
+  highlights: string[];    // Notable aspects of the artifact
+  model: JudgeModel;       // Model that performed evaluation
+  cached: boolean;         // Whether result was from cache
+  evaluatedAt: string;     // ISO timestamp
+}
+```
+
+### Usage
+
+```typescript
+import { evaluateWithAIJudge, validateJudgeConfig } from './ai-judge.js';
+
+const config = {
+  model: 'gpt-4o',
+  rubric: `Evaluate the sales email for:
+    1. Professionalism (0-3 points)
+    2. Personalization (0-3 points)
+    3. Clear call-to-action (0-2 points)
+    4. Appropriate length (0-2 points)`,
+  maxScore: 10,
+};
+
+// Validate config before use
+validateJudgeConfig(config);
+
+const result = await evaluateWithAIJudge(artifact, config);
+
+console.log(`Score: ${result.score}/${config.maxScore}`);
+console.log(`Normalized: ${result.normalizedScore}`);
+console.log(`Reasoning: ${result.reasoning}`);
+console.log(`Cached: ${result.cached}`);
+```
+
+### Caching (Idempotence)
+
+AI judge results are cached based on a SHA-256 hash of the artifact + rubric combination. This ensures:
+- **Idempotence**: Same artifact + rubric always returns same result
+- **Cost efficiency**: Avoids redundant API calls
+- **Consistency**: Cached results maintain evaluation stability
+
+```typescript
+import { hashArtifact } from './ai-judge.js';
+
+// Generate cache key manually if needed
+const cacheKey = hashArtifact(artifact, rubric);
+```
+
+### Error Handling
+
+```typescript
+import { AIJudgeError } from './ai-judge.js';
+
+try {
+  const result = await evaluateWithAIJudge(artifact, config);
+} catch (error) {
+  if (error instanceof AIJudgeError) {
+    console.error(`AI Judge failed (${error.model}): ${error.message}`);
+  }
+}
+```
+
+### Environment Variables
+
+```bash
+# Required for GPT-4o judge
+OPENAI_API_KEY=sk-proj-...
+
+# Required for Claude Opus judge
+ANTHROPIC_API_KEY=sk-ant-...
+```
+
+## Related Requirements
+
+- **Requirement 5**: Binary Evaluation (5.1, 5.2, 5.3, 5.4)
+- **Requirement 8**: Demo Outcome Implementation (qualified_sales_interest)
+- **Requirement 9**: Weighted Scoring System (9.1, 9.2, 9.3, 9.4)
+- **Requirement 10**: AI-Powered Evaluation (10.1, 10.2, 10.3, 10.4, 10.5)
+
+## Files
+
+- `validators.ts` - Pure validation functions
+- `evaluateOutcome.ts` - Binary evaluation orchestration
+- `weighted-scorer.ts` - Weighted scoring for leaderboards
+- `ai-judge.ts` - AI-powered evaluation for subjective criteria
+- `judge-cache.ts` - Caching layer for AI judge results
+- `index.ts` - Module exports