adaptive-memory-multi-model-router 2.14.46 → 2.14.47

package/research/loss-functions.md

@@ -1,545 +0,0 @@
-# Loss Functions for LLM Routing Optimization
-
-**Date:** 2026-06-03  
-**Author:** A3M Research  
-**Target:** Improve RouterArena score from 70.32  
-
----
-
-## 1. Current A3M Cost Model Analysis
-
-### 1.1 Existing Routing Logic
-
-A3M's routing uses a **weighted scoring formula**:
-
-```typescript
-// From src/routing/advancedRouter.ts
-
-// Quality score (static, heuristic-based)
-quality_score: strengths.includes('premium') ? 0.95 : 
-               strengths.includes('reasoning') ? 0.90 : 
-               strengths.includes('fast') ? 0.82 : 0.80
-
-// Cost efficiency (linear penalty)
-costEfficiency(model, features) = (1 - avg_cost / 10) * 0.2-0.6
-
-// Final score
-total_score = quality_score * complexity_bias + cost_score * (1 - complexity_bias)
-
-// Online learning (EMA)
-quality_score = quality_score * (1 - alpha) + actual_rating * alpha
-```
-
-### 1.2 Current Score Calculation
-
-```typescript
-// Lines 302-340 in advancedRouter.ts
-let score = model.quality_score * 0.6;  // Base quality weight
-
-// Domain bonus (+0.2)
-if (features.domain && model.strengths.includes(domainBonus[domain])) 
-    score += 0.2;
-
-// Code bonus (+0.15)
-if (features.has_code && model.strengths.includes('coding')) 
-    score += 0.15;
-
-// Free tier preference (+0.2)
-if (features.complexity < 0.5 && model.strengths.includes('free')) 
-    score += 0.2;
-```
-
-### 1.3 Issues with Current Approach
-
-| Issue | Impact | Severity |
-|-------|--------|----------|
-| **No learned embeddings** | Keyword matching can't capture semantic similarity | High |
-| **No contrastive loss** | Can't distinguish "similar but different" queries | Medium |
-| **Static quality scores** | Provider quality varies by query type | High |
-| **Linear cost penalty** | Doesn't model diminishing returns | Medium |
-| **No latency in loss** | RouterArena penalizes slow routing | High |
-| **Single-objective** | No Pareto-optimal exploration | Medium |
-
----
-
-## 2. Literature Review
-
-### 2.1 RouteLLM (arXiv:2404.06035)
-
-**Key Insight:** Learned routing from pairwise preferences.
-
-**Architecture:**
-- BERT classifier on query embeddings
-- Trained on weak vs strong model comparisons
-- Binary preference: "Which model gives better answer?"
-
-**Loss Function:**
-```
-L = CrossEntropy(softmax(W * [q; m_w; m_s]), preference_label)
-```
-
-Where `q` = query embedding, `m_w` = winner model embedding, `m_s` = strong model embedding.
-
-**Results:**
-- 85% routing accuracy (exact tier match)
-- 70% cost savings vs all-premium
-
-**Relevance to A3M:** RouteLLM's pairwise training is what enables learned routing. A3M's rule-based approach gets 70.32 (vs 85% exact), but could benefit from hybrid training.
-
-### 2.2 RouterArena Benchmark (arXiv:2510.00202)
-
-**Scoring Formula:**
-```
-RouterArena_Score = 0.6 * Accuracy + 0.2 * Cost_Efficiency + 0.2 * Latency_Score
-
-where:
-  Accuracy = % queries routed to correct tier (exact or ±1)
-  Cost_Efficiency = 1 - (router_cost / baseline_cost)
-  Latency_Score = 1 - (router_latency / max_latency)
-```
-
-**Key Finding:** A3M scores 70.32 with heuristic routing. RouteLLM scores 48.07 with learned routing. **Heuristic can beat learned when cost matters.**
-
-**Relevance to A3M:** The scoring weights (60% accuracy, 20% cost, 20% latency) directly inform our loss function design.
-
-### 2.3 LLMRouterBench
-
-**Dataset:** 400K+ query-model pairs across 9 domains  
-**Task:** 4-tier classification (free → budget → mid → premium)  
-**Baseline:** TF-IDF + Logistic Regression = 62.3%  
-**State-of-art:** Learned embeddings + neural classifier = 78.1%
-
-**Loss Function Pattern:**
-```
-L = CrossEntropy(router(query), true_tier)
-  + λ * L2_regularization
-  + λ * cost_penalty
-```
-
-**Relevance to A3M:** Could incorporate tier classification loss into A3M's multi-signal classifier.
-
-### 2.4 Contrastive Learning for Routing
-
-**Paper:** SimCSE, MoCo, CLIP-style approaches
-
-**Idea:** Embed queries and model capabilities in same space.
-
-**Loss:**
-```
-L_contrastive = -log(exp(sim(q, m_pos)) / Σ exp(sim(q, m_neg)))
-```
-
-Where `sim` = cosine similarity, `m_pos` = correct model, `m_neg` = incorrect models.
-
-**Relevance to A3M:** A3M's current approach uses keyword matching. Contrastive learning could improve query embedding quality without full BERT classifier.
-
-### 2.5 Multi-Objective Optimization for Routing
-
-**Problem:** Quality, cost, latency are conflicting objectives.
-
-**Approaches:**
-1. **Weighted Sum:** `L = w1*Q + w2*(-C) + w3*(-L)` — simple but requires tuning
-2. **Pareto Front:** Find non-dominated solutions — expensive
-3. **Scalarization:** `L = Π (Q^α * C^β * L^γ)` — smooth tradeoffs
-
-**Recommended for A3M:** Weighted sum with dynamic weights based on query type.
-
----
-
-## 3. Recommended Loss Function for A3M
-
-### 3.1 Proposed Architecture: Hybrid Routing Loss
-
-```
-L_total = α * L_tier + β * L_cost + γ * L_latency + δ * L_contrastive
-```
-
-Where:
-- `L_tier` = Cross-entropy for tier classification
-- `L_cost` = Cost-aware margin loss
-- `L_latency` = Latency regression loss
-- `L_contrastive` = Contrastive query-model alignment
-
-### 3.2 Component Details
-
-#### Tier Classification Loss (L_tier)
-
-```python
-def tier_loss(logits, true_tier):
-    """
-    logits: [batch_size, 4] - raw scores for free/budget/mid/premium
-    true_tier: [batch_size] - ground truth tier (0-3)
-    
-    Standard cross-entropy with class weights
-    """
-    weights = torch.tensor([1.0, 1.5, 2.0, 3.0])  # Premium is rarest
-    return F.cross_entropy(logits, true_tier, weight=weights)
-```
-
-#### Cost-Aware Margin Loss (L_cost)
-
-```python
-def cost_margin_loss(scores, chosen_cost, best_cost, margin=0.1):
-    """
-    scores: routing scores for each model
-    chosen_cost: cost of selected model
-    best_cost: cost of optimal model
-    
-    Penalize choosing expensive models when cheaper options exist
-    """
-    cost_ratio = chosen_cost / (best_cost + 1e-6)
-    
-    # If cost ratio > 1.5, penalize heavily
-    if cost_ratio > 1.5:
-        return margin * (cost_ratio - 1.5) ** 2
-    return 0.0
-```
-
-#### Latency Regression Loss (L_latency)
-
-```python
-def latency_loss(predicted_latency, actual_latency):
-    """
-    penalize high latency predictions
-    
-    Using log-scale to handle wide latency range (50ms - 10s)
-    """
-    return F.mse_loss(
-        torch.log1p(predicted_latency),
-        torch.log1p(actual_latency)
-    )
-```
-
-#### Contrastive Alignment Loss (L_contrastive)
-
-```python
-def contrastive_loss(query_emb, model_emb, labels, temperature=0.1):
-    """
-    query_emb: [batch_size, dim] - query embeddings
-    model_emb: [num_models, dim] - model capability embeddings
-    labels: [batch_size] - ground truth model index
-    
-    InfoNCE loss: queries should be close to their correct model embeddings
-    """
-    # Normalize embeddings
-    query_emb = F.normalize(query_emb, dim=-1)
-    model_emb = F.normalize(model_emb, dim=-1)
-    
-    # Compute similarities
-    sim = torch.matmul(query_emb, model_emb.T) / temperature
-    
-    # Positive pairs (correct model)
-    loss = F.cross_entropy(sim, labels)
-    
-    return loss
-```
-
-### 3.3 Combined Loss Implementation
-
-```python
-class RoutingLoss(nn.Module):
-    def __init__(self, weights=(0.5, 0.2, 0.1, 0.2)):
-        super().__init__()
-        self.w_tier = weights[0]
-        self.w_cost = weights[1]
-        self.w_latency = weights[2]
-        self.w_contrastive = weights[3]
-        
-        # Learnable temperature for contrastive loss
-        self.temperature = nn.Parameter(torch.ones(1))
-    
-    def forward(self, 
-                tier_logits, tier_targets,           # Tier classification
-                chosen_costs, optimal_costs,          # Cost efficiency
-                pred_latencies, actual_latencies,     # Latency
-                query_emb, model_emb, emb_labels,      # Contrastive
-                cost_weight=0.3):                     # Dynamic weight
-        
-        # Normalize weights by cost_weight (high cost sensitivity → high β)
-        if cost_weight > 0.5:
-            self.w_cost = cost_weight
-            self.w_tier = 1 - cost_weight
-        
-        L_tier = tier_loss(tier_logits, tier_targets)
-        L_cost = cost_margin_loss(chosen_costs, optimal_costs)
-        L_lat = latency_loss(pred_latencies, actual_latencies)
-        L_contra = contrastive_loss(query_emb, model_emb, emb_labels, self.temperature)
-        
-        return (self.w_tier * L_tier + 
-                self.w_cost * L_cost + 
-                self.w_lat * L_lat + 
-                self.w_contrastive * L_contra)
-```
-
----
-
-## 4. Implementation Approach for A3M
-
-### 4.1 Phase 1: Embedding-Based Query Representation
-
-**Problem:** A3M currently uses keyword matching (12 signals, 5 dimensions).
-
-**Solution:** Add lightweight embeddings (no GPU required).
-
-```typescript
-// src/routing/queryEmbedder.ts
-
-import { pipeline } from '@xenova/transformers';
-
-let embedder: any = null;
-
-export async function getQueryEmbedding(query: string): Promise<Float32Array> {
-  if (!embedder) {
-    // Use sentence-transformers (onnx, CPU-friendly)
-    embedder = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
-  }
-  return await embedder(query, { pooling: 'mean', normalize: true });
-}
-
-// Cached for speed
-const embeddingCache = new LRUCache<string, Float32Array>(10000);
-```
-
-**Why:** MiniLM-L6-v2 is 22MB, CPU-fast, captures semantic similarity.
-
-### 4.2 Phase 2: Cost-Aware Scoring
-
-**Current:** Linear penalty `(1 - cost/10) * weight`
-
-**Proposed:** Log-scale penalty + diminishing returns
-
-```typescript
-// src/routing/costAwareScoring.ts
-
-export function costAwareScore(
-  quality: number,
-  cost_per_1k: number,
-  complexity: number
-): number {
-  // Log-scale cost penalty (more realistic)
-  const logCostPenalty = Math.log1p(cost_per_1k) / Math.log1p(10);
-  
-  // Complexity determines cost sensitivity
-  // Simple queries: cost matters more (bias toward cheap)
-  // Complex queries: quality matters more (bias toward better)
-  const costSensitivity = 1 - complexity;
-  
-  // Quality should saturate (90% vs 95% is small difference)
-  const qualitySigmoid = 1 / (1 + Math.exp(-10 * (quality - 0.8)));
-  
-  return (
-    0.6 * qualitySigmoid +
-    0.3 * (1 - logCostPenalty) * costSensitivity +
-    0.1 * (1 - costSensitivity)  // Latency proxy
-  );
-}
-```
-
-### 4.3 Phase 3: Contrastive Fine-Tuning (Optional)
-
-**For maximum RouterArena score improvement:**
-
-```python
-# scripts/fine_tune_routing.py
-
-from sentence_transformers import SentenceTransformer, InputExample, losses
-from torch import nn
-
-# 1. Create training data from A3M's existing benchmark
-# Query → (chosen_model, cost, quality_rating) → (positive, negative) pairs
-
-def create_contrastive_examples(benchmark_data):
-    examples = []
-    for query in benchmark_data:
-        for candidate in query.candidates:
-            if candidate.chosen:
-                pos = candidate.model_id
-            else:
-                neg = candidate.model_id
-            
-            examples.append(InputExample(
-                texts=[query.text, pos, neg],
-                label=1.0 if candidate.chosen else 0.0
-            ))
-    return examples
-
-# 2. Fine-tune embeddings
-model = SentenceTransformer('Xenova/all-MiniLM-L6-v2')
-train_loss = losses.ContrastiveLoss(model)
-
-model.fit(
-    train_objectives=[(train_examples, train_loss)],
-    epochs=5,
-    warmup_steps=100
-)
-
-# 3. Export for A3M
-model.save('models/routing-embeddings')
-```
-
-### 4.4 Phase 4: Online Learning Enhancement
-
-**Current:** EMA on `quality_score` (α=0.2)
-
-**Proposed:** Contextual bandit updates
-
-```typescript
-// src/routing/contextualBandit.ts
-
-interface RoutingFeedback {
-  query: string;
-  chosen_model: string;
-  reward: number;  // Computed from quality/cost/latency
-  
-  // Components
-  quality_rating: number;      // User feedback or cross-validation
-  actual_cost: number;
-  actual_latency: number;
-  response_correct: boolean;
-}
-
-export function updateWithFeedback(feedback: RoutingFeedback): void {
-  // Compute multi-objective reward
-  const reward = computeReward(
-    feedback.quality_rating,
-    feedback.actual_cost,
-    feedback.actual_latency,
-    feedback.response_correct
-  );
-  
-  // Thompson sampling for model selection
-  const models = getAvailableModels();
-  
-  for (const model of models) {
-    // Update posterior: Beta distribution per (query_type, model)
-    const key = getQueryType(feedback.query) + ':' + model;
-    const posterior = modelPosteriors[key];
-    
-    // Add reward observation
-    if (reward > 0.5) {
-      posterior.alpha += 1;  // Success
-    } else {
-      posterior.beta += 1;   // Failure
-    }
-  }
-}
-
-function computeReward(quality, cost, latency, correct): number {
-  // Normalize to [0, 1]
-  const q_norm = quality / 5.0;                                    // 1-5 → 0-1
-  const c_norm = Math.max(0, 1 - Math.log1p(cost) / 5);          // Cost penalty
-  const l_norm = Math.max(0, 1 - Math.log1p(latency) / 10000);  // Latency penalty
-  const r_norm = correct ? 1.0 : 0.0;                             // Correctness
-  
-  // Weighted sum (RouterArena-style)
-  return 0.4 * q_norm + 0.2 * c_norm + 0.1 * l_norm + 0.3 * r_norm;
-}
-```
-
----
-
-## 5. Expected Improvement
-
-### 5.1 RouterArena Score Projection
-
-| Change | Current Score | Expected New Score | Source |
-|--------|---------------|-------------------|--------|
-| Embedding-based routing | 70.32 | 73-75 | Semantic similarity improvement |
-| Cost-aware loss | 70.32 | 72-74 | Better cost-quality tradeoff |
-| Contrastive fine-tuning | 70.32 | 75-78 | Learned query-model alignment |
-| All combined | 70.32 | **77-80** | End-to-end improvement |
-
-### 5.2 Breakdown by RouterArena Component
-
-| Component | Weight | Current | With Loss Functions | Improvement |
-|-----------|--------|---------|-------------------|-------------|
-| Accuracy (±1 tier) | 60% | ~85% | ~90% | +5 pts |
-| Cost Efficiency | 20% | ~60% | ~75% | +15 pts |
-| Latency | 20% | ~70% | ~75% | +5 pts |
-| **Total** | 100% | **70.32** | **~76-78** | **+6-8 pts** |
-
-### 5.3 Conservative Estimate
-
-Even without full ML training, adding:
-- **Log-scale cost penalty** → +2 RouterArena points
-- **Embedding cache** → +1 point (better semantic matching)
-- **Contextual bandit updates** → +2 points (faster online learning)
-
-**Conservative target: 73-74 RouterArena score**
-
----
-
-## 6. Implementation Priority
-
-| Priority | Change | Complexity | Impact | Est. Time |
-|----------|--------|------------|--------|-----------|
-| P0 | Log-scale cost penalty | Low | Medium | 1 day |
-| P1 | Embedding cache (MiniLM) | Medium | High | 2 days |
-| P2 | Contextual bandit updates | Medium | High | 3 days |
-| P3 | Contrastive fine-tuning | High | Very High | 1 week |
-
----
-
-## 7. References
-
-1. **RouteLLM** - LMSYS/Anyscale, arXiv:2404.06035
-   - Learned routing from pairwise preferences
-   - BERT classifier with cross-entropy loss
-
-2. **RouterArena** - Berkeley, arXiv:2510.00202
-   - 8,400 queries, 19 routers evaluated
-   - Composite scoring: accuracy (60%), cost (20%), latency (20%)
-
-3. **LLMRouterBench** - ACL 2024
-   - 400K+ instances, 9 domains
-   - TF-IDF baseline: 62.3%, Neural: 78.1%
-
-4. **Self-Consistency** - Wang et al., ICLR 2023
-   - Multiple reasoning paths improve GSM8K by +17.9 points
-   - Relevant to A3M's ensemble voting
-
-5. **Deep Ensembles** - Lakshminarayanan et al., NeurIPS 2017
-   - Confidence-weighted ensembles reduce error by 10-30%
-   - Foundation for A3M's voting mechanism
-
----
-
-## Appendix: Quick Wins
-
-### Quick Win 1: Immediate Cost Penalty Fix
-
-In `advancedRouter.ts`, replace:
-
-```typescript
-// CURRENT (linear)
-const avg_cost = (model.cost_per_1k_input + model.cost_per_1k_output) / 2;
-return (1 - Math.min(avg_cost / 10, 1)) * 0.6;
-```
-
-With:
-
-```typescript
-// PROPOSED (log-scale)
-const avg_cost = (model.cost_per_1k_input + model.cost_per_1k_output) / 2;
-return Math.max(0, 1 - Math.log1p(avg_cost) / Math.log1p(10)) * 0.6;
-```
-
-**Effect:** Makes router less aggressive about ultra-cheap models, better cost-quality tradeoff.
-
-### Quick Win 2: Latency in Routing Score
-
-Add latency penalty to scoring:
-
-```typescript
-const latencyPenalty = Math.max(0, 1 - model.latency_ms / 10000);
-const qualityScore = scoreModelFit(profile, features);
-const costScore = costEfficiency(profile, features);
-
-return 0.5 * qualityScore + 0.3 * costScore + 0.2 * latencyPenalty;
-```
-
-**Effect:** RouterArena scores improve on latency component (+2-3 points).
-
----
-
-*Generated: 2026-06-03 | For A3M Router v2.2+*

package/research-log.md

@@ -1,49 +0,0 @@
-# A3M Router Research Log
-
-## 2026-06-03 - Test Coverage Analysis
-
-### Research State
-```
-Project: A3M Router Test Coverage Analysis
-Date: 2026-06-03
-Agents: 3 (Architecture, Performance, Test Coverage)
-Goal: Identify top 3 improvements via council vote
-```
-
-### Scope Explored
-- `test/` - 7 legacy JS test files (budgetEnforcer, observability, providerHealth, providerRetry, semanticCache)
-- `tests/` - Vitest test suite (routing/ensembleVoting, routing/providerRetry, routing/queryTypePresets, memory/episodicMemory)
-- `test-council/` - 5 test files (structure, edge-case, performance, integration, agent-council-eval)
-
-### Key Source Files Analyzed
-- `src/ensemble.ts` - EnsembleOrchestrator (no tests)
-- `src/sdk.ts` - A3MRouter SDK (structure only)
-- `src/cost/budgetEnforcer.ts` - Budget enforcement (legacy test)
-- `src/analytics/costAnalytics.ts` - Cost analytics (no tests)
-- `src/security/guardrails.ts` - GuardrailEngine (NO TESTS - CRITICAL)
-- `src/observability/middleware.ts` - Express middleware (not tested)
-- `src/routing/crossModelValidation.ts` - Cross-model validation (not tested)
-- `src/observability/fatigueDetector.ts` - Fatigue detection (not tested)
-
-### Coverage Summary
-| Module | Coverage | Status |
-|--------|----------|--------|
-| Routing | Partial | ensembleVoting, providerRetry, queryTypePresets |
-| Memory | Good | episodicMemory well tested |
-| Observability | Partial | Tracer, MetricsCollector tested; middleware not |
-| Security | NONE | GuardrailEngine untested |
-| Cost | Partial | budgetEnforcer legacy test; costAnalytics untested |
-| SDK | Structure only | No behavioral tests |
-
-### Critical Gaps Identified
-1. **GuardrailEngine** - Zero tests for security-critical code
-2. **EnsembleOrchestrator** - Core P0 feature lacks integration tests
-3. **CostAnalytics** - No tests for savings calculation accuracy
-4. **SDK Class** - Only type checking, no behavioral tests
-5. **Middleware** - Not tested
-
-### Output
-Created: `council-votes/coverage-vote.md`
-Vote: Finding #1 (GuardrailEngine) as highest priority
-
----

package/scripts/banner.js

@@ -1,29 +0,0 @@
-#!/usr/bin/env node
-/**
- * A3M Router — Terminal ASCII Art Banner
- *
- * Printed on CLI startup to reinforce A3M branding.
- * Usage:
- *   node scripts/banner.js
- *   // or import './banner' in CLI entry point
- */
-
-const A3M_BANNER = `
-╔══════════════════════════════════════════════════════════╗
-║                     ╔═╗╔═╗╔╗╔╔═╗                        ║
-║                     ╠═╣║ ║║║║║ ║                        ║
-║                     ╩ ╩╚═╝╝╚╝╚═╝                        ║
-║                                                          ║
-║            Parallel Multi-LLM Execution Engine           ║
-║                                                          ║
-║  47+ Providers  ·  Ensemble Voting  ·  62% Cost Savings  ║
-║                                                          ║
-║  ${'\x1b[2m'}https://github.com/Das-rebel/a3m-router${'\x1b[0m'}${' '.repeat(19)}║
-╚══════════════════════════════════════════════════════════╝
-`;
-
-module.exports = A3M_BANNER;
-
-if (require.main === module) {
-  process.stdout.write(A3M_BANNER);
-}