adaptive-memory-multi-model-router 2.14.46 → 2.14.47

package/submissions/benchmarks/ROUTERARENA_UPDATE.md

@@ -1,83 +0,0 @@
-# RouterArena Update Request - A3M Router v2.14.41
-
-## Summary
-
-A3M Router requests re-evaluation with version **2.14.41**, featuring enhanced game-theoretic ensemble credit assignment and multi-round dialog optimization.
-
-**NPM:** `npm install adaptive-memory-multi-model-router@2.14.41`
-
-**PR:** https://github.com/Das-rebel/router-arena (submitting)
-
----
-
-## Improvements Since v2.14.23
-
-### Performance Metrics
-
-| Metric | v2.14.23 | v2.14.41 | Change |
-|--------|----------|----------|--------|
-| Exact Tier Accuracy | 67% | **67%** | - |
-| ±1 Tier Accuracy | 96% | **96%** | - |
-| Cost Savings | 62.9% | **63.5%** | +0.6pp |
-| Robustness Score | 0.8524 | **0.86** | +0.0076 |
-| Ensemble Efficiency | baseline | **+12%** | NEW |
-
-### Key Algorithm Improvements
-
-#### 1. Enhanced Shapley Value (v2.14.41)
-- **Ethnocentrism** - Loyalty matrix tracks model collaboration success
-- **Handicap Principle** - Costly signaling for honest quality indication
-- Formula: φ_i* = 0.5·Shapley + 0.3·Ethnocentrism + 0.2·Handicap
-
-#### 2. Multi-Round Dialog Optimization (v2.14.41)
-- Conversation state management across turns
-- Topic tracking with complexity scoring
-- Model performance history per topic
-- Optimized context building
-
-#### 3. Game-Theoretic Routing (Wolfram's Ruliology)
-- Provider strategies: aggressive/balanced/conservative
-- Query risk profiles: HIGH/MEDIUM/LOW
-- Risk-strategy matching
-
----
-
-## Code Changes
-
-### New Files
-- `src/ensemble/shapleyValue.ts` - Enhanced Shapley with loyalty + handicap
-- `src/ensemble/multiRoundDialog.ts` - Multi-round dialog optimizer
-
-### Updated Files
-- `src/ensemble.ts` - Integrated enhanced Shapley + dialog
-- `src/index.ts` - Added new exports
-
----
-
-## Ensemble Credit Comparison
-
-| Model | Base Shapley | +Ethnocentrism | +Handicap | Combined |
-|-------|-------------|----------------|-----------|----------|
-| DeepSeek | 1.000 | 0.090 | 0.378 | 0.522 |
-| Mistral | 1.000 | 0.000 | 0.263 | 0.478 |
-| NVIDIA | 0.000 | 0.000 | 0.000 | 0.000 |
-
----
-
-## Testing Results
-
-```
-✅ All 28 unit tests passed
-✅ Shapley + Ethnocentrism + Handicap working correctly
-✅ Multi-round dialog optimizer working correctly
-```
-
----
-
-## Submission
-
-```bash
-npm install adaptive-memory-multi-model-router@2.14.41
-```
-
-Full source: https://github.com/Das-rebel/adaptive-memory-multi-model-router

package/submissions/benchmarks/ROUTERBENCH_SUBMISSION.md

@@ -1,225 +0,0 @@
-# RouterBench Submission - A3M Router
-
-## Comprehensive LLM Routing Benchmark
-
-**Repository:** https://github.com/withmartian/routerbench
-**Stars:** 165 | **Status:** Active benchmark
-**Submission Date:** 2026-06-04
-**Version:** 2.14.23
-
----
-
-## Summary
-
-A3M Router is an open-source adaptive LLM gateway with **parallel multi-LLM execution** and **confidence-weighted ensemble voting**. Unlike sequential fallback routers, A3M executes multiple providers simultaneously and merges results for better accuracy and robustness.
-
-**NPM:** `npm install adaptive-memory-multi-model-router@2.14.23`
-
----
-
-## Core Innovation: Parallel Ensemble
-
-### Why Parallel Matters
-
-All existing routers use **sequential fallback**:
-```
-litellm:     try gpt-4 → if fails try gpt-3.5 → if fails try claude
-one-api:     try openai → try anthropic → try deepseek
-gpt-researcher: try single provider per query
-```
-
-A3M uses **parallel execution + ensemble voting**:
-```
-A3M Router:  [gpt-4, claude, gemini] in parallel → confidence-weighted vote → best response
-```
-
-**Result:** Higher accuracy through diversity, not higher cost through retries.
-
----
-
-## Benchmark Results
-
-| Metric | Value | Notes |
-|--------|-------|-------|
-| **Exact Tier Accuracy** | 67% | Target >50% ✅ |
-| **±1 Tier Accuracy** | 96% | Target >85% ✅ |
-| **Cost Savings** | 62.9% | vs all-premium baseline |
-| **Robustness Score** | 0.8524 | **Highest among all routers** |
-| **Premium Accuracy** | 57.5% | Complex queries |
-| **Free Tier Accuracy** | 96% | Simple queries |
-| **Over-routing** | 6.5% | Very low |
-| **Under-routing** | 26.5% | Room for improvement |
-
-### Performance Comparison
-
-| Router | Exact Tier | ±1 Tier | Robustness | Cost Savings |
-|--------|-----------|---------|------------|-------------|
-| **A3M Router** | **67%** | **96%** | **0.8524** | **62.9%** |
-| litellm | 62% | 91% | 0.7891 | 48.2% |
-| one-api | 58% | 87% | 0.7512 | 52.1% |
-| direct-api | 70% | 98% | N/A | 0% |
-
----
-
-## Key Features
-
-### 1. Parallel Multi-LLM Execution
-- Execute 2-5 providers simultaneously
-- Confidence-weighted voting for response selection
-- Configurable ensemble strategies
-
-### 2. Memory-Enhanced Routing
-```typescript
-// A3M learns from routing history
-const router = new A3MRouter({
-  memory: {
-    enabled: true,
-    persistence: '.memory.json'
-  }
-});
-```
-
-### 3. Cost Optimization
-- **62.9% savings** vs all-premium
-- Auto-route to cheapest adequate model
-- Per-query cost tracking
-- Budget enforcement
-
-### 4. Provider Reliability
-- Circuit breaker pattern
-- Automatic failover
-- Health monitoring
-- **Robustness: 0.8524** (highest)
-
-### 5. 47+ Providers Supported
-OpenAI, Anthropic, Google Gemini, Groq, Cerebras, Mistral, DeepSeek, and 40+ more via unified API.
-
----
-
-## Research-Backed Design
-
-### Complexity Signal Analysis
-5 validated signals for query difficulty assessment:
-- Jargon Density (+15%)
-- Task Formality (+10%)
-- Depth Markers (+8%)
-- Stakes Language (+5%)
-- Multi-Step Structure (+5%)
-
-### Mathematical Optimization
-- **Thompson Sampling** - Bayesian exploration/exploitation
-- **UCB1 Bandits** - Optimal exploration bounds
-- **Pareto Optimization** - Multi-objective routing
-- **Log-scale Cost Penalty** - Better cost-accuracy tradeoff
-
----
-
-## How to Evaluate
-
-### Install A3M Router
-```bash
-npm install adaptive-memory-multi-model-router@2.14.23
-```
-
-### Run RouterBench Evaluation
-```bash
-# Clone routerbench
-git clone https://github.com/withmartian/routerbench.git
-cd routerbench
-
-# Install A3M
-npm install adaptive-memory-multi-model-router@2.14.23
-
-# Run benchmark
-npm run benchmark -- --router a3m
-
-# Or compare all routers
-npm run compare -- --routers a3m,litellm,one-api
-```
-
-### Programmatic Evaluation
-```javascript
-const { A3MRouter } = require('adaptive-memory-multi-model-router');
-
-async function evaluate() {
-  const router = new A3MRouter({
-    parallel: true,
-    ensemble: true,
-    maxProviders: 3
-  });
-
-  const queries = require('./benchmark/queries.json');
-  const results = [];
-
-  for (const query of queries) {
-    const decision = await router.route(query.text);
-    results.push({
-      query: query.text,
-      selected: decision.provider,
-      tier: decision.tier,
-      confidence: decision.confidence
-    });
-  }
-
-  // Calculate metrics
-  const exactTier = results.filter(r => r.tier === query.tier).length / results.length;
-  const withinOneTier = results.filter(r => Math.abs(r.tier - query.tier) <= 1).length / results.length;
-  
-  console.log({ exactTier, withinOneTier });
-}
-
-evaluate();
-```
-
----
-
-## Unique Value Proposition
-
-| Feature | Sequential Fallback (litellm, one-api) | A3M Parallel Ensemble |
-|---------|----------------------------------------|----------------------|
-| Execution | One at a time | Multiple simultaneously |
-| Accuracy | Single provider | Ensemble voting |
-| Latency | Retry overhead | Parallel execution |
-| Robustness | Failover dependent | Built-in redundancy |
-| Cost | Retry costs add up | Fixed ensemble cost |
-
----
-
-## Submission Package
-
-```
-submissions/
-├── A3M_ROUTER.md          # This file
-├── README.md               # Setup instructions
-├── eval/
-│   ├── run_eval.js         # Evaluation script
-│   └── results.jsonl       # Results
-└── package.json            # npm package info
-```
-
----
-
-## Verification
-
-```bash
-# 1. Install
-npm install adaptive-memory-multi-model-router@2.14.23
-
-# 2. Verify installation
-npx a3m-router --version
-
-# 3. Run self-test
-npx a3m-router test
-
-# 4. Run evaluation
-node eval/run_eval.js > results.jsonl
-```
-
----
-
-## Contact
-
-- **GitHub:** https://github.com/Das-rebel/a3m-router
-- **NPM:** https://www.npmjs.com/package/adaptive-memory-multi-model-router
-- **Documentation:** https://das-rebel.github.io/a3m-router/
-- **Benchmark Results:** `eval/results.jsonl`

package/test-council/1-structure-tests.test.js

@@ -1,353 +0,0 @@
-/**
- * Structure Tests - Code Structure & Export Coverage
- * 
- * Tests that verify all exported functions, classes, and types work correctly.
- * This file provides comprehensive coverage of the public API surface.
- */
-
-import { describe, it, expect, vi, beforeEach } from 'vitest';
-
-// Import all exports from the main module
-const {
-  // Routing engine
-  routeQuery,
-  routeBatch,
-  recommendForTask,
-  extractQueryFeatures,
-  MODEL_PROFILES,
-  updateModelProfile,
-  getProviderHealth,
-  
-  // Retry handling
-  ProviderRetryHandler,
-  createRetryHandler,
-  getDefaultRetryHandler,
-  DEFAULT_RETRY_CONFIG,
-  DEFAULT_PROVIDER_CONFIG,
-  PROVIDER_CONTEXT_LIMITS,
-  
-  // Providers
-  DEFAULT_PROVIDERS,
-  getAvailableProviders,
-  registerProvider,
-  deregisterProvider,
-  updateProvider,
-  healthCheck,
-  checkAllProviders,
-  findCheapestAvailableProvider,
-  findFastestAvailableProvider,
-  loadConfig,
-  saveConfig,
-  
-  // Cost tracking
-  CostTracker,
-  BudgetEnforcer,
-  BudgetExceededError,
-  createBudgetEnforcer,
-  
-  // Memory
-  MemoryTree,
-  
-  // Utilities
-  countTokens,
-  estimateTokens,
-  
-  // Cache
-  SemanticCache,
-  
-  // Security
-  GuardrailEngine,
-  
-  // Analytics
-  CostAnalytics,
-  
-  // Observability
-  Tracer,
-  getTracer,
-  createTracer,
-  MetricsCollector,
-  getMetrics,
-  createMetricsCollector,
-  observabilityMiddleware,
-  observabilityPlugin,
-  budgetAlertMiddleware,
-  
-  // Ensemble
-  HALOOrchestrator,
-  
-  // Factory
-  createA3MRouter,
-} = require('../dist/index.js');
-
-// ============================================================
-// STRUCTURE TESTS
-// ============================================================
-
-describe('1. Structure - Routing Engine Exports', () => {
-  
-  describe('routeQuery', () => {
-    it('is a function', () => {
-      expect(typeof routeQuery).toBe('function');
-    });
-    
-    it('returns object with primary_model', () => {
-      const result = routeQuery('test query');
-      expect(result).toHaveProperty('primary_model');
-    });
-    
-    it('returns object with fallback_models array', () => {
-      const result = routeQuery('test query');
-      expect(result).toHaveProperty('fallback_models');
-      expect(Array.isArray(result.fallback_models)).toBe(true);
-    });
-    
-    it('returns object with estimated_cost number', () => {
-      const result = routeQuery('test query');
-      expect(result).toHaveProperty('estimated_cost');
-      expect(typeof result.estimated_cost).toBe('number');
-    });
-    
-    it('returns object with confidence number', () => {
-      const result = routeQuery('test query');
-      expect(result).toHaveProperty('confidence');
-      expect(typeof result.confidence).toBe('number');
-    });
-  });
-  
-  describe('routeBatch', () => {
-    it('is a function', () => {
-      expect(typeof routeBatch).toBe('function');
-    });
-    
-    it('returns array of same length as input', () => {
-      const queries = ['a', 'b', 'c'];
-      const results = routeBatch(queries);
-      expect(results.length).toBe(queries.length);
-    });
-  });
-  
-  describe('recommendForTask', () => {
-    it('is a function', () => {
-      expect(typeof recommendForTask).toBe('function');
-    });
-    
-    it('returns object with primary field', () => {
-      const result = recommendForTask('coding');
-      expect(result).toHaveProperty('primary');
-    });
-  });
-  
-  describe('extractQueryFeatures', () => {
-    it('is a function', () => {
-      expect(typeof extractQueryFeatures).toBe('function');
-    });
-    
-    it('returns object', () => {
-      const features = extractQueryFeatures('test');
-      expect(typeof features).toBe('object');
-    });
-  });
-  
-  describe('MODEL_PROFILES', () => {
-    it('is an object', () => {
-      expect(typeof MODEL_PROFILES).toBe('object');
-    });
-    
-    it('has at least one model', () => {
-      expect(Object.keys(MODEL_PROFILES).length).toBeGreaterThan(0);
-    });
-  });
-});
-
-describe('2. Structure - Provider Configuration Exports', () => {
-  
-  describe('DEFAULT_PROVIDERS', () => {
-    it('is an object', () => {
-      expect(typeof DEFAULT_PROVIDERS).toBe('object');
-    });
-  });
-  
-  describe('getAvailableProviders', () => {
-    it('is a function', () => {
-      expect(typeof getAvailableProviders).toBe('function');
-    });
-    
-    it('returns an object', () => {
-      const providers = getAvailableProviders();
-      expect(typeof providers).toBe('object');
-    });
-  });
-  
-  describe('registerProvider', () => {
-    it('is a function', () => {
-      expect(typeof registerProvider).toBe('function');
-    });
-  });
-  
-  describe('deregisterProvider', () => {
-    it('is a function', () => {
-      expect(typeof deregisterProvider).toBe('function');
-    });
-  });
-  
-  describe('updateProvider', () => {
-    it('is a function', () => {
-      expect(typeof updateProvider).toBe('function');
-    });
-  });
-  
-  describe('healthCheck', () => {
-    it('is a function', () => {
-      expect(typeof healthCheck).toBe('function');
-    });
-  });
-});
-
-describe('3. Structure - Retry Handler Exports', () => {
-  
-  describe('ProviderRetryHandler', () => {
-    it('is a class', () => {
-      expect(typeof ProviderRetryHandler).toBe('function');
-    });
-    
-    it('can be instantiated', () => {
-      const handler = new ProviderRetryHandler();
-      expect(handler).toBeInstanceOf(ProviderRetryHandler);
-    });
-    
-    it('has getConfig method', () => {
-      const handler = new ProviderRetryHandler();
-      expect(typeof handler.getConfig).toBe('function');
-    });
-    
-    it('has isRetryableError method', () => {
-      const handler = new ProviderRetryHandler();
-      expect(typeof handler.isRetryableError).toBe('function');
-    });
-  });
-  
-  describe('createRetryHandler', () => {
-    it('is a function', () => {
-      expect(typeof createRetryHandler).toBe('function');
-    });
-  });
-  
-  describe('DEFAULT_RETRY_CONFIG', () => {
-    it('is an object', () => {
-      expect(typeof DEFAULT_RETRY_CONFIG).toBe('object');
-    });
-  });
-  
-  describe('PROVIDER_CONTEXT_LIMITS', () => {
-    it('is an object', () => {
-      expect(typeof PROVIDER_CONTEXT_LIMITS).toBe('object');
-    });
-  });
-});
-
-describe('4. Structure - Cost Tracking Exports', () => {
-  
-  describe('CostTracker', () => {
-    it('is a class', () => {
-      expect(typeof CostTracker).toBe('function');
-    });
-    
-    it('can be instantiated', () => {
-      const tracker = new CostTracker();
-      expect(tracker).toBeTruthy();
-    });
-  });
-  
-  describe('BudgetEnforcer', () => {
-    it('is a class', () => {
-      expect(typeof BudgetEnforcer).toBe('function');
-    });
-  });
-  
-  describe('BudgetExceededError', () => {
-    it('is an Error class', () => {
-      expect(typeof BudgetExceededError).toBe('function');
-      const error = new BudgetExceededError('test');
-      expect(error).toBeInstanceOf(Error);
-    });
-  });
-});
-
-describe('5. Structure - Memory Exports', () => {
-  
-  describe('MemoryTree', () => {
-    it('is a class', () => {
-      expect(typeof MemoryTree).toBe('function');
-    });
-    
-    it('can be instantiated', () => {
-      const memory = new MemoryTree({ maxSize: 100 });
-      expect(memory).toBeTruthy();
-    });
-    
-    it('has add method', () => {
-      const memory = new MemoryTree({ maxSize: 100 });
-      expect(typeof memory.add).toBe('function');
-    });
-    
-    it('has search method', () => {
-      const memory = new MemoryTree({ maxSize: 100 });
-      expect(typeof memory.search).toBe('function');
-    });
-    
-    it('has getStats method', () => {
-      const memory = new MemoryTree({ maxSize: 100 });
-      expect(typeof memory.getStats).toBe('function');
-    });
-  });
-});
-
-describe('6. Structure - Utility Exports', () => {
-  
-  describe('countTokens', () => {
-    it('is a function', () => {
-      expect(typeof countTokens).toBe('function');
-    });
-    
-    it('returns positive number for non-empty string', () => {
-      const tokens = countTokens('hello world');
-      expect(typeof tokens).toBe('number');
-      expect(tokens).toBeGreaterThan(0);
-    });
-  });
-  
-  describe('estimateTokens', () => {
-    it('is a function', () => {
-      expect(typeof estimateTokens).toBe('function');
-    });
-  });
-});
-
-describe('7. Structure - Factory Exports', () => {
-  
-  describe('createA3MRouter', () => {
-    it('is a function', () => {
-      expect(typeof createA3MRouter).toBe('function');
-    });
-    
-    it('returns an object', () => {
-      const router = createA3MRouter({});
-      expect(typeof router).toBe('object');
-    });
-    
-    it('returns object with route function', () => {
-      const router = createA3MRouter({});
-      expect(typeof router.route).toBe('function');
-    });
-    
-    it('returns object with costTracker', () => {
-      const router = createA3MRouter({});
-      expect(router.costTracker).toBeTruthy();
-    });
-    
-    it('returns object with memoryTree', () => {
-      const router = createA3MRouter({});
-      expect(router.memoryTree).toBeTruthy();
-    });
-  });
-});