adaptive-memory-multi-model-router 2.14.46 → 2.14.47

package/tmlpd-pi-extension/skill/SKILL.md

@@ -1,238 +0,0 @@
----
-name: tmlpd
-description: Research-backed Multi-LLM Router with parallel execution, streaming, caching, token compression (ISON), local provider support (Ollama/vLLM/LM Studio), batch processing. Based on arXiv research: RouteLLM routing, RadixAttention prefix caching, Medusa/EAGLE speculative decoding. Python bindings for LangChain/LlamaIndex/AutoGen/CrewAI. 120+ keywords for LLM/ML discoverability. Use for multi-model comparison, cost optimization, batch processing, local privacy, context compression, adaptive routing.
----
-
-# TMLPD PI Extension
-
-**Research-backed Multi-LLM Router** with advanced optimization features.
-
-## Direct Imports (TypeScript)
-
-```typescript
-import {
-  createTMLPD,           // Core instance
-  HALOOrchestrator,      // Hierarchical orchestration
-  EpisodicMemoryStore,    // Learn from past tasks
-  // Advanced Routing (RouteLLM-style)
-  routeQuery,            // Learned routing decision
-  routeBatch,            // Batch routing
-  extractQueryFeatures,   // Feature extraction
-  MODEL_PROFILES,         // Model cost/quality profiles
-  // Prefix Cache (RadixAttention-style)
-  PrefixCache,           // 5-10x speedup for shared prompts
-  createWarmedCache,     // Pre-warmed cache
-  // Speculative Decoding (Medusa/EAGLE)
-  SpeculativeDecoder,     // 2-3x faster generation
-  estimateSpeedupPotential,
-  // Compression
-  isonEncode,            // 20-40% token reduction
-  truncateMessages,      // Context window management
-  // Local providers
-  createOllamaProvider,  // Ollama
-  createVLLMProvider,    // vLLM
-  // Batch processing
-  BatchProcessor,        // Priority queuing
-  TMLPD_PI_TOOLS         // 13 PI tool definitions
-} from "tmlpd-pi";
-```
-
-## Direct Imports (Python)
-
-```python
-from tmlpd import (
-    TMLPDLite,       # Lite client (sync, no deps)
-    TMLPDClient,     # Async production client
-    TaskType,        # CODING, FAST, PREMIUM, etc.
-    quick_process    # One-liner function
-)
-```
-
-## 13 PI Tools
-
-| Tool | Input | Output |
-|------|-------|--------|
-| `tmlpd_execute` | `{prompt, models?}` | `{content, model, cost}` |
-| `tmlpd_execute_single` | `{prompt, model?}` | `{content, model}` |
-| `tmlpd_cost_summary` | `{}` | `{total_cost, by_provider}` |
-| `tmlpd_cache_stats` | `{}` | `{hits, misses, hit_rate}` |
-| `tmlpd_provider_status` | `{}` | `{ready_providers}` |
-| `tmlpd_invalidate_cache` | `{model?}` | `{invalidated}` |
-| `tmlpd_get_budget` | `{}` | `{daily, monthly}` |
-| `tmlpd_halo_execute` | `{task, max_concurrent?}` | `{success, results}` |
-| `tmlpd_episodic_query` | `{task, limit?}` | `EpisodicEntry[]` |
-| `tmlpd_count_tokens` | `{text, model?}` | `{tokens}` |
-| `tmlpd_compress_context` | `{messages, strategy?}` | `{compressed, ratio}` |
-| `tmlpd_local_generate` | `{prompt, runtime, model?}` | `{content, cost:0}` |
-| `tmlpd_batch_execute` | `{prompts, concurrency?}` | `BatchResult[]` |
-
-## Research-Backed Features (arXiv)
-
-### RouteLLM-Style Learned Routing (arXiv:2404.06035)
-
-```typescript
-// Automatic cost-quality tradeoff routing
-const decision = routeQuery('Write a Python async function');
-// Returns: { primary_model, fallback_models, confidence, reasoning }
-
-const features = extractQueryFeatures(prompt);
-// Extracts: complexity, has_code, has_math, is_multilingual, etc.
-
-// MODEL_PROFILES contains cost/latency/quality for each provider
-console.log(MODEL_PROFILES['openai/gpt-4o'].quality_score);  // 0.95
-```
-
-| Model | Quality | Latency | Best For |
-|-------|---------|---------|----------|
-| gpt-4o | 0.95 | 2000ms | reasoning |
-| gpt-4o-mini | 0.85 | 500ms | fast |
-| claude-3.5-sonnet | 0.96 | 2500ms | creative |
-| gemini-2.0-flash | 0.88 | 800ms | multilingual |
-| groq/llama-3.3-70b | 0.82 | 400ms | fast/budget |
-
-### RadixAttention-Style Prefix Caching (arXiv:2312.07104)
-
-```typescript
-// 5-10x speedup for shared system prompts
-const cache = new PrefixCache({ max_entries: 10000 });
-cache.warmup([
-  "You are a helpful assistant.",
-  "You are a coding assistant.",
-  "Analyze the following code..."
-]);
-
-// Automatic prefix matching
-const result = cache.lookup("You are a helpful assistant. Please explain...");
-// Returns cached if prefix matches
-
-const stats = cache.getStats();
-// { total_entries, hit_rate, memory_estimate_mb }
-```
-
-### Medusa/EAGLE Speculative Decoding (arXiv:2401.10774)
-
-```typescript
-// 2-3x faster generation with same quality
-const decoder = new SpeculativeDecoder();
-const result = await decoder.decode(
-  prompt,
-  fastModelFn,    // Draft model
-  slowModelFn,    // Target model
-  5               // Max draft tokens
-);
-// { accepted, rejected, speedup, final_text }
-
-const speedup = estimateSpeedupPotential(100, 200, 50, 200);
-// Returns estimated speedup (capped at 3x)
-```
-
-## Token Utilities
-
-```typescript
-// Count tokens (no API call)
-const tokens = countTokens("Your prompt", "claude-3.5-sonnet");
-
-// Estimate cost before execution
-const cost = estimateCost(500, 200, "gpt-4o");  // $0.0095
-
-// Find cheapest models for task
-const cheap = findCheapestModels("fast", 3);
-```
-
-## ISON Compression (20-40% token reduction)
-
-```typescript
-// Remove articles, normalize whitespace
-const encoded = isonEncode("The quick brown fox jumps over the lazy dog");
-// "quick brown fox jumps lazy dog"
-
-// Truncate long conversations
-const truncated = truncateMessages(messages, 4000, "smart");
-```
-
-## Local LLM Support
-
-```typescript
-// Zero cost, privacy-preserving
-const ollama = createOllamaProvider("llama-3.3-70b");
-const vllm = createVLLMProvider("http://localhost:8000");
-
-// Parallel across local + cloud
-const results = await manager.executeParallel("Prompt", {
-  models: ["ollama/llama-3.3-70b", "openai/gpt-4o"]
-});
-```
-
-## Batch Processing
-
-```typescript
-const batch = new BatchProcessor({ concurrency: 5 });
-batch.add({ prompt: "Task 1", priority: "high" });
-batch.add({ prompt: "Task 2", priority: "normal" });
-batch.onProgress((progress, result) => {
-  console.log(`Completed: ${progress.completed}/${progress.total}`);
-});
-await batch.execute(executor);
-```
-
-## Python Task Routing
-
-```python
-from tmlpd import TMLPDLite, TaskType
-
-lite = TMLPDLite()
-task = lite.classify_task("Write Python async function")
-# TaskType.CODING
-
-models = lite.get_optimal_models(task, 3)
-# ["codex", "claude-minimax", "claude"]
-```
-
-| TaskType | Keywords | Models |
-|----------|----------|--------|
-| CODING | python, javascript, code | codex, claude-minimax |
-| FRONTEND | react, vue, component | codex, claude-minimax |
-| CHINESE | 中文, 汉语 | claude-glm, claude-minimax |
-| FAST | quick, simple | gemini, claude-haiku |
-
-## Framework Integrations
-
-```python
-# LangChain
-class TMLPDLLM(BaseLLM):
-    def _call(self, prompt): return lite.process(prompt)["content"]
-
-# LlamaIndex
-class TMLPDLLM(LLM):
-    def complete(self, prompt): return lite.process(prompt)["content"]
-
-# AutoGen
-class TMLPDAgent(AssistantAgent):
-    def generate_reply(self, messages):
-        return lite.process(messages[-1]["content"])["content"]
-```
-
-## 120+ Keywords for Discoverability
-
-```
-routellm, prefix-caching, radix-attention, speculative-decoding, medusa, eagle,
-flashattention, pagedattention, kv-cache-quantization, llmlingua, streamingllm,
-tensor-parallelism, continuous-batching, multi-model-orchestration,
-multi-agent-debate, self-consistency, adaptive-router, intelligent-router,
-context-aware-router, task-aware-router, memory-augmented-llm,
-episodic-memory-router, semantic-memory-router, arxiv, research-backed,
-icml, neurips, iclr, token-compression, context-compression
-```
-
-## npm
-
-**Package:** https://npmjs.com/package/tmlpd-pi  
-**Version:** 1.2.0 | **Files:** 94 | **Size:** 543KB unpacked
-
-## Reference
-
-- RouteLLM: arXiv:2404.06035
-- RadixAttention: arXiv:2312.07104
-- Medusa: arXiv:2401.10774
-- FlashAttention: arXiv:2304.05195
-- PagedAttention: SOSP 2023

package/tmlpd-pi-extension/src/cache/responseCache.ts

@@ -1,147 +0,0 @@
-/**
- * TMLPD Response Cache
- * 
- * Caches LLM responses to avoid redundant API calls.
- * Uses content hash for cache key and supports TTL.
- */
-
-import * as crypto from "crypto";
-
-export interface CacheEntry {
-  content: string;
-  model: string;
-  provider: string;
-  tokens: number;
-  cost: number;
-  cached_at: number;
-  expires_at: number;
-}
-
-export interface CacheConfig {
-  enabled: boolean;
-  ttl_seconds: number;
-  max_entries: number;
-  cache_dir?: string;
-}
-
-export class ResponseCache {
-  private cache: Map<string, CacheEntry> = new Map();
-  private config: CacheConfig;
-  private hits = 0;
-  private misses = 0;
-
-  constructor(config: Partial<CacheConfig> = {}) {
-    this.config = {
-      enabled: config.enabled ?? true,
-      ttl_seconds: config.ttl_seconds ?? 3600, // 1 hour default
-      max_entries: config.max_entries ?? 1000,
-      cache_dir: config.cache_dir,
-    };
-  }
-
-  /**
-   * Generate cache key from prompt + model
-   */
-  generateKey(prompt: string, model: string): string {
-    const hash = crypto.createHash("sha256");
-    hash.update(prompt + "|" + model);
-    return hash.digest("hex").substring(0, 32);
-  }
-
-  /**
-   * Get cached response if available and not expired
-   */
-  get(prompt: string, model: string): CacheEntry | null {
-    if (!this.config.enabled) return null;
-
-    const key = this.generateKey(prompt, model);
-    const entry = this.cache.get(key);
-
-    if (!entry) {
-      this.misses++;
-      return null;
-    }
-
-    // Check expiration
-    if (Date.now() > entry.expires_at) {
-      this.cache.delete(key);
-      this.misses++;
-      return null;
-    }
-
-    this.hits++;
-    return entry;
-  }
-
-  /**
-   * Store response in cache
-   */
-  set(prompt: string, model: string, response: Partial<CacheEntry>): void {
-    if (!this.config.enabled) return;
-
-    const key = this.generateKey(prompt, model);
-    const now = Date.now();
-
-    // Evict oldest if at capacity
-    if (this.cache.size >= this.config.max_entries) {
-      this.evictOldest();
-    }
-
-    this.cache.set(key, {
-      content: response.content ?? "",
-      model: response.model ?? model,
-      provider: response.provider ?? "unknown",
-      tokens: response.tokens ?? 0,
-      cost: response.cost ?? 0,
-      cached_at: now,
-      expires_at: now + this.config.ttl_seconds * 1000,
-    });
-  }
-
-  /**
-   * Invalidate cache for specific model or all
-   */
-  invalidate(model?: string): void {
-    if (model) {
-      for (const [key, entry] of this.cache.entries()) {
-        if (entry.model.includes(model)) {
-          this.cache.delete(key);
-        }
-      }
-    } else {
-      this.cache.clear();
-    }
-  }
-
-  /**
-   * Get cache statistics
-   */
-  getStats(): { hits: number; misses: number; size: number; hit_rate: number } {
-    const total = this.hits + this.misses;
-    return {
-      hits: this.hits,
-      misses: this.misses,
-      size: this.cache.size,
-      hit_rate: total > 0 ? this.hits / total : 0,
-    };
-  }
-
-  /**
-   * Evict oldest entry by cached_at timestamp
-   */
-  private evictOldest(): void {
-    let oldestKey: string | null = null;
-    let oldestTime = Infinity;
-
-    for (const [key, entry] of this.cache.entries()) {
-      if (entry.cached_at < oldestTime) {
-        oldestTime = entry.cached_at;
-        oldestKey = key;
-      }
-    }
-
-    if (oldestKey) {
-      this.cache.delete(oldestKey);
-    }
-  }
-}

package/tmlpd-pi-extension/src/cost/costTracker.ts

@@ -1,302 +0,0 @@
-/**
- * TMLPD Cost Tracker
- * 
- * Tracks real-time spending across all providers.
- * Supports per-model budgets, spending alerts, and cost analysis.
- */
-
-// Cost per 1M tokens for known models (USD)
-const MODEL_COSTS: Record<string, { input: number; output: number }> = {
-  // Anthropic
-  "claude-3-5-sonnet-20241022": { input: 3.0, output: 15.0 },
-  "claude-3-opus-20240229": { input: 15.0, output: 75.0 },
-  "claude-3-sonnet-20240229": { input: 3.0, output: 15.0 },
-  "claude-3-haiku-20240307": { input: 0.25, output: 1.25 },
-  // OpenAI
-  "gpt-4o": { input: 2.5, output: 10.0 },
-  "gpt-4-turbo": { input: 10.0, output: 30.0 },
-  "gpt-4": { input: 30.0, output: 60.0 },
-  "gpt-3.5-turbo": { input: 0.5, output: 1.5 },
-  // Google
-  "gemini-1.5-pro": { input: 1.25, output: 5.0 },
-  "gemini-1.5-flash": { input: 0.075, output: 0.3 },
-  // Groq
-  "llama-3.3-70b-versatile": { input: 0.59, output: 0.79 },
-  "llama-3.1-8b-instant": { input: 0.05, output: 0.08 },
-  // Cerebras
-  "llama-3.3-70b": { input: 0.1, output: 0.1 },
-  // Mistral
-  "mistral-large-latest": { input: 2.0, output: 6.0 },
-  "mistral-small-latest": { input: 0.2, output: 0.6 },
-  // xAI
-  "grok-2": { input: 2.0, output: 8.0 },
-  "grok-2-mini": { input: 0.2, output: 0.8 },
-  // OpenRouter (varies by model)
-  "openai/gpt-4o": { input: 2.5, output: 10.0 },
-  "anthropic/claude-3.5-sonnet": { input: 3.0, output: 15.0 },
-  // ZAI (default estimate)
-  "glm-5": { input: 0.1, output: 0.3 },
-  "glm-4": { input: 0.1, output: 0.3 },
-};
-
-export interface BudgetConfig {
-  daily_limit?: number;
-  monthly_limit?: number;
-  per_model_limits?: Record<string, number>;
-}
-
-export interface CostAlert {
-  type: "daily" | "monthly" | "model" | "budget";
-  threshold: number;
-  current: number;
-  provider?: string;
-  model?: string;
-}
-
-export interface CostSnapshot {
-  provider: string;
-  model: string;
-  input_tokens: number;
-  output_tokens: number;
-  input_cost: number;
-  output_cost: number;
-  total_cost: number;
-  timestamp: number;
-}
-
-export interface CostSummary {
-  total_cost: number;
-  by_provider: Record<string, number>;
-  by_model: Record<string, number>;
-  daily_costs: Record<string, number>;
-  monthly_costs: Record<string, number>;
-  request_count: number;
-  token_count: { input: number; output: number };
-  average_cost_per_request: number;
-}
-
-export class CostTracker {
-  private history: CostSnapshot[] = [];
-  private budgets: BudgetConfig;
-  private alerts: CostAlert[] = [];
-  private alerts_callback: ((alert: CostAlert) => void) | null = null;
-  private daily_reset: number;
-  private monthly_reset: number;
-
-  constructor(budgets: BudgetConfig = {}) {
-    this.budgets = budgets;
-    const now = new Date();
-    this.daily_reset = new Date(now.getFullYear(), now.getMonth(), now.getDate() + 1).getTime();
-    this.monthly_reset = new Date(now.getFullYear(), now.getMonth() + 1, 1).getTime();
-  }
-
-  /**
-   * Calculate cost for a model based on tokens
-   */
-  calculateCost(model: string, input_tokens: number, output_tokens: number): { input: number; output: number; total: number } {
-    const model_key = model.split("/").pop() || model;
-    const rates = MODEL_COSTS[model_key] || { input: 1.0, output: 5.0 }; // Default estimate
-
-    const input_cost = (input_tokens / 1_000_000) * rates.input;
-    const output_cost = (output_tokens / 1_000_000) * rates.output;
-
-    return {
-      input: Math.round(input_cost * 1000000) / 1000000, // 6 decimal precision
-      output: Math.round(output_cost * 1000000) / 1000000,
-      total: Math.round((input_cost + output_cost) * 1000000) / 1000000,
-    };
-  }
-
-  /**
-   * Record a request's cost
-   */
-  record(provider: string, model: string, input_tokens: number, output_tokens: number): CostSnapshot {
-    const costs = this.calculateCost(model, input_tokens, output_tokens);
-    const snapshot: CostSnapshot = {
-      provider,
-      model,
-      input_tokens,
-      output_tokens,
-      input_cost: costs.input,
-      output_cost: costs.output,
-      total_cost: costs.total,
-      timestamp: Date.now(),
-    };
-
-    this.history.push(snapshot);
-    this.checkBudgets(snapshot);
-    return snapshot;
-  }
-
-  /**
-   * Check budgets and trigger alerts
-   */
-  private checkBudgets(snapshot: CostSnapshot): void {
-    const summary = this.getSummary();
-    const today = new Date().toISOString().split("T")[0];
-    const month = today.substring(0, 7);
-
-    // Check daily budget
-    if (this.budgets.daily_limit) {
-      const daily_cost = summary.daily_costs[today] || 0;
-      if (daily_cost >= this.budgets.daily_limit * 0.9) { // Alert at 90%
-        this.emitAlert({
-          type: "daily",
-          threshold: this.budgets.daily_limit,
-          current: daily_cost,
-        });
-      }
-    }
-
-    // Check monthly budget
-    if (this.budgets.monthly_limit) {
-      const monthly_cost = summary.monthly_costs[month] || 0;
-      if (monthly_cost >= this.budgets.monthly_limit * 0.9) {
-        this.emitAlert({
-          type: "monthly",
-          threshold: this.budgets.monthly_limit,
-          current: monthly_cost,
-        });
-      }
-    }
-
-    // Check per-model budgets
-    if (this.budgets.per_model_limits) {
-      const model_limit = this.budgets.per_model_limits[snapshot.model];
-      if (model_limit) {
-        const model_cost = summary.by_model[snapshot.model] || 0;
-        if (model_cost >= model_limit * 0.9) {
-          this.emitAlert({
-            type: "model",
-            threshold: model_limit,
-            current: model_cost,
-            model: snapshot.model,
-          });
-        }
-      }
-    }
-  }
-
-  /**
-   * Emit an alert via callback
-   */
-  private emitAlert(alert: CostAlert): void {
-    // Avoid duplicate alerts for same threshold
-    const recent = this.alerts.find(
-      (a) =>
-        a.type === alert.type &&
-        a.threshold === alert.threshold &&
-        Date.now() - (a as any)._emitted_at < 3600000 // 1 hour cooldown
-    );
-    if (recent) return;
-
-    (alert as any)._emitted_at = Date.now();
-    this.alerts.push(alert);
-    if (this.alerts_callback) {
-      this.alerts_callback(alert);
-    }
-  }
-
-  /**
-   * Register alert callback
-   */
-  onAlert(callback: (alert: CostAlert) => void): void {
-    this.alerts_callback = callback;
-  }
-
-  /**
-   * Get comprehensive cost summary
-   */
-  getSummary(): CostSummary {
-    const nowMs = Date.now();
-    const today = new Date().toISOString().split("T")[0];
-    const month = today.substring(0, 7);
-
-    // Reset if new day/month
-    const nowDate = new Date(nowMs);
-    if (nowMs >= this.daily_reset) {
-      this.daily_reset = new Date(nowDate.getFullYear(), nowDate.getMonth(), nowDate.getDate() + 1).getTime();
-    }
-    if (nowMs >= this.monthly_reset) {
-      this.monthly_reset = new Date(nowDate.getFullYear(), nowDate.getMonth() + 1, 1).getTime();
-    }
-
-    const by_provider: Record<string, number> = {};
-    const by_model: Record<string, number> = {};
-    const daily_costs: Record<string, number> = {};
-    const monthly_costs: Record<string, number> = {};
-    let total_cost = 0;
-    let total_input_tokens = 0;
-    let total_output_tokens = 0;
-
-    for (const entry of this.history) {
-      total_cost += entry.total_cost;
-      total_input_tokens += entry.input_tokens;
-      total_output_tokens += entry.output_tokens;
-
-      by_provider[entry.provider] = (by_provider[entry.provider] || 0) + entry.total_cost;
-      by_model[entry.model] = (by_model[entry.model] || 0) + entry.total_cost;
-
-      const entry_date = new Date(entry.timestamp).toISOString().split("T")[0];
-      const entry_month = entry_date.substring(0, 7);
-
-      daily_costs[entry_date] = (daily_costs[entry_date] || 0) + entry.total_cost;
-      monthly_costs[entry_month] = (monthly_costs[entry_month] || 0) + entry.total_cost;
-    }
-
-    return {
-      total_cost: Math.round(total_cost * 1000000) / 1000000,
-      by_provider,
-      by_model,
-      daily_costs,
-      monthly_costs,
-      request_count: this.history.length,
-      token_count: { input: total_input_tokens, output: total_output_tokens },
-      average_cost_per_request:
-        this.history.length > 0
-          ? Math.round((total_cost / this.history.length) * 1000000) / 1000000
-          : 0,
-    };
-  }
-
-  /**
-   * Get remaining budget
-   */
-  getRemainingBudget(): { daily: number | null; monthly: number | null; per_model: Record<string, number> } {
-    const summary = this.getSummary();
-    const today = new Date().toISOString().split("T")[0];
-    const month = today.substring(0, 7);
-
-    return {
-      daily: this.budgets.daily_limit
-        ? Math.max(0, this.budgets.daily_limit - (summary.daily_costs[today] || 0))
-        : null,
-      monthly: this.budgets.monthly_limit
-        ? Math.max(0, this.budgets.monthly_limit - (summary.monthly_costs[month] || 0))
-        : null,
-      per_model: this.budgets.per_model_limits
-        ? Object.fromEntries(
-            Object.entries(this.budgets.per_model_limits).map(([model, limit]) => [
-              model,
-              Math.max(0, limit - (summary.by_model[model] || 0)),
-            ])
-          )
-        : {},
-    };
-  }
-
-  /**
-   * Reset cost history
-   */
-  reset(): void {
-    this.history = [];
-    this.alerts = [];
-  }
-
-  /**
-   * Export cost data for analysis
-   */
-  export(): CostSnapshot[] {
-    return [...this.history];
-  }
-}