adaptive-memory-multi-model-router 2.14.49 → 2.14.52

package/test-council/council.md

@@ -0,0 +1,183 @@
+# A3M Router Test Council
+
+## Overview
+
+The A3M Router Test Council is a multi-agent testing approach inspired by the agent council pattern used in production LLM systems. Instead of a single monolithic test suite, we use **3 specialist agents**, each evaluating the codebase from a distinct perspective.
+
+## Why Agent Council?
+
+Traditional testing treats all tests equally. A council approach recognizes that:
+
+1. **Different aspects of code quality require different testing strategies**
+2. **Specialized agents can find issues a general test would miss**
+3. **Council decisions are more robust than single-agent decisions**
+
+## The 3 Specialist Agents
+
+### 1. Structure Agent (`1-structure-tests.ts`)
+
+**Focus:** Code architecture, type safety, and export validation
+
+**Specialization:**
+- Verifies all exports are present and correctly typed
+- Validates provider configuration structure
+- Checks model profile completeness
+- Ensures class instantiation works
+- Validates enum and constant values
+
+**Key Questions:**
+- Are all public APIs exported correctly?
+- Do model profiles have required fields?
+- Are provider configs valid?
+- Do class constructors work?
+
+**Example Findings:**
+- Missing required fields in provider config
+- Inconsistent type definitions
+- Invalid enum values
+- Missing method exports
+
+### 2. Edge-Case Agent (`2-edge-case-tests.ts`)
+
+**Focus:** Boundary conditions, error handling, and invalid inputs
+
+**Specialization:**
+- Tests empty and null inputs
+- Validates extremely long inputs
+- Checks unknown/invalid models
+- Tests missing API key handling
+- Verifies concurrent request safety
+- Tests special characters and unicode
+- Validates error recovery
+
+**Key Questions:**
+- What happens with empty strings?
+- How does the system handle null inputs?
+- What about 10,000 word queries?
+- Does concurrent access cause race conditions?
+- Are special characters handled safely?
+
+**Example Findings:**
+- Null pointer exceptions on empty inputs
+- Buffer overflow on very long queries
+- Race conditions in shared state
+- Missing error boundaries
+
+### 3. Performance Agent (`3-performance-tests.ts`)
+
+**Focus:** Latency, throughput, and scalability
+
+**Specialization:**
+- Benchmarks token counting (< 1ms target)
+- Measures route decision latency (< 5ms target)
+- Tests memory tree operations
+- Validates cost estimation accuracy
+- Measures batch routing throughput
+- Tests concurrent operation performance
+
+**Key Questions:**
+- Is token counting fast enough?
+- Can we route 100 queries per second?
+- Do memory operations scale?
+- Is cost estimation accurate?
+- Are operations deterministic?
+
+**Example Findings:**
+- Token counting too slow for real-time use
+- Memory tree operations don't scale
+- Inconsistent results under load
+- Cost calculation drift over time
+
+## Supporting Test Files
+
+### 4. Integration Tests (`4-integration-tests.ts`)
+
+**Focus:** Full pipeline workflows across components
+
+**Tests:**
+- Extract features → Route → Return structure
+- Memory tree add → search → verify
+- Cost tracker add → verify total
+- Provider register → get → deregister
+- Cross-component workflows
+
+### 5. Cost Model Tests (`5-cost-model-tests.ts`)
+
+**Focus:** Financial accuracy and budget enforcement
+
+**Tests:**
+- MODEL_COSTS matches actual provider pricing
+- estimateCost accuracy across tiers (free/cheap/mid/premium)
+- Cost comparison between models is correct
+- Budget enforcer respects limits
+- CostTracker calculations match expected values
+
+## How the Council Evaluates
+
+Each agent runs independently and reports:
+
+```
+Agent: Structure
+✅ All exports present
+✅ Provider configs valid
+❌ MODEL_PROFILES missing context_window field
+✅ Class instantiation works
+
+Result: 47/48 passed
+```
+
+The council **passes** only if **all agents pass**. This ensures:
+- No single aspect degrades
+- Issues in any dimension are caught
+- Overall system quality is maintained
+
+## Success Criteria
+
+For the council to pass, we require:
+
+| Agent | Metric | Target |
+|-------|--------|--------|
+| Structure | Tests passed | 100% |
+| Edge-Case | Tests passed | 100% |
+| Performance | Latency targets met | 100% |
+| Integration | Pipeline tests passed | 100% |
+| Cost Model | Cost accuracy | 100% |
+
+## Running the Council
+
+```bash
+# Run all tests
+node test-council/1-structure-tests.ts
+node test-council/2-edge-case-tests.ts
+node test-council/3-performance-tests.ts
+node test-council/4-integration-tests.ts
+node test-council/5-cost-model-tests.ts
+
+# Or use the test runner
+./test-council/run-all.sh
+```
+
+## Design Principles
+
+1. **Independence**: Each test file runs standalone
+2. **Isolation**: Tests don't depend on each other
+3. **Clear reporting**: Pass/fail with actionable messages
+4. **Focused scope**: Each agent has clear responsibilities
+5. **Realistic inputs**: Tests use real-world query patterns
+
+## Evolution
+
+The council pattern can be extended:
+
+- **Security Agent**: Input sanitization, injection attacks
+- **Reliability Agent**: Network failure simulation
+- **Scalability Agent**: Load testing with 1000+ concurrent users
+- **Compatibility Agent**: Node.js version, dependency conflicts
+
+## Credits
+
+Inspired by:
+- Ensemble methods in machine learning
+- Agent councils in production LLM systems (Mixtral, GPT-4)
+- Test-driven development principles
+- Separation of concerns in software architecture

package/tests/__mocks__/tokenUtils.ts

@@ -0,0 +1,8 @@
+// Mock for token utilities used by providerRetry
+export function countTokens(text: string): number {
+  return Math.ceil(text.length / 4);
+}
+
+export function estimateTokens(text: string): number {
+  return countTokens(text);
+}

package/tests/memory/episodicMemory.test.ts

@@ -0,0 +1,227 @@
+import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
+import { EpisodicMemoryStore } from '../../tmlpd-pi-extension/src/memory/episodicMemory';
+
+describe('EpisodicMemoryStore', () => {
+  let store: EpisodicMemoryStore;
+
+  const sampleEntry = {
+    task: {
+      description: 'Test task execution',
+      type: 'code_generation',
+      complexity: 3,
+    },
+    result: {
+      success: true,
+      output: 'Successfully generated code',
+      duration_ms: 1500,
+    },
+    agent: {
+      id: 'agent-1',
+      model: 'gpt-4o',
+      provider: 'openai',
+    },
+    metadata: { source: 'test' },
+    importance: 5,
+  };
+
+  beforeEach(() => {
+    store = new EpisodicMemoryStore(100);
+  });
+
+  describe('store and query', () => {
+    it('stores an entry and returns a non-empty id', () => {
+      const id = store.store(sampleEntry);
+      expect(id).toBeTruthy();
+      expect(typeof id).toBe('string');
+      expect(id.length).toBeGreaterThan(0);
+    });
+
+    it('stores multiple entries and retrieves them', () => {
+      store.store(sampleEntry);
+      store.store({ ...sampleEntry, task: { ...sampleEntry.task, description: 'Second task' } });
+
+      const results = store.query({ limit: 10 });
+      expect(results.length).toBe(2);
+    });
+
+    it('filters by task type', () => {
+      store.store(sampleEntry);
+      store.store({
+        ...sampleEntry,
+        task: { ...sampleEntry.task, type: 'data_analysis' },
+      });
+
+      const codeResults = store.query({ task_type: 'code_generation' });
+      expect(codeResults.length).toBe(1);
+      expect(codeResults[0].task.type).toBe('code_generation');
+
+      const analysisResults = store.query({ task_type: 'data_analysis' });
+      expect(analysisResults.length).toBe(1);
+    });
+
+    it('filters by keywords', () => {
+      store.store(sampleEntry);
+      store.store({
+        ...sampleEntry,
+        task: { ...sampleEntry.task, description: 'Database query optimization' },
+      });
+
+      const results = store.query({ keywords: ['database'] });
+      expect(results.length).toBe(1);
+      expect(results[0].task.description).toContain('Database');
+    });
+
+    it('returns newest entries first', () => {
+      store.store(sampleEntry);
+      store.store({ ...sampleEntry, task: { ...sampleEntry.task, description: 'Second' } });
+
+      const results = store.query({ limit: 10 });
+      expect(results[results.length - 1].task.description).toBe('Second');
+    });
+
+    it('respects limit parameter', () => {
+      for (let i = 0; i < 10; i++) {
+        store.store({ ...sampleEntry, task: { ...sampleEntry.task, description: `Task ${i}` } });
+      }
+
+      expect(store.query({ limit: 3 }).length).toBe(3);
+      expect(store.query({ limit: 20 }).length).toBe(10);
+    });
+  });
+
+  describe('max entries eviction', () => {
+    it('evicts oldest entries when at capacity', () => {
+      const smallStore = new EpisodicMemoryStore(3);
+      smallStore.store({ ...sampleEntry, task: { ...sampleEntry.task, description: 'A' } });
+      smallStore.store({ ...sampleEntry, task: { ...sampleEntry.task, description: 'B' } });
+      smallStore.store({ ...sampleEntry, task: { ...sampleEntry.task, description: 'C' } });
+      smallStore.store({ ...sampleEntry, task: { ...sampleEntry.task, description: 'D' } });
+
+      expect(smallStore.query({ limit: 10 }).length).toBe(3);
+      const descriptions = smallStore.query({ limit: 10 }).map(e => e.task.description);
+      expect(descriptions).not.toContain('A');
+      expect(descriptions).toContain('D');
+    });
+  });
+
+  describe('getSimilarTasks', () => {
+    it('finds tasks with overlapping words', () => {
+      store.store({ ...sampleEntry, task: { ...sampleEntry.task, description: 'Build REST API endpoint' } });
+      store.store({ ...sampleEntry, task: { ...sampleEntry.task, description: 'Deploy database cluster' } });
+      store.store({ ...sampleEntry, task: { ...sampleEntry.task, description: 'Fix API authentication bug' } });
+
+      const similar = store.getSimilarTasks('Build API endpoint');
+      expect(similar.length).toBeGreaterThanOrEqual(1);
+      // "API" should match entries 0 and 2
+      const descriptions = similar.map(e => e.task.description);
+      expect(descriptions.some(d => d.includes('API'))).toBe(true);
+    });
+
+    it('returns empty array for no matches', () => {
+      store.store(sampleEntry);
+      const similar = store.getSimilarTasks('xyznonexistent12345');
+      expect(similar.length).toBe(0);
+    });
+
+    it('respects limit parameter', () => {
+      store.store({ ...sampleEntry, task: { ...sampleEntry.task, description: 'Task A' } });
+      store.store({ ...sampleEntry, task: { ...sampleEntry.task, description: 'Task B' } });
+      store.store({ ...sampleEntry, task: { ...sampleEntry.task, description: 'Task C' } });
+
+      const similar = store.getSimilarTasks('Task', 2);
+      expect(similar.length).toBeLessThanOrEqual(2);
+    });
+  });
+
+  describe('getStats', () => {
+    it('returns zeroed stats for empty store', () => {
+      const stats = store.getStats();
+      expect(stats.total_entries).toBe(0);
+      expect(stats.indexed_keywords).toBe(0);
+      expect(stats.success_rate).toBe(0);
+      expect(stats.avg_duration_ms).toBe(0);
+    });
+
+    it('returns accurate stats after entries', () => {
+      store.store(sampleEntry);
+      store.store({
+        ...sampleEntry,
+        result: { ...sampleEntry.result, success: false },
+      });
+
+      const stats = store.getStats();
+      expect(stats.total_entries).toBe(2);
+      expect(stats.success_rate).toBe(0.5);
+      expect(stats.avg_duration_ms).toBe(1500);
+    });
+  });
+
+  describe('clear', () => {
+    it('removes all entries', () => {
+      store.store(sampleEntry);
+      store.store(sampleEntry);
+      expect(store.query({ limit: 10 }).length).toBe(2);
+
+      store.clear();
+      expect(store.query({ limit: 10 }).length).toBe(0);
+      expect(store.getStats().total_entries).toBe(0);
+    });
+  });
+
+  describe('persistence', () => {
+    let fsMock: any;
+
+    beforeEach(() => {
+      fsMock = {
+        existsSync: vi.fn().mockReturnValue(false),
+        readFileSync: vi.fn(),
+        writeFileSync: vi.fn(),
+        unlinkSync: vi.fn(),
+      };
+
+      vi.doMock('fs', () => fsMock);
+    });
+
+    afterEach(() => {
+      vi.restoreAllMocks();
+    });
+
+    it('loadFromDisk returns false when file does not exist', () => {
+      const store2 = new EpisodicMemoryStore(100, '/tmp/test-memory.json');
+      expect(store2['loadFromDisk']()).toBe(false);
+    });
+
+    it('saveToDisk returns false when no persistence path', () => {
+      expect(store.saveToDisk()).toBe(false);
+    });
+
+    it('autoPersist triggers after every 3 entries', () => {
+      const saveSpy = vi.spyOn(store as any, 'autoPersist');
+      store.store(sampleEntry);
+      store.store(sampleEntry);
+      expect(saveSpy).toHaveBeenCalledTimes(2); // called on each store
+      expect((store as any).entries.length % 3 === 0).toBe(false);
+
+      store.store(sampleEntry); // 3rd entry
+      expect((store as any).entries.length % 3 === 0).toBe(true);
+    });
+  });
+
+  describe('keyword indexing', () => {
+    it('indexes words longer than 3 characters', () => {
+      store.store(sampleEntry);
+      const index = (store as any).keywordIndex as Map<string, string[]>;
+
+      // 'test' (4 chars) should be indexed, 'is' (2 chars) should not
+      const keys = Array.from(index.keys());
+      expect(keys.some(k => k.length <= 3)).toBe(false);
+    });
+
+    it('automatically indexes on store', () => {
+      store.store(sampleEntry);
+      const index = (store as any).keywordIndex as Map<string, string[]>;
+      expect(index.has('test')).toBe(true);
+      expect(index.has('task')).toBe(true);
+    });
+  });
+});