agentic-flow 1.9.0 → 1.9.1

package/bench/README.md

@@ -1,217 +0,0 @@
-# ReasoningBank Benchmark Suite
-
-Comprehensive benchmarking system to evaluate ReasoningBank's closed-loop learning capabilities against a baseline system without memory.
-
-## 📊 Benchmark Overview
-
-This suite measures the impact of ReasoningBank's 4-phase learning loop (RETRIEVE → JUDGE → DISTILL → CONSOLIDATE) on agent performance across multiple dimensions.
-
-### Key Metrics
-
-1. **Success Rate**: Task completion success over time (0% → 100% transformation)
-2. **Learning Velocity**: Speed of improvement (tasks until first success)
-3. **Memory Efficiency**: Storage and retrieval performance
-4. **Token Efficiency**: Reduction in token usage via memory
-5. **Latency Impact**: Overhead from memory operations
-6. **Accuracy**: Quality of learned patterns
-7. **Generalization**: Transfer learning across domains
-
-## 📁 Directory Structure
-
-```
-bench/
-├── scenarios/        # Test scenarios (coding, debugging, API design, etc.)
-├── agents/          # Baseline vs ReasoningBank agents
-├── metrics/         # Metrics collection and analysis
-├── results/         # Raw benchmark results
-├── reports/         # Generated reports and visualizations
-├── lib/             # Shared utilities
-├── benchmark.ts     # Main orchestrator
-└── README.md        # This file
-```
-
-## 🚀 Quick Start
-
-```bash
-# Run all benchmarks
-npm run bench
-
-# Run specific scenario
-npm run bench -- --scenario coding-tasks
-
-# Run with different configurations
-npm run bench -- --iterations 100 --agents 5
-
-# Generate report only
-npm run bench:report
-```
-
-## 📋 Benchmark Scenarios
-
-### 1. Coding Tasks (10 scenarios)
-- Implement functions from specifications
-- Fix bugs in existing code
-- Refactor code for best practices
-- Add error handling
-- Write unit tests
-
-### 2. Debugging Tasks (10 scenarios)
-- Identify and fix runtime errors
-- Resolve type errors
-- Fix logical errors
-- Handle edge cases
-- Debug async issues
-
-### 3. API Design Tasks (10 scenarios)
-- Design REST endpoints
-- Create authentication systems
-- Implement rate limiting
-- Design database schemas
-- Build validation logic
-
-### 4. Problem Solving Tasks (10 scenarios)
-- Algorithm challenges
-- Data structure problems
-- System design questions
-- Optimization problems
-- Pattern recognition
-
-### 5. Continuous Learning (Progressive)
-- Same task repeated with variations
-- Measures learning curve
-- Tests memory consolidation
-- Validates pattern generalization
-
-## 📈 Expected Results
-
-Based on the ReasoningBank paper (https://arxiv.org/html/2509.25140v1):
-
-- **Success Rate**: 0% → 100% over 20-30 iterations
-- **Token Reduction**: ~32.3% via memory injection
-- **Latency Overhead**: <500ms for retrieval
-- **Memory Efficiency**: 95%+ deduplication rate
-- **Learning Velocity**: 2.8-4.4x faster than baseline
-
-## 🔬 Benchmark Types
-
-### 1. Cold Start (No Prior Memory)
-- Both agents start from scratch
-- Measures raw learning capability
-- Tests distillation quality
-
-### 2. Warm Start (Pre-populated Memory)
-- ReasoningBank has domain memories
-- Tests retrieval effectiveness
-- Measures transfer learning
-
-### 3. Incremental Learning
-- Sequential task variations
-- Tests consolidation
-- Measures adaptation speed
-
-### 4. Stress Test
-- High memory load (1000+ patterns)
-- Concurrent operations
-- Edge cases and failures
-
-## 📊 Output Formats
-
-### Console Progress
-```
-🧪 Running Benchmark: coding-tasks (1/50)
-  ├─ Baseline:      [=====>    ] 50% (5/10 success)
-  └─ ReasoningBank: [==========>] 100% (10/10 success)
-```
-
-### JSON Results
-```json
-{
-  "scenario": "coding-tasks",
-  "baseline": {
-    "successRate": 0.50,
-    "avgTokens": 15230,
-    "avgLatency": 2341
-  },
-  "reasoningbank": {
-    "successRate": 1.00,
-    "avgTokens": 10311,
-    "avgLatency": 2756,
-    "memoriesUsed": 12,
-    "memoriesCreated": 8
-  },
-  "improvement": {
-    "successRate": "+100%",
-    "tokenEfficiency": "+32.3%",
-    "latencyOverhead": "+17.7%"
-  }
-}
-```
-
-### Markdown Report
-- Executive summary
-- Detailed metrics tables
-- Comparison charts
-- Learning curves
-- Recommendations
-
-## 🛠️ Configuration
-
-Edit `bench/config.json`:
-
-```json
-{
-  "iterations": 50,
-  "parallelAgents": 5,
-  "scenarios": ["coding", "debugging", "api-design"],
-  "enableWarmStart": false,
-  "memorySize": 1000,
-  "outputFormats": ["json", "markdown", "csv"]
-}
-```
-
-## 📖 Interpretation Guide
-
-### Success Rate Delta
-- `> +50%`: ReasoningBank significantly improves learning
-- `20-50%`: Moderate improvement, memory helps
-- `< 20%`: Minimal impact, task may not benefit from memory
-
-### Token Efficiency
-- `> +30%`: Excellent memory utilization
-- `15-30%`: Good reduction via context injection
-- `< 15%`: Limited memory reuse
-
-### Latency Overhead
-- `< 500ms`: Acceptable for production
-- `500-1000ms`: Noticeable but manageable
-- `> 1000ms`: Optimization needed
-
-## 🔧 Extending Benchmarks
-
-Add new scenarios in `bench/scenarios/`:
-
-```typescript
-export const myScenario: BenchmarkScenario = {
-  name: 'my-scenario',
-  description: 'Custom benchmark scenario',
-  tasks: [/* tasks */],
-  successCriteria: (result) => /* validation */,
-  metrics: ['success', 'tokens', 'latency']
-};
-```
-
-## 📚 References
-
-- ReasoningBank Paper: https://arxiv.org/html/2509.25140v1
-- 4-Factor Scoring Formula: α·sim + β·rec + γ·rel + δ·div
-- MaTTS (Memory-aware Test-Time Scaling)
-- MMR (Maximal Marginal Relevance) for diversity
-
-## 🤝 Contributing
-
-To add benchmarks:
-1. Create scenario in `scenarios/`
-2. Add to `benchmark.ts` orchestrator
-3. Update metrics collection
-4. Run and validate results
-5. Document in this README

package/bench/agents/baseline-agent.js

@@ -1,69 +0,0 @@
-/**
- * Baseline Agent (No Memory)
- *
- * Executes tasks without any learning or memory capabilities.
- * Used as the control group in benchmarks.
- */
-import Anthropic from '@anthropic-ai/sdk';
-export class BaselineAgent {
-    type = 'baseline';
-    client;
-    model;
-    constructor(apiKey, model = 'claude-sonnet-4-5-20250929') {
-        this.client = new Anthropic({
-            apiKey: apiKey || process.env.ANTHROPIC_API_KEY
-        });
-        this.model = model;
-    }
-    async executeTask(task) {
-        const startTime = Date.now();
-        try {
-            // Execute task with Claude (no memory injection)
-            const response = await this.client.messages.create({
-                model: this.model,
-                max_tokens: 4096,
-                messages: [
-                    {
-                        role: 'user',
-                        content: `${task.description}\n\nInput:\n${task.input}`
-                    }
-                ]
-            });
-            const latency = Date.now() - startTime;
-            const output = response.content[0].type === 'text' ? response.content[0].text : '';
-            const tokens = response.usage.input_tokens + response.usage.output_tokens;
-            // Evaluate success
-            const success = task.successCriteria({
-                taskId: task.id,
-                agentType: 'baseline',
-                success: false, // Will be updated
-                output,
-                tokens,
-                latency
-            });
-            return {
-                taskId: task.id,
-                agentType: 'baseline',
-                success,
-                output,
-                tokens,
-                latency
-            };
-        }
-        catch (error) {
-            const latency = Date.now() - startTime;
-            return {
-                taskId: task.id,
-                agentType: 'baseline',
-                success: false,
-                output: '',
-                tokens: 0,
-                latency,
-                error: error instanceof Error ? error.message : String(error)
-            };
-        }
-    }
-    async reset() {
-        // Baseline has no state to reset
-    }
-}

package/bench/agents/baseline-agent.ts

@@ -1,78 +0,0 @@
-/**
- * Baseline Agent (No Memory)
- *
- * Executes tasks without any learning or memory capabilities.
- * Used as the control group in benchmarks.
- */
-
-import Anthropic from '@anthropic-ai/sdk';
-import type { Agent, Task, TaskResult } from '../lib/types.js';
-
-export class BaselineAgent implements Agent {
-  readonly type = 'baseline';
-  private client: Anthropic;
-  private model: string;
-
-  constructor(apiKey?: string, model: string = 'claude-sonnet-4-5-20250929') {
-    this.client = new Anthropic({
-      apiKey: apiKey || process.env.ANTHROPIC_API_KEY
-    });
-    this.model = model;
-  }
-
-  async executeTask(task: Task): Promise<TaskResult> {
-    const startTime = Date.now();
-
-    try {
-      // Execute task with Claude (no memory injection)
-      const response = await this.client.messages.create({
-        model: this.model,
-        max_tokens: 4096,
-        messages: [
-          {
-            role: 'user',
-            content: `${task.description}\n\nInput:\n${task.input}`
-          }
-        ]
-      });
-
-      const latency = Date.now() - startTime;
-      const output = response.content[0].type === 'text' ? response.content[0].text : '';
-      const tokens = response.usage.input_tokens + response.usage.output_tokens;
-
-      // Evaluate success
-      const success = task.successCriteria({
-        taskId: task.id,
-        agentType: 'baseline',
-        success: false, // Will be updated
-        output,
-        tokens,
-        latency
-      });
-
-      return {
-        taskId: task.id,
-        agentType: 'baseline',
-        success,
-        output,
-        tokens,
-        latency
-      };
-    } catch (error) {
-      const latency = Date.now() - startTime;
-      return {
-        taskId: task.id,
-        agentType: 'baseline',
-        success: false,
-        output: '',
-        tokens: 0,
-        latency,
-        error: error instanceof Error ? error.message : String(error)
-      };
-    }
-  }
-
-  async reset(): Promise<void> {
-    // Baseline has no state to reset
-  }
-}

package/bench/agents/reasoningbank-agent.js

@@ -1,141 +0,0 @@
-/**
- * ReasoningBank-Enabled Agent
- *
- * Executes tasks with full ReasoningBank capabilities:
- * - RETRIEVE: Fetches relevant memories
- * - JUDGE: Evaluates success/failure
- * - DISTILL: Extracts learnings
- * - CONSOLIDATE: Deduplicates and prunes
- */
-import Anthropic from '@anthropic-ai/sdk';
-import { retrieveMemories, formatMemoriesForPrompt, judgeTrajectory, distillMemories, shouldConsolidate, consolidate, db } from '../../agentic-flow/src/reasoningbank/index.js';
-import { ulid } from 'ulid';
-export class ReasoningBankAgent {
-    type = 'reasoningbank';
-    client;
-    model;
-    agentId;
-    constructor(apiKey, model = 'claude-sonnet-4-5-20250929') {
-        this.client = new Anthropic({
-            apiKey: apiKey || process.env.ANTHROPIC_API_KEY
-        });
-        this.model = model;
-        this.agentId = 'benchmark-agent';
-    }
-    async executeTask(task) {
-        const startTime = Date.now();
-        const taskId = ulid();
-        try {
-            // 1. RETRIEVE: Get relevant memories
-            const memories = await retrieveMemories(task.description, {
-                domain: task.domain,
-                agent: this.agentId
-            });
-            const memoriesUsed = memories.length;
-            // 2. EXECUTE: Run task with memory context
-            const systemPrompt = memories.length > 0
-                ? formatMemoriesForPrompt(memories)
-                : '';
-            const messages = [
-                {
-                    role: 'user',
-                    content: `${task.description}\n\nInput:\n${task.input}`
-                }
-            ];
-            const response = await this.client.messages.create({
-                model: this.model,
-                max_tokens: 4096,
-                system: systemPrompt || undefined,
-                messages
-            });
-            const output = response.content[0].type === 'text' ? response.content[0].text : '';
-            const tokens = response.usage.input_tokens + response.usage.output_tokens;
-            // Build trajectory for judging
-            const trajectory = {
-                steps: [
-                    { action: 'retrieve', memoriesFound: memories.length },
-                    { action: 'execute', output, tokens }
-                ],
-                metadata: {
-                    task: task.description,
-                    input: task.input,
-                    output
-                }
-            };
-            // 3. JUDGE: Evaluate success/failure
-            const verdict = await judgeTrajectory(trajectory, task.description);
-            const success = task.successCriteria({
-                taskId: task.id,
-                agentType: 'reasoningbank',
-                success: false,
-                output,
-                tokens,
-                latency: 0
-            }) && verdict.label === 'Success';
-            // 4. DISTILL: Extract learnings from this execution
-            const newMemories = await distillMemories(trajectory, verdict, task.description, {
-                taskId,
-                agentId: this.agentId,
-                domain: task.domain
-            });
-            const memoriesCreated = newMemories.length;
-            // 5. CONSOLIDATE: Run if threshold reached
-            if (shouldConsolidate()) {
-                await consolidate();
-            }
-            const latency = Date.now() - startTime;
-            return {
-                taskId: task.id,
-                agentType: 'reasoningbank',
-                success,
-                output,
-                tokens,
-                latency,
-                memoriesUsed,
-                memoriesCreated,
-                confidence: verdict.confidence,
-                trajectory
-            };
-        }
-        catch (error) {
-            const latency = Date.now() - startTime;
-            console.error(`[ReasoningBankAgent] Error executing task ${task.id}:`, error);
-            return {
-                taskId: task.id,
-                agentType: 'reasoningbank',
-                success: false,
-                output: '',
-                tokens: 0,
-                latency,
-                memoriesUsed: 0,
-                memoriesCreated: 0,
-                error: error instanceof Error ? error.message : String(error)
-            };
-        }
-    }
-    async reset() {
-        // Clear all memories for this agent
-        const dbConn = db.getDb();
-        dbConn.prepare(`
-      DELETE FROM patterns
-      WHERE type = 'reasoning_memory'
-      AND json_extract(pattern_data, '$.source.agent_id') = ?
-    `).run(this.agentId);
-        console.log(`[ReasoningBankAgent] Reset - cleared all memories`);
-    }
-    async getMemoryStats() {
-        const dbConn = db.getDb();
-        const result = dbConn.prepare(`
-      SELECT
-        COUNT(*) as total,
-        AVG(confidence) as avgConfidence
-      FROM patterns
-      WHERE type = 'reasoning_memory'
-      AND json_extract(pattern_data, '$.source.agent_id') = ?
-    `).get(this.agentId);
-        return {
-            total: result.total || 0,
-            avgConfidence: result.avgConfidence || 0
-        };
-    }
-}

package/bench/agents/reasoningbank-agent.ts

@@ -1,173 +0,0 @@
-/**
- * ReasoningBank-Enabled Agent
- *
- * Executes tasks with full ReasoningBank capabilities:
- * - RETRIEVE: Fetches relevant memories
- * - JUDGE: Evaluates success/failure
- * - DISTILL: Extracts learnings
- * - CONSOLIDATE: Deduplicates and prunes
- */
-
-import Anthropic from '@anthropic-ai/sdk';
-import {
-  retrieveMemories,
-  formatMemoriesForPrompt,
-  judgeTrajectory,
-  distillMemories,
-  shouldConsolidate,
-  consolidate,
-  db
-} from '../../agentic-flow/src/reasoningbank/index.js';
-import { ulid } from 'ulid';
-import type { Agent, Task, TaskResult } from '../lib/types.js';
-
-export class ReasoningBankAgent implements Agent {
-  readonly type = 'reasoningbank';
-  private client: Anthropic;
-  private model: string;
-  private agentId: string;
-
-  constructor(apiKey?: string, model: string = 'claude-sonnet-4-5-20250929') {
-    this.client = new Anthropic({
-      apiKey: apiKey || process.env.ANTHROPIC_API_KEY
-    });
-    this.model = model;
-    this.agentId = 'benchmark-agent';
-  }
-
-  async executeTask(task: Task): Promise<TaskResult> {
-    const startTime = Date.now();
-    const taskId = ulid();
-
-    try {
-      // 1. RETRIEVE: Get relevant memories
-      const memories = await retrieveMemories(task.description, {
-        domain: task.domain,
-        agent: this.agentId
-      });
-
-      const memoriesUsed = memories.length;
-
-      // 2. EXECUTE: Run task with memory context
-      const systemPrompt = memories.length > 0
-        ? formatMemoriesForPrompt(memories)
-        : '';
-
-      const messages: Anthropic.Messages.MessageParam[] = [
-        {
-          role: 'user',
-          content: `${task.description}\n\nInput:\n${task.input}`
-        }
-      ];
-
-      const response = await this.client.messages.create({
-        model: this.model,
-        max_tokens: 4096,
-        system: systemPrompt || undefined,
-        messages
-      });
-
-      const output = response.content[0].type === 'text' ? response.content[0].text : '';
-      const tokens = response.usage.input_tokens + response.usage.output_tokens;
-
-      // Build trajectory for judging
-      const trajectory = {
-        steps: [
-          { action: 'retrieve', memoriesFound: memories.length },
-          { action: 'execute', output, tokens }
-        ],
-        metadata: {
-          task: task.description,
-          input: task.input,
-          output
-        }
-      };
-
-      // 3. JUDGE: Evaluate success/failure
-      const verdict = await judgeTrajectory(trajectory, task.description);
-
-      const success = task.successCriteria({
-        taskId: task.id,
-        agentType: 'reasoningbank',
-        success: false,
-        output,
-        tokens,
-        latency: 0
-      }) && verdict.label === 'Success';
-
-      // 4. DISTILL: Extract learnings from this execution
-      const newMemories = await distillMemories(trajectory, verdict, task.description, {
-        taskId,
-        agentId: this.agentId,
-        domain: task.domain
-      });
-
-      const memoriesCreated = newMemories.length;
-
-      // 5. CONSOLIDATE: Run if threshold reached
-      if (shouldConsolidate()) {
-        await consolidate();
-      }
-
-      const latency = Date.now() - startTime;
-
-      return {
-        taskId: task.id,
-        agentType: 'reasoningbank',
-        success,
-        output,
-        tokens,
-        latency,
-        memoriesUsed,
-        memoriesCreated,
-        confidence: verdict.confidence,
-        trajectory
-      };
-    } catch (error) {
-      const latency = Date.now() - startTime;
-      console.error(`[ReasoningBankAgent] Error executing task ${task.id}:`, error);
-
-      return {
-        taskId: task.id,
-        agentType: 'reasoningbank',
-        success: false,
-        output: '',
-        tokens: 0,
-        latency,
-        memoriesUsed: 0,
-        memoriesCreated: 0,
-        error: error instanceof Error ? error.message : String(error)
-      };
-    }
-  }
-
-  async reset(): Promise<void> {
-    // Clear all memories for this agent
-    const dbConn = db.getDb();
-    dbConn.prepare(`
-      DELETE FROM patterns
-      WHERE type = 'reasoning_memory'
-      AND json_extract(pattern_data, '$.source.agent_id') = ?
-    `).run(this.agentId);
-
-    console.log(`[ReasoningBankAgent] Reset - cleared all memories`);
-  }
-
-  async getMemoryStats(): Promise<{ total: number; avgConfidence: number }> {
-    const dbConn = db.getDb();
-
-    const result = dbConn.prepare(`
-      SELECT
-        COUNT(*) as total,
-        AVG(confidence) as avgConfidence
-      FROM patterns
-      WHERE type = 'reasoning_memory'
-      AND json_extract(pattern_data, '$.source.agent_id') = ?
-    `).get(this.agentId) as { total: number; avgConfidence: number };
-
-    return {
-      total: result.total || 0,
-      avgConfidence: result.avgConfidence || 0
-    };
-  }
-}