@agentic-qe/v3 3.0.0-alpha.3

package/implementation/adrs/ADR-042-v3-qe-token-tracking-integration.md

@@ -0,0 +1,517 @@
+# ADR-042: V3 QE Token Tracking and Consumption Reduction
+
+**Status**: Proposed
+**Date**: 2026-01-11
+**Author**: Claude Code
+**Related**: ADR-040 (Agentic-Flow Integration)
+
+## Context
+
+### Problem Statement
+
+V3 QE agents currently lack:
+1. **Token consumption tracking** - No visibility into how many tokens each agent/task consumes
+2. **Token optimization patterns** - No mechanisms to reduce token usage while maintaining quality
+3. **Cost management** - No way to estimate, track, or optimize API costs per operation
+4. **Pattern-based token reduction** - Missing opportunity to reuse successful patterns to avoid re-computation
+
+### Analysis of agentic-flow (v2.0.1-alpha)
+
+After analyzing the [agentic-flow repository](https://github.com/ruvnet/agentic-flow), we identified proven token reduction mechanisms:
+
+| Feature | Claimed Reduction | Mechanism |
+|---------|-------------------|-----------|
+| ReasoningBank Pattern Reuse | -25% tokens | Vector similarity search avoids re-discovering solutions |
+| Adaptive Learner | -32.3% tokens | Iterative improvement with pattern memory |
+| Context Synthesizer | -15% redundancy | Focused context generation |
+| Memory Optimizer | -20% retrieval | Prunes low-value patterns |
+
+### Current V3 State
+
+```
+v3/src/learning/qe-reasoning-bank.ts  - Has QE patterns but NO token tracking
+v3/src/mcp/metrics/metrics-collector.ts - Tracks CPU/memory but NO tokens
+v3/src/learning/pattern-store.ts - Stores patterns but NO tokensUsed field
+```
+
+## Decision
+
+Implement comprehensive token tracking and consumption reduction in V3 QE by:
+
+1. **Add token tracking to all agent operations**
+2. **Port agentic-flow's proven reduction patterns**
+3. **Create MCP tools for token analysis**
+4. **Implement early-exit optimizations for high-confidence patterns**
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                     V3 QE Token Tracking System                          │
+├─────────────────────────────────────────────────────────────────────────┤
+│                                                                          │
+│  ┌────────────────────────────────────────────────────────────────┐     │
+│  │                    Token Metrics Collector                      │     │
+│  │  • Per-task token tracking (input/output/total)                │     │
+│  │  • Per-agent aggregation                                       │     │
+│  │  • Per-domain breakdown                                        │     │
+│  │  • Session-level summaries                                     │     │
+│  └────────────────────────────────────────────────────────────────┘     │
+│                              │                                           │
+│         ┌────────────────────┼────────────────────┐                     │
+│         ▼                    ▼                    ▼                     │
+│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐              │
+│  │   Pattern    │    │    Cache     │    │  Early Exit  │              │
+│  │    Reuse     │    │   Manager    │    │  Optimizer   │              │
+│  │              │    │              │    │              │              │
+│  │ • HNSW search│    │ • File cache │    │ • High conf  │              │
+│  │ • >0.85 sim  │    │ • Query cache│    │   patterns   │              │
+│  │ • Skip regen │    │ • 5min TTL   │    │ • Skip LLM   │              │
+│  └──────────────┘    └──────────────┘    └──────────────┘              │
+│         │                    │                    │                     │
+│         └────────────────────┼────────────────────┘                     │
+│                              ▼                                           │
+│  ┌────────────────────────────────────────────────────────────────┐     │
+│  │                      Token Reduction Layer                      │     │
+│  │  • Pattern-based shortcuts (avoid re-computation)              │     │
+│  │  • Batch similar operations (fewer API calls)                  │     │
+│  │  • Context compression (focused prompts)                       │     │
+│  │  • Response caching (avoid duplicate requests)                 │     │
+│  └────────────────────────────────────────────────────────────────┘     │
+│                                                                          │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+## Implementation
+
+### 1. Token Tracking Interface
+
+```typescript
+// v3/src/learning/token-tracker.ts
+
+export interface TokenUsage {
+  inputTokens: number;
+  outputTokens: number;
+  totalTokens: number;
+  estimatedCostUsd?: number;
+}
+
+export interface TaskTokenMetric {
+  taskId: string;
+  agentId: string;
+  domain: DomainName;
+  operation: string;
+  timestamp: number;
+  usage: TokenUsage;
+  patternReused: boolean;      // Was a cached pattern used?
+  tokensSaved?: number;        // Estimated tokens saved via caching
+}
+
+export interface AgentTokenMetrics {
+  agentId: string;
+  totalInputTokens: number;
+  totalOutputTokens: number;
+  totalTokens: number;
+  totalCost: number;
+  tasksExecuted: number;
+  patternsReused: number;      // How many times we skipped LLM calls
+  estimatedTokensSaved: number;
+}
+
+export interface SessionTokenSummary {
+  sessionId: string;
+  startTime: number;
+  endTime?: number;
+  byAgent: Map<string, AgentTokenMetrics>;
+  byDomain: Map<DomainName, TokenUsage>;
+  totalUsage: TokenUsage;
+  optimizationStats: {
+    patternsReused: number;
+    cacheHits: number;
+    earlyExits: number;
+    tokensSaved: number;
+    savingsPercentage: number;
+  };
+}
+```
+
+### 2. Extended QE Pattern with Token Tracking
+
+```typescript
+// Update v3/src/learning/qe-patterns.ts
+
+export interface QEPattern {
+  id: string;
+  // ... existing fields ...
+
+  // NEW: Token tracking fields (from agentic-flow)
+  tokensUsed?: number;
+  inputTokens?: number;
+  outputTokens?: number;
+  latencyMs?: number;
+
+  // NEW: Optimization metadata
+  reusable: boolean;           // Can this pattern be reused?
+  reuseCount: number;          // How many times it's been reused
+  averageTokenSavings: number; // Avg tokens saved when reused
+}
+```
+
+### 3. MCP Tool: Token Usage Analysis
+
+```typescript
+// v3/src/mcp/tools/analysis/token-usage.ts
+
+export const tokenUsageTool = {
+  name: 'token_usage',
+  description: 'Analyze token consumption patterns and identify optimization opportunities',
+  inputSchema: {
+    type: 'object',
+    properties: {
+      operation: {
+        type: 'string',
+        enum: ['session', 'agent', 'domain', 'task'],
+        description: 'Scope of analysis'
+      },
+      timeframe: {
+        type: 'string',
+        enum: ['1h', '24h', '7d', '30d'],
+        description: 'Time period to analyze'
+      },
+      agentId: {
+        type: 'string',
+        description: 'Specific agent to analyze (optional)'
+      },
+      domain: {
+        type: 'string',
+        description: 'Specific domain to analyze (optional)'
+      }
+    },
+    required: ['operation']
+  },
+
+  async execute(args: TokenUsageArgs): Promise<TokenUsageResult> {
+    const collector = TokenMetricsCollector.getInstance();
+
+    switch (args.operation) {
+      case 'session':
+        return collector.getSessionSummary(args.timeframe);
+      case 'agent':
+        return collector.getAgentMetrics(args.agentId, args.timeframe);
+      case 'domain':
+        return collector.getDomainMetrics(args.domain, args.timeframe);
+      case 'task':
+        return collector.getTaskMetrics(args.timeframe);
+    }
+  }
+};
+
+// Example output:
+{
+  "summary": {
+    "totalTokens": 45230,
+    "totalCost": "$0.68",
+    "tokensSaved": 12450,
+    "savingsPercentage": 27.5
+  },
+  "breakdown": {
+    "byAgent": {
+      "qe-test-generator": { tokens: 15200, cost: "$0.23" },
+      "qe-coverage-analyzer": { tokens: 8900, cost: "$0.13" }
+    },
+    "byDomain": {
+      "test-generation": { tokens: 18500 },
+      "coverage-analysis": { tokens: 12000 }
+    }
+  },
+  "optimization": {
+    "patternsReused": 34,
+    "cacheHits": 89,
+    "earlyExits": 12,
+    "recommendations": [
+      "High reuse potential for test-generation patterns (>0.85 similarity)",
+      "Consider batching coverage-analysis requests"
+    ]
+  }
+}
+```
+
+### 4. Token-Aware Pattern Store
+
+```typescript
+// Update v3/src/learning/pattern-store.ts
+
+export interface PatternStoreConfig {
+  // ... existing config ...
+
+  // NEW: Token optimization config
+  tokenTracking: {
+    enabled: boolean;
+    trackInputOutput: boolean;    // Track input/output separately
+    estimateCosts: boolean;       // Calculate cost estimates
+    costPerInputToken: number;    // e.g., 0.003 / 1000
+    costPerOutputToken: number;   // e.g., 0.015 / 1000
+  };
+
+  // NEW: Reuse optimization config
+  reuseOptimization: {
+    enabled: boolean;
+    minSimilarityForReuse: number;  // e.g., 0.85
+    minSuccessRateForReuse: number; // e.g., 0.90
+    maxAgeForReuse: number;         // e.g., 7 days
+  };
+}
+
+// Pattern search now returns token optimization hints
+export interface PatternSearchResult {
+  pattern: QEPattern;
+  similarity: number;
+
+  // NEW: Reuse recommendation
+  canReuse: boolean;
+  estimatedTokenSavings: number;
+  reuseConfidence: number;
+}
+```
+
+### 5. Early Exit Optimizer
+
+```typescript
+// v3/src/optimization/early-exit-token-optimizer.ts
+
+export class EarlyExitTokenOptimizer {
+  private patternStore: PatternStore;
+  private minConfidenceForExit = 0.85;
+
+  /**
+   * Check if we can skip LLM call by reusing a pattern
+   */
+  async checkEarlyExit(task: QETask): Promise<EarlyExitResult> {
+    // Search for highly similar patterns
+    const matches = await this.patternStore.search(task.description, {
+      limit: 5,
+      minSimilarity: this.minConfidenceForExit,
+      onlySuccessful: true
+    });
+
+    if (matches.length === 0) {
+      return { canExit: false };
+    }
+
+    const bestMatch = matches[0];
+
+    // Check if pattern is reusable
+    if (bestMatch.canReuse && bestMatch.reuseConfidence > 0.9) {
+      return {
+        canExit: true,
+        reusedPattern: bestMatch.pattern,
+        estimatedTokensSaved: bestMatch.pattern.tokensUsed || 0,
+        confidence: bestMatch.reuseConfidence
+      };
+    }
+
+    return { canExit: false };
+  }
+}
+```
+
+### 6. Integration with Existing MetricsCollector
+
+```typescript
+// Update v3/src/mcp/metrics/metrics-collector.ts
+
+export interface TaskMetric {
+  taskId: string;
+  agentId: string;
+  startTime: [number, number];
+  endTime?: [number, number];
+  durationMs?: number;
+  success: boolean;
+  retries: number;
+
+  // NEW: Token tracking
+  tokenUsage?: TokenUsage;
+  patternReused?: boolean;
+  tokensSaved?: number;
+}
+
+// Add methods to MetricsCollectorImpl
+class MetricsCollectorImpl {
+  // ... existing methods ...
+
+  /**
+   * Record token usage for a completed task
+   */
+  recordTokenUsage(taskId: string, usage: TokenUsage): void {
+    const task = this.completedTasks.get(taskId);
+    if (task) {
+      task.tokenUsage = usage;
+      this.updateAgentTokenMetrics(task.agentId, usage);
+      this.updateDomainTokenMetrics(task.domain, usage);
+    }
+  }
+
+  /**
+   * Record pattern reuse (tokens saved)
+   */
+  recordPatternReuse(taskId: string, tokensSaved: number): void {
+    const task = this.completedTasks.get(taskId);
+    if (task) {
+      task.patternReused = true;
+      task.tokensSaved = tokensSaved;
+      this.totalTokensSaved += tokensSaved;
+    }
+  }
+
+  /**
+   * Get token efficiency metrics
+   */
+  getTokenEfficiency(): TokenEfficiencyReport {
+    return {
+      totalTokensUsed: this.totalTokens,
+      totalTokensSaved: this.totalTokensSaved,
+      savingsPercentage: (this.totalTokensSaved /
+        (this.totalTokens + this.totalTokensSaved)) * 100,
+      patternReuseRate: this.patternsReused / this.totalTasks,
+      averageTokensPerTask: this.totalTokens / this.totalTasks
+    };
+  }
+}
+```
+
+### 7. CLI Commands
+
+```bash
+# Check token usage for current session
+aqe token-usage --period 24h
+
+# Get breakdown by agent
+aqe token-usage --by-agent
+
+# Get breakdown by domain
+aqe token-usage --by-domain
+
+# Export detailed report
+aqe token-usage --period 7d --export tokens.csv
+
+# Show optimization recommendations
+aqe token-usage --recommendations
+```
+
+## Token Reduction Strategies
+
+### Strategy 1: Pattern Reuse (Target: -25%)
+
+```typescript
+// Before executing a task, check for reusable patterns
+const earlyExitCheck = await tokenOptimizer.checkEarlyExit(task);
+if (earlyExitCheck.canExit) {
+  // Skip LLM call, use cached pattern
+  return applyPattern(earlyExitCheck.reusedPattern, task);
+}
+// Otherwise, execute normally and store pattern
+```
+
+### Strategy 2: Batch Operations (Target: -15%)
+
+```typescript
+// Instead of N separate file reads, batch them
+const files = await batchFileRead(paths);  // 1 operation
+// Instead of
+for (const path of paths) {
+  await readFile(path);  // N operations
+}
+```
+
+### Strategy 3: Context Compression (Target: -10%)
+
+```typescript
+// Provide focused context, not entire files
+const context = await getRelevantContext(task, {
+  maxTokens: 2000,
+  focusAreas: ['error handling', 'edge cases'],
+  excludeBoilerplate: true
+});
+```
+
+### Strategy 4: Response Caching (Target: -8%)
+
+```typescript
+// Cache responses for identical/similar queries
+const cacheKey = hashQuery(task.description);
+const cached = await responseCache.get(cacheKey);
+if (cached && cached.age < 5 * 60 * 1000) {  // 5 min TTL
+  return cached.response;
+}
+```
+
+## Performance Targets
+
+| Metric | Baseline | Target | Mechanism |
+|--------|----------|--------|-----------|
+| Token reduction | 0% | -25% | Pattern reuse + caching |
+| Cost savings | 0% | -30% | All strategies combined |
+| Cache hit rate | 0% | >40% | Smart caching |
+| Pattern reuse rate | 0% | >35% | HNSW similarity search |
+| Early exit rate | 0% | >15% | High-confidence patterns |
+
+## Migration Path
+
+### Phase 1: Token Tracking (Week 1)
+1. Add `TokenUsage` fields to `TaskMetric`
+2. Implement `TokenMetricsCollector`
+3. Add token tracking to MCP handlers
+
+### Phase 2: MCP Tools (Week 2)
+1. Implement `token_usage` MCP tool
+2. Add CLI commands
+3. Create dashboard integration
+
+### Phase 3: Pattern Optimization (Week 3)
+1. Add token fields to `QEPattern`
+2. Implement `EarlyExitTokenOptimizer`
+3. Enable pattern reuse
+
+### Phase 4: Advanced Reduction (Week 4)
+1. Implement batch operations
+2. Add context compression
+3. Enable response caching
+
+## Consequences
+
+### Positive
+- **Visibility**: Know exactly where tokens are spent
+- **Cost control**: Predict and optimize API costs
+- **Performance**: Faster responses via caching/reuse
+- **Sustainability**: -25% token reduction = -25% API costs
+- **Learning**: Patterns improve over time
+
+### Negative
+- **Complexity**: Additional tracking infrastructure
+- **Storage**: Pattern store grows over time
+- **Cache staleness**: Risk of stale cached responses
+
+### Mitigation
+- **Pruning**: Auto-prune low-value patterns (like agentic-flow's `BatchOperations.pruneData()`)
+- **TTL**: Time-based cache expiration
+- **Confidence thresholds**: Only reuse high-confidence patterns
+
+## Related Work
+
+### From agentic-flow
+- `ReasoningBankController` - Pattern storage with `tokensUsed` tracking
+- `BatchOperations.pruneData()` - Data hygiene for pattern store
+- `TelemetryManager` - OpenTelemetry integration for metrics
+- Reasoning agents (`adaptive-learner`, `pattern-matcher`) - Token reduction strategies
+
+### V3 ADRs
+- ADR-021: QE ReasoningBank - Base pattern learning (extend with tokens)
+- ADR-033: Early Exit Testing - Apply to token optimization
+- ADR-038: Memory Unification - Integrate token tracking
+- ADR-040: Agentic-Flow Integration - Parent ADR for this work
+
+## References
+
+- [agentic-flow Token Efficiency Docs](https://github.com/ruvnet/agentic-flow/blob/main/.claude/commands/analysis/token-efficiency.md)
+- [ReasoningBank Paper](https://arxiv.org/html/2509.25140v1) - Closed-loop learning with 32.3% token reduction
+- [OpenTelemetry for Node.js](https://opentelemetry.io/docs/languages/js/) - Metrics infrastructure