@crownpeak/dqm-react-component-dev-mcp 1.2.0

package/data/probe/npm/CONTEXT_COMPACTION.md

@@ -0,0 +1,303 @@
+# Context Window Compaction
+
+ProbeAgent includes automatic context window compaction to handle scenarios where conversations exceed the AI model's token limits.
+
+## Overview
+
+When interacting with AI models through ProbeAgent, conversations can grow large over multiple turns. If the context window limit is exceeded, ProbeAgent automatically compacts the conversation history by intelligently removing intermediate reasoning steps while preserving essential information.
+
+## How It Works
+
+### Conversation Structure
+
+ProbeAgent conversations follow this structure:
+
+```
+<user message>
+  ↓
+<internal agentic monologue> (thinking, tool planning)
+  ↓
+<tool execution> (search, extract, etc.)
+  ↓
+<tool result>
+  ↓
+<final agent answer>
+```
+
+A "segment" consists of:
+- User message (starting point)
+- 0+ assistant monologue messages (internal reasoning with `<thinking>` tags, tool calls)
+- Final answer (tool results or `attempt_completion`)
+
+### Compaction Strategy
+
+When a context limit error is detected, ProbeAgent:
+
+1. **Identifies segments** in the conversation history
+2. **Keeps all user messages** - preserves original questions/requests
+3. **Keeps all final answers** - preserves tool results and completions
+4. **Removes intermediate monologues** from older segments (thinking, tool planning)
+5. **Preserves recent segments** completely (configurable, defaults to last 2 segments)
+6. **Retries the request** with compacted messages
+
+This ensures:
+- ✅ No loss of user intent
+- ✅ No loss of completed work
+- ✅ Continuation from current state
+- ✅ Significant token reduction
+
+## Error Detection
+
+The compactor automatically detects context limit errors from various AI providers:
+
+- **Anthropic**: "context_length_exceeded", "prompt is too long"
+- **OpenAI**: "maximum context length is X tokens"
+- **Google/Gemini**: "input token count exceeds limit"
+- **Generic patterns**: "tokens exceed", "too long", "over limit", etc.
+
+## Manual Compaction
+
+You can manually compact conversation history at any time using the `compactHistory()` method:
+
+```javascript
+const agent = new ProbeAgent({
+  sessionId: 'my-session',
+  path: './my-project'
+});
+
+// ... have some conversations ...
+
+// Manually compact history
+const stats = await agent.compactHistory();
+
+console.log(`Removed ${stats.removed} messages`);
+console.log(`Token savings: ${stats.tokensSaved}`);
+
+// Compact with custom options
+const stats2 = await agent.compactHistory({
+  keepLastSegment: true,
+  minSegmentsToKeep: 2  // Keep last 2 segments fully
+});
+```
+
+This is useful when:
+- You want to proactively reduce context before hitting limits
+- You're monitoring token usage and want to optimize
+- You want to clean up history at specific checkpoints
+- Testing or debugging compaction behavior
+
+## Configuration
+
+Context compaction is enabled **automatically** when context limits are exceeded. No configuration required!
+
+For manual compaction or advanced use:
+
+```javascript
+// Manual compaction
+await agent.compactHistory({
+  keepLastSegment: true,    // Keep the most recent segment intact
+  minSegmentsToKeep: 1      // Number of recent segments to preserve fully
+});
+```
+
+### Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `keepLastSegment` | `true` | Always preserve the active/most recent segment |
+| `minSegmentsToKeep` | `1` | Number of recent segments to keep fully (including monologues) |
+
+## Example
+
+### Before Compaction (150+ messages)
+
+```
+[System message]
+Turn 1: User → Assistant(thinking) → Assistant(search) → Tool result
+Turn 2: User → Assistant(thinking) → Assistant(extract) → Tool result
+Turn 3: User → Assistant(thinking) → Assistant(implement) → Tool result
+Turn 4: User → Assistant(thinking) → Assistant(bash) → Tool result
+Turn 5: User → Assistant(thinking) → Assistant(search) ← ACTIVE
+```
+
+### After Compaction (~50 messages)
+
+```
+[System message]
+Turn 1: User → Tool result (monologue removed)
+Turn 2: User → Tool result (monologue removed)
+Turn 3: User → Tool result (monologue removed)
+Turn 4: User → Tool result (monologue removed)
+Turn 5: User → Assistant(thinking) → Assistant(search) (preserved, active)
+```
+
+**Result**: ~66% reduction in messages, ~60% reduction in tokens
+
+Only the **active segment** (Turn 5) retains its full internal monologue. All completed segments (Turns 1-4) are compressed to just User → Final Result.
+
+## Statistics
+
+When compaction occurs, ProbeAgent logs statistics:
+
+```
+[INFO] Context window limit exceeded. Compacting conversation...
+[INFO] Removed 42 messages (66.7% reduction)
+[INFO] Estimated token savings: 8450 tokens
+```
+
+With debug mode enabled:
+
+```javascript
+const agent = new ProbeAgent({
+  debug: true,
+  // ... other options
+});
+```
+
+You'll see detailed compaction information:
+
+```
+[DEBUG] Compaction stats: {
+  originalCount: 63,
+  compactedCount: 21,
+  removed: 42,
+  reductionPercent: 66.7,
+  originalTokens: 12800,
+  compactedTokens: 4350,
+  tokensSaved: 8450
+}
+```
+
+## API Reference
+
+### `agent.compactHistory(options)`
+
+Manually compact conversation history.
+
+**Parameters:**
+- `options` (Object, optional)
+  - `keepLastSegment` (boolean, default: `true`) - Preserve active segment
+  - `minSegmentsToKeep` (number, default: `1`) - Number of recent segments to keep fully
+
+**Returns:** Promise<Object> - Compaction statistics
+```javascript
+{
+  originalCount: 63,          // Original message count
+  compactedCount: 21,         // Compacted message count
+  removed: 42,                // Messages removed
+  reductionPercent: 66.7,     // Percentage reduction
+  originalTokens: 12800,      // Estimated original tokens
+  compactedTokens: 4350,      // Estimated compacted tokens
+  tokensSaved: 8450           // Estimated tokens saved
+}
+```
+
+**Example:**
+```javascript
+const stats = await agent.compactHistory();
+console.log(`Saved ${stats.tokensSaved} tokens`);
+```
+
+## Testing
+
+The context compaction functionality is fully tested. Run tests with:
+
+```bash
+npm test -- contextCompactor.test.js
+npm test -- agent-compact-history.test.js
+```
+
+Test coverage includes:
+- Error detection across multiple AI providers
+- Message segment identification
+- Compaction logic with various configurations
+- Token estimation and statistics
+- Real-world conversation scenarios
+- Manual compaction API
+
+## Technical Details
+
+### Architecture
+
+Context compaction uses a **two-layer error handling strategy**:
+
+1. **Transient Errors** (handled by `RetryManager`/`FallbackManager`)
+   - Rate limits (429)
+   - Network issues (ECONNRESET, timeouts)
+   - Server errors (500, 502, 503, 504)
+   - Retries with exponential backoff
+   - Falls back to alternative providers
+
+2. **Context Limit Errors** (handled by compaction)
+   - Context window exceeded
+   - Token limit reached
+   - Compacts once per iteration
+   - Retries with reduced message set
+   - Separate from transient retry loop
+   - Fails immediately if no messages can be removed
+
+This separation ensures:
+- Transient errors get multiple retries
+- Context errors trigger structural changes (compaction)
+- No interference between retry strategies
+- No infinite loops (compaction attempted only once per iteration)
+
+### Files
+
+- **`src/agent/contextCompactor.js`** - Core compaction logic
+- **`src/agent/ProbeAgent.js`** - Integration with error handling and API
+  - Lines 1453-1565: Automatic compaction retry loop with infinite loop protection
+  - Lines 2421-2482: Manual compaction method
+- **`tests/contextCompactor.test.js`** - Core compaction test suite
+- **`tests/agent-compact-history.test.js`** - Manual API test suite
+
+### Functions
+
+#### `isContextLimitError(error)`
+Detects if an error indicates context window overflow.
+
+#### `identifyMessageSegments(messages)`
+Analyzes conversation history and identifies logical segments.
+
+#### `compactMessages(messages, options)`
+Performs intelligent compaction by removing intermediate monologues.
+
+#### `calculateCompactionStats(originalMessages, compactedMessages)`
+Computes reduction statistics and token savings.
+
+#### `handleContextLimitError(error, messages, options)`
+Main handler that orchestrates detection and compaction.
+
+#### `agent.compactHistory(options)`
+Public API method for manual history compaction.
+
+## Limitations
+
+1. **Minimum context**: Compaction cannot help if even the compacted history exceeds limits. The system will fail with a clear error message if no messages can be removed.
+2. **System message**: System messages are never removed (contain critical instructions)
+3. **Token estimation**: Token counts are approximations (1 token ≈ 4 characters)
+4. **Recent segments**: Always preserves configured minimum segments to maintain context quality
+5. **Single retry**: Compaction is attempted only once per iteration to prevent infinite loops
+
+## Best Practices
+
+1. **Use reasonable iteration limits**: Set `maxIterations` appropriately to avoid excessive history
+2. **Monitor logs**: Check compaction logs to understand when/why it triggers
+3. **Enable debug mode**: For development, use `debug: true` to see detailed statistics
+4. **Test edge cases**: Ensure your application handles compaction gracefully
+
+## Future Enhancements
+
+Potential improvements being considered:
+
+- Smarter segment importance scoring (keep more important monologues)
+- Configurable compaction aggressiveness
+- Semantic compression using embeddings
+- Progressive compaction (multiple levels)
+- Custom compaction strategies per use case
+
+## Related Documentation
+
+- [ProbeAgent README](./README.md) - Main ProbeAgent documentation
+- [Retry and Fallback](./npm/src/agent/RetryManager.js) - Error handling and retry logic
+- [MCP Integration](./MCP_INTEGRATION_SUMMARY.md) - Model Context Protocol support

package/data/probe/npm/DELEGATE_TOOL_README.md

@@ -0,0 +1,166 @@
+# Delegate Tool
+
+The delegate tool is used **automatically by AI agents** within the agentic loop to delegate big distinct tasks to specialized probe subagents. When an AI agent encounters complex multi-part requests, it should automatically identify opportunities for task separation and use delegation without explicit user instruction.
+
+## Agentic Usage Pattern
+
+The delegate tool is designed for **automatic use by AI agents**, not direct developer calls. The AI agent recognizes when a user's request involves multiple large, distinct components and automatically breaks them down into focused, parallel tasks.
+
+### Automatic Task Recognition
+
+When users make complex requests, the AI agent should automatically:
+
+1. **Identify** multiple distinct components in the request
+2. **Separate** them into self-contained tasks  
+3. **Delegate** each task to a specialized subagent
+4. **Combine** results from all subagents
+
+### XML Tool Format (AI Agent Usage)
+
+```xml
+<delegate>
+<task>Analyze all authentication and authorization code in the codebase for security vulnerabilities and provide specific remediation recommendations</task>
+</delegate>
+
+<delegate>
+<task>Review database queries and API endpoints for performance bottlenecks and suggest optimization strategies</task>
+</delegate>
+```
+
+## How It Works
+
+1. **Task Definition**: You provide a complete, self-contained task description
+2. **SDK-Based Agent Creation**: The delegate tool creates a new ProbeAgent instance in-process with:
+   - Default 'code-researcher' prompt (not inherited from parent)
+   - Schema validation disabled for simpler responses
+   - Mermaid validation disabled for faster processing
+   - Delegation explicitly disabled to prevent recursion
+3. **Independent Execution**: The subagent processes your task in isolation within the same process
+4. **Response Waiting**: The main agent awaits the subagent's completion (Promise-based)
+5. **Result Return**: The delegate tool returns the subagent's clean response
+
+## Subagent Environment
+
+Each delegated task runs in a clean environment with automatic configuration:
+- **Prompt**: Automatically uses the default `code-researcher` prompt, regardless of the parent agent's prompt
+- **Validation**: Schema and Mermaid validation are automatically disabled for faster, simpler responses
+- **Iterations**: Automatically limited to remaining parent iterations to respect global limits
+- **Isolation**: Each subagent runs independently without inheriting parent context or history
+- **No Recursion**: Delegation is explicitly disabled in subagents (`enableDelegate: false`) to prevent infinite delegation chains
+- **Inherited Config**: Path, provider, and model are inherited from parent for consistency
+- **In-Process**: Runs within the same Node.js process using the ProbeAgent SDK (no process spawning)
+
+All these settings are applied automatically - no manual configuration needed.
+
+## Security and Resource Management
+
+The delegate tool includes built-in protection against resource exhaustion:
+
+### Concurrent Delegation Limits
+
+- **Global Limit**: Maximum of 3 concurrent delegations by default (configurable via `MAX_CONCURRENT_DELEGATIONS` environment variable)
+- **Per-Session Limit**: Maximum of 10 delegations per parent session (configurable via `MAX_DELEGATIONS_PER_SESSION` environment variable)
+- **Automatic Cleanup**: Counters are automatically decremented when delegations complete, fail, or timeout
+
+### Recursion Prevention
+
+- Subagents cannot spawn their own subagents - the `delegate` tool is explicitly disabled in delegated sessions
+- This prevents infinite delegation chains and ensures bounded resource usage
+
+### Rate Limiting
+
+Delegation requests that exceed limits will fail immediately with clear error messages:
+- `Maximum concurrent delegations (N) reached. Please wait for some delegations to complete.`
+- `Maximum delegations per session (N) reached for session {id}`
+
+### Environment Variables
+
+```bash
+# Set custom limits (optional)
+export MAX_CONCURRENT_DELEGATIONS=5
+export MAX_DELEGATIONS_PER_SESSION=20
+```
+
+## Configuration Options
+
+- `task` (required): The specific task to delegate. Should be a complete, self-contained task that can be executed independently.
+- `timeout` (optional, default: 300): Maximum time to wait for the subagent in seconds
+- `debug` (optional, default: false): Enable debug logging for troubleshooting
+- `parentSessionId` (optional): Parent session ID for tracking per-session limits (automatically provided by ProbeAgent)
+- `currentIteration` (optional): Current iteration count (automatically provided by ProbeAgent)
+- `maxIterations` (optional): Maximum iterations allowed (automatically provided by ProbeAgent)
+
+## Error Handling
+
+The delegate tool includes comprehensive error handling:
+- Process spawn failures
+- Timeouts
+- Agent execution errors
+- Empty responses
+
+## Agentic Use Cases
+
+The AI agent automatically uses delegation for:
+
+1. **Task Separation**: When user requests involve multiple distinct domains
+2. **Complex Analysis**: Breaking comprehensive analysis into specialized areas  
+3. **Parallel Processing**: Distributing work across multiple focused subagents
+4. **Domain Expertise**: Delegating to subagents optimized for specific areas
+5. **Large Scope**: Decomposing overwhelming requests into manageable pieces
+
+## Automatic Task Separation Examples
+
+**User Request**: "Analyze my entire codebase for security, performance, and maintainability issues"
+
+**AI Agent automatically separates into**:
+```xml
+<delegate>
+<task>Analyze all authentication, authorization, input validation, and cryptographic code for security vulnerabilities and provide specific remediation recommendations with code examples</task>
+</delegate>
+
+<delegate>
+<task>Review all database queries, API endpoints, algorithms, and resource usage patterns for performance bottlenecks and suggest concrete optimization strategies</task>
+</delegate>
+
+<delegate>  
+<task>Examine code structure, design patterns, documentation, and maintainability across all modules and provide refactoring recommendations</task>
+</delegate>
+```
+
+**User Request**: "Help me understand how the payment system works"
+
+**AI Agent automatically separates into**:
+```xml
+<delegate>
+<task>Analyze the payment processing flow including transaction handling, validation, and error management to explain the complete payment workflow</task>
+</delegate>
+
+<delegate>
+<task>Examine payment security measures including encryption, authentication, fraud detection, and compliance implementations</task>
+</delegate>
+
+<delegate>
+<task>Review payment database schema, data models, and storage patterns to explain how payment data is structured and managed</task>
+</delegate>
+```
+
+## Integration
+
+The delegate tool is integrated into AI agent systems through:
+- **ACP (Agent Communication Protocol)**: For advanced agent systems
+- **XML Tool Parsing**: For AI agent tool call recognition
+- **ProbeAgent Class**: For programmatic agent implementations  
+- **Vercel AI SDK**: For AI framework compatibility
+- **LangChain Tools**: Via Vercel compatibility layer
+
+## AI Agent Decision Making
+
+The AI agent should automatically use delegation when it recognizes:
+
+- **Multi-domain requests**: User asks about multiple technical areas
+- **Large scope tasks**: Requests that would benefit from parallel processing
+- **Specialized expertise**: Tasks requiring focused domain knowledge
+- **Complex analysis**: Comprehensive reviews that can be divided
+- **Performance optimization**: When parallel execution improves response time
+
+The delegate tool operates transparently within the agentic loop - users don't need to know it's being used. They simply get faster, more focused responses to complex requests.