mcp-agent-foundry 1.3.1 → 2.0.0

package/README.md

@@ -24,6 +24,11 @@ Agent Foundry extends Claude Code with multi-provider AI orchestration and git w
 - **State Persistence** - Survives server restarts with automatic state recovery
 - **Crash Recovery** - Orphan detection, stuck worktree handling, and cleanup commands
 - **Intelligent Failover** - Automatic provider switching with health tracking, circuit breakers, and cost-aware routing
+- **Context Manager** - Token tracking per provider/model with 40+ model limits, auto-compact at 90%, blocking at 98%
+- **Hooks System** - 9 event types for customizing agent behavior with variable substitution and hot-reload
+- **Background Task Runner** - Queue-based async task execution with AbortController cancellation and progress tracking
+- **MCP Auto Mode** - Intelligent tool selection under context pressure with priority-based deferral
+- **Skills Hot-Reload** - Custom skill definitions with auto-discovery from user and project directories
 
 ---
 
@@ -153,6 +158,166 @@ Agent Foundry includes automatic provider failover with health tracking:
 - **Circuit Breakers** - Prevents cascading failures
 - **Cost-Aware Routing** - Optionally prefer cheaper providers
 - **Configurable Error Codes** - Define which errors trigger failover
+- **Rate Limit Warnings** - Proactive warnings at 70% capacity with header parsing for OpenAI/Anthropic/generic formats
+
+---
+
+## Hooks System
+
+Agent Foundry provides a flexible hooks system that lets you customize agent behavior by running commands at key lifecycle events.
+
+### Supported Events
+
+| Event | Description |
+|-------|-------------|
+| `PreToolUse` | Before a tool is invoked |
+| `PostToolUse` | After a tool completes |
+| `Setup` | During agent initialization |
+| `ToolError` | When a tool encounters an error |
+| `TaskStart` | When a task begins execution |
+| `TaskComplete` | When a task finishes successfully |
+| `TaskFail` | When a task fails |
+| `ProviderError` | When an AI provider returns an error |
+| `Failover` | When switching to a fallback provider |
+
+### Configuration
+
+```yaml
+# In ~/.config/agent-foundry/config.yaml
+hooks:
+  enabled: true
+  configPath: ~/.agent-foundry/hooks.yaml
+  events:
+    PreToolUse:
+      - command: "echo 'Tool: ${TOOL_NAME}'"
+        timeout_ms: 5000
+        filter:
+          tools: ["execute_task", "invoke_agent"]
+    TaskComplete:
+      - command: "./notify.sh ${TASK_ID}"
+        timeout_ms: 10000
+    ProviderError:
+      - command: "logger -t agent-foundry 'Provider ${PROVIDER} failed: ${ERROR}'"
+```
+
+### Variable Substitution
+
+Hooks support variable substitution with context-specific values:
+- `${TOOL_NAME}` - Name of the tool being invoked
+- `${TASK_ID}` - Current task identifier
+- `${PROVIDER}` - AI provider name
+- `${ERROR}` - Error message (for error events)
+- `${RESULT}` - Result data (for post-completion events)
+
+### Features
+
+- **Filtering** - Run hooks only for specific tools, providers, or roles
+- **Timeouts** - Configurable per-hook timeouts prevent blocking
+- **Hot-Reload** - Changes to hooks.yaml are picked up automatically
+
+---
+
+## Skills Hot-Reload
+
+Define custom skills that extend agent capabilities with automatic discovery and hot-reload.
+
+### Skill Locations
+
+Skills are loaded from:
+1. `~/.agent-foundry/skills/` - User-level skills (shared across projects)
+2. `.agent-foundry/skills/` - Project-level skills (project-specific)
+
+### Skill Definition
+
+```yaml
+# ~/.agent-foundry/skills/code-review.yaml
+name: code-review
+description: Comprehensive code review skill
+context_mode: inherit  # inherit | fork | isolated
+
+prompts:
+  system: |
+    You are an expert code reviewer focusing on:
+    - Security vulnerabilities
+    - Performance issues
+    - Best practices
+    - Code clarity
+
+triggers:
+  - pattern: "review this code"
+  - pattern: "code review"
+
+actions:
+  - type: invoke_agent
+    role: reviewer
+```
+
+### Context Modes
+
+| Mode | Description |
+|------|-------------|
+| `inherit` | Skill runs in the parent agent's context |
+| `fork` | Skill gets a copy of the parent context |
+| `isolated` | Skill runs with a fresh, empty context |
+
+### Features
+
+- **Auto-Discovery** - Skills are automatically loaded on startup
+- **Hot-Reload** - Changes to skill files are picked up without restart
+- **Priority Ordering** - Project skills override user skills with the same name
+
+---
+
+## Context Manager
+
+Agent Foundry includes intelligent context management to prevent token limit issues and optimize AI provider usage.
+
+### Features
+
+- **Token Tracking** - Tracks token usage per provider/model with 40+ pre-configured model limits
+- **Auto-Compact** - Automatically compacts context when reaching 90% capacity
+- **Blocking Threshold** - Blocks new requests at 98% to prevent errors
+- **Visualization** - Context usage visible in debug logs and status commands
+
+### Configuration
+
+```yaml
+# In ~/.config/agent-foundry/config.yaml
+context:
+  autoCompact: true
+  compactThreshold: 0.9    # Trigger compaction at 90%
+  blockThreshold: 0.98     # Block new requests at 98%
+```
+
+### Supported Model Limits
+
+Pre-configured limits for 40+ models including:
+- OpenAI: GPT-4o (128K), GPT-4 Turbo (128K), o1 (200K)
+- Anthropic: Claude 3.5/4 (200K)
+- Google: Gemini 2.5 Pro (1M), Gemini 2.5 Flash (1M)
+- DeepSeek: DeepSeek R1 (64K), DeepSeek Chat (128K)
+- And many more...
+
+---
+
+## MCP Auto Mode
+
+When context pressure builds, MCP Auto Mode intelligently manages tool availability.
+
+### How It Works
+
+1. **Context Budget** - Tool descriptions are deferred when exceeding 10% of context budget
+2. **Priority-Based Selection** - High-priority tools remain available; lower-priority tools are deferred
+3. **Automatic Restoration** - Deferred tools become available again when context decreases
+
+### Tool Priorities
+
+| Priority | Tools |
+|----------|-------|
+| Critical | `invoke_agent`, `execute_task` |
+| High | `list_agents`, `get_task_status` |
+| Medium | `execute_pipeline`, `compare_agents` |
+| Low | `cleanup_worktrees`, `list_worktrees` |
 
 ---
 
@@ -174,6 +339,7 @@ Agent Foundry includes automatic provider failover with health tracking:
 | `cleanup_worktrees` | Clean up stale or orphaned worktrees |
 | `resolve_conflicts` | Resolve merge conflicts using various strategies |
 | `get_worktree_status` | Get detailed status of a specific worktree |
+| `delete_task` | Delete a task from the queue |
 
 ---
 
@@ -279,6 +445,31 @@ worktrees:
     enabled: true
     minAvailable: 2
     maxSize: 5
+
+# Hooks configuration
+hooks:
+  enabled: true
+  configPath: ~/.agent-foundry/hooks.yaml
+  events:
+    PreToolUse:
+      - command: "echo 'Tool: ${TOOL_NAME}'"
+        timeout_ms: 5000
+    TaskComplete:
+      - command: "./notify.sh ${TASK_ID}"
+
+# Skills configuration
+skills:
+  enabled: true
+  directories:
+    - ~/.agent-foundry/skills
+    - .agent-foundry/skills
+  hotReload: true
+
+# Context management
+context:
+  autoCompact: true
+  compactThreshold: 0.9    # Auto-compact at 90% capacity
+  blockThreshold: 0.98     # Block new requests at 98% capacity
 ```
 
 ### Environment Variables
@@ -305,6 +496,11 @@ export FIREWORKS_API_KEY="..."
 
 # Web Search
 export PERPLEXITY_API_KEY="pplx-..."
+
+# Debug/Observability
+export AGENT_FOUNDRY_DEBUG=true                    # Enable debug logging
+export AGENT_FOUNDRY_DEBUG_FILE=/path/to/debug.log # Write debug logs to file
+export AGENT_FOUNDRY_DEBUG_VERBOSE=true            # Include verbose details
 ```
 
 ---
@@ -455,7 +651,8 @@ agent-foundry/
 │   ├── cli/                  # CLI command implementations
 │   ├── mcp/
 │   │   ├── tools/            # MCP tool implementations
-│   │   └── transport/        # stdio transport
+│   │   ├── transport/        # stdio transport
+│   │   └── auto-mode.ts      # MCP Auto Mode - intelligent tool selection
 │   ├── worktrees/
 │   │   ├── manager.ts        # Worktree lifecycle
 │   │   ├── branch.ts         # Branch operations
@@ -482,6 +679,18 @@ agent-foundry/
 │   │   ├── orchestrator.ts   # Retry and failover logic
 │   │   ├── health-tracker.ts # Provider health monitoring
 │   │   └── pricing.ts        # Cost-aware routing
+│   ├── background/           # Background task execution
+│   │   └── task-runner.ts    # Queue-based async task runner
+│   ├── hooks/                # Hook system
+│   │   ├── hook-executor.ts  # Hook command execution
+│   │   └── hook-manager.ts   # Hook registration and dispatch
+│   ├── skills/               # Skills hot-reload
+│   │   ├── skill-loader.ts   # Skill file discovery and parsing
+│   │   ├── skill-executor.ts # Skill action execution
+│   │   └── hot-reloader.ts   # File watcher for hot-reload
+│   ├── observability/        # Logging and debugging
+│   │   ├── logger.ts         # Structured logging
+│   │   └── debug-logger.ts   # Debug output with environment control
 │   ├── persistence/          # State persistence
 │   ├── config/               # Configuration management
 │   └── router/               # Routing engine
@@ -516,6 +725,28 @@ pnpm lint:fix
 pnpm typecheck
 ```
 
+### Debug Logging
+
+Agent Foundry provides enhanced debug logging controlled via environment variables:
+
+```bash
+# Enable debug logging
+export AGENT_FOUNDRY_DEBUG=true
+
+# Write debug logs to a file
+export AGENT_FOUNDRY_DEBUG_FILE=/path/to/debug.log
+
+# Include verbose details (API payloads, full stack traces)
+export AGENT_FOUNDRY_DEBUG_VERBOSE=true
+```
+
+Debug logging includes:
+- Provider API calls and responses
+- Failover decisions and health tracking
+- Hook execution and timing
+- Skill loading and hot-reload events
+- Context manager token tracking
+
 ---
 
 ## Security

package/dist/background/index.d.ts

@@ -0,0 +1,33 @@
+/**
+ * Background Module
+ *
+ * Provides background task execution for non-blocking agent invocations
+ * and pipeline steps. Supports concurrency control, cancellation,
+ * progress tracking, and event-based notifications.
+ *
+ * @example
+ * ```typescript
+ * import { BackgroundTaskRunner } from './background/index.js';
+ *
+ * const runner = new BackgroundTaskRunner({ maxConcurrent: 3 }, logger);
+ *
+ * // Submit a background agent invocation
+ * const taskId = await runner.run(
+ *   'agent_invocation',
+ *   'Review authentication code',
+ *   async () => {
+ *     return await orchestrator.executeWithFailover('reviewer', messages, options);
+ *   },
+ *   { role: 'reviewer', provider: 'openai' }
+ * );
+ *
+ * // Check status later
+ * const task = runner.getTask(taskId);
+ *
+ * // Or wait for completion
+ * const completed = await runner.waitFor(taskId, 60000);
+ * ```
+ */
+export * from './types.js';
+export { BackgroundTaskRunner } from './task-runner.js';
+//# sourceMappingURL=index.d.ts.map

package/dist/background/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/background/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6BG;AAEH,cAAc,YAAY,CAAC;AAC3B,OAAO,EAAE,oBAAoB,EAAE,MAAM,kBAAkB,CAAC"}

package/dist/background/index.js

@@ -0,0 +1,33 @@
+/**
+ * Background Module
+ *
+ * Provides background task execution for non-blocking agent invocations
+ * and pipeline steps. Supports concurrency control, cancellation,
+ * progress tracking, and event-based notifications.
+ *
+ * @example
+ * ```typescript
+ * import { BackgroundTaskRunner } from './background/index.js';
+ *
+ * const runner = new BackgroundTaskRunner({ maxConcurrent: 3 }, logger);
+ *
+ * // Submit a background agent invocation
+ * const taskId = await runner.run(
+ *   'agent_invocation',
+ *   'Review authentication code',
+ *   async () => {
+ *     return await orchestrator.executeWithFailover('reviewer', messages, options);
+ *   },
+ *   { role: 'reviewer', provider: 'openai' }
+ * );
+ *
+ * // Check status later
+ * const task = runner.getTask(taskId);
+ *
+ * // Or wait for completion
+ * const completed = await runner.waitFor(taskId, 60000);
+ * ```
+ */
+export * from './types.js';
+export { BackgroundTaskRunner } from './task-runner.js';
+//# sourceMappingURL=index.js.map

package/dist/background/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/background/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6BG;AAEH,cAAc,YAAY,CAAC;AAC3B,OAAO,EAAE,oBAAoB,EAAE,MAAM,kBAAkB,CAAC"}

package/dist/background/task-runner.d.ts

@@ -0,0 +1,177 @@
+/**
+ * Background Task Runner
+ *
+ * Manages background task execution for non-blocking agent invocations
+ * and pipeline steps. Supports concurrency limits, cancellation,
+ * progress tracking, and event-based notifications.
+ */
+import { EventEmitter } from 'events';
+import type { Logger } from '../observability/logger.js';
+import type { BackgroundTask, BackgroundTaskConfig, BackgroundTaskType, SubmitTaskOptions, SerializedBackgroundTask } from './types.js';
+/**
+ * Manages background task execution with concurrency control.
+ *
+ * Features:
+ * - Queue-based execution with configurable concurrency
+ * - Task cancellation via AbortController
+ * - Progress tracking and notifications
+ * - Event emission for task lifecycle
+ * - Automatic cleanup of old tasks
+ *
+ * @example
+ * ```typescript
+ * const runner = new BackgroundTaskRunner({ maxConcurrent: 3 }, logger);
+ *
+ * // Submit a background task
+ * const taskId = await runner.run(
+ *   'agent_invocation',
+ *   'Review authentication code',
+ *   async () => {
+ *     return await orchestrator.executeWithFailover('reviewer', messages, options);
+ *   },
+ *   { role: 'reviewer', provider: 'openai' }
+ * );
+ *
+ * // Check status later
+ * const task = runner.getTask(taskId);
+ * if (task?.status === 'completed') {
+ *   console.log('Result:', task.result);
+ * }
+ *
+ * // Or wait for completion
+ * const completed = await runner.waitFor(taskId, 60000);
+ * ```
+ */
+export declare class BackgroundTaskRunner extends EventEmitter {
+    private readonly tasks;
+    private readonly queue;
+    private runningCount;
+    private readonly config;
+    private readonly logger;
+    private isEnabled;
+    private isShuttingDown;
+    constructor(config: Partial<BackgroundTaskConfig>, logger: Logger);
+    /**
+     * Check if background tasks are enabled.
+     * Respects AGENT_FOUNDRY_DISABLE_BACKGROUND_TASKS env var.
+     */
+    isBackgroundEnabled(): boolean;
+    /**
+     * Enable or disable background task execution.
+     */
+    setEnabled(enabled: boolean): void;
+    /**
+     * Submit a task to run in the background.
+     * Returns immediately with task ID.
+     *
+     * @param task - Task definition (without id, status, createdAt)
+     * @returns Task ID for tracking
+     */
+    submit(task: Omit<BackgroundTask, 'id' | 'status' | 'createdAt'>): string;
+    /**
+     * Execute a function as a background task.
+     *
+     * @param type - Task type
+     * @param description - Task description
+     * @param fn - Async function to execute
+     * @param options - Additional options
+     * @returns Task ID for tracking
+     */
+    run<T>(type: BackgroundTaskType, description: string, fn: () => Promise<T>, options?: SubmitTaskOptions): Promise<string>;
+    /**
+     * Get task by ID.
+     */
+    getTask(taskId: string): BackgroundTask | undefined;
+    /**
+     * Get all tasks.
+     */
+    getAllTasks(): BackgroundTask[];
+    /**
+     * Get running tasks.
+     */
+    getRunningTasks(): BackgroundTask[];
+    /**
+     * Get pending tasks in queue.
+     */
+    getPendingTasks(): BackgroundTask[];
+    /**
+     * Get completed tasks.
+     */
+    getCompletedTasks(): BackgroundTask[];
+    /**
+     * Cancel a running or pending task.
+     *
+     * @param taskId - ID of task to cancel
+     * @returns true if task was cancelled, false if not found or already complete
+     */
+    cancel(taskId: string): boolean;
+    /**
+     * Wait for a task to complete.
+     *
+     * @param taskId - ID of task to wait for
+     * @param timeout - Maximum time to wait in ms (default: 60000)
+     * @returns The completed task
+     * @throws Error if task not found or timeout exceeded
+     */
+    waitFor(taskId: string, timeout?: number): Promise<BackgroundTask>;
+    /**
+     * Update task progress.
+     *
+     * @param taskId - ID of task to update
+     * @param progress - Progress percentage (0-100)
+     * @param message - Optional progress message
+     */
+    updateProgress(taskId: string, progress: number, message?: string): void;
+    /**
+     * Clean up completed/failed tasks older than specified age.
+     *
+     * @param maxAgeMs - Maximum age in ms (default: 1 hour)
+     * @returns Number of tasks cleaned up
+     */
+    cleanup(maxAgeMs?: number): number;
+    /**
+     * Shutdown the runner - cancel pending tasks and wait for running tasks.
+     *
+     * @param timeout - Maximum time to wait for running tasks in ms (default: 30000)
+     */
+    shutdown(timeout?: number): Promise<void>;
+    /**
+     * Get statistics about the task runner.
+     */
+    getStats(): {
+        pending: number;
+        running: number;
+        completed: number;
+        failed: number;
+        cancelled: number;
+        total: number;
+        queueLength: number;
+        maxConcurrent: number;
+    };
+    /**
+     * Serialize tasks for persistence.
+     * Only serializes metadata, not results or errors.
+     */
+    serializeTasks(): SerializedBackgroundTask[];
+    /**
+     * Process the task queue, starting tasks up to maxConcurrent.
+     */
+    private processQueue;
+    /**
+     * Execute a single task.
+     */
+    private executeTask;
+    /**
+     * Mark a task as completed.
+     */
+    private markCompleted;
+    /**
+     * Mark a task as failed.
+     */
+    private markFailed;
+    /**
+     * Send a notification if enabled.
+     */
+    private notify;
+}
+//# sourceMappingURL=task-runner.d.ts.map

package/dist/background/task-runner.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"task-runner.d.ts","sourceRoot":"","sources":["../../src/background/task-runner.ts"],"names":[],"mappings":"AAAA;;;;;;GAMG;AAEH,OAAO,EAAE,YAAY,EAAE,MAAM,QAAQ,CAAC;AAGtC,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,4BAA4B,CAAC;AACzD,OAAO,KAAK,EACV,cAAc,EACd,oBAAoB,EAEpB,kBAAkB,EAGlB,iBAAiB,EACjB,wBAAwB,EACzB,MAAM,YAAY,CAAC;AAwBpB;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAiCG;AACH,qBAAa,oBAAqB,SAAQ,YAAY;IACpD,OAAO,CAAC,QAAQ,CAAC,KAAK,CAA0C;IAChE,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAwB;IAC9C,OAAO,CAAC,YAAY,CAAa;IACjC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAuB;IAC9C,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAS;IAChC,OAAO,CAAC,SAAS,CAAiB;IAClC,OAAO,CAAC,cAAc,CAAkB;gBAE5B,MAAM,EAAE,OAAO,CAAC,oBAAoB,CAAC,EAAE,MAAM,EAAE,MAAM;IAejE;;;OAGG;IACH,mBAAmB,IAAI,OAAO;IAQ9B;;OAEG;IACH,UAAU,CAAC,OAAO,EAAE,OAAO,GAAG,IAAI;IAKlC;;;;;;OAMG;IACH,MAAM,CACJ,IAAI,EAAE,IAAI,CAAC,cAAc,EAAE,IAAI,GAAG,QAAQ,GAAG,WAAW,CAAC,GACxD,MAAM;IAqCT;;;;;;;;OAQG;IACG,GAAG,CAAC,CAAC,EACT,IAAI,EAAE,kBAAkB,EACxB,WAAW,EAAE,MAAM,EACnB,EAAE,EAAE,MAAM,OAAO,CAAC,CAAC,CAAC,EACpB,OAAO,CAAC,EAAE,iBAAiB,GAC1B,OAAO,CAAC,MAAM,CAAC;IA0BlB;;OAEG;IACH,OAAO,CAAC,MAAM,EAAE,MAAM,GAAG,cAAc,GAAG,SAAS;IAInD;;OAEG;IACH,WAAW,IAAI,cAAc,EAAE;IAI/B;;OAEG;IACH,eAAe,IAAI,cAAc,EAAE;IAInC;;OAEG;IACH,eAAe,IAAI,cAAc,EAAE;IAInC;;OAEG;IACH,iBAAiB,IAAI,cAAc,EAAE;IAMrC;;;;;OAKG;IACH,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,OAAO;IAsC/B;;;;;;;OAOG;IACG,OAAO,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,GAAE,MAAc,GAAG,OAAO,CAAC,cAAc,CAAC;IAoD/E;;;;;;OAMG;IACH,cAAc,CAAC,MAAM,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,MAAM,GAAG,IAAI;IAoBxE;;;;;OAKG;IACH,OAAO,CAAC,QAAQ,GAAE,MAA+B,GAAG,MAAM;IAsB1D;;;;OAIG;IACG,QAAQ,CAAC,OAAO,GAAE,MAAc,GAAG,OAAO,CAAC,IAAI,CAAC;IA+BtD;;OAEG;IACH,QAAQ,IAAI;QACV,OAAO,EAAE,MAAM,CAAC;QAChB,OAAO,EAAE,MAAM,CAAC;QAChB,SAAS,EAAE,MAAM,CAAC;QAClB,MAAM,EAAE,MAAM,CAAC;QACf,SAAS,EAAE,MAAM,CAAC;QAClB,KAAK,EAAE,MAAM,CAAC;QACd,WAAW,EAAE,MAAM,CAAC;QACpB,aAAa,EAAE,MAAM,CAAC;KACvB;IAcD;;;OAGG;IACH,cAAc,IAAI,wBAAwB,EAAE;IAuB5C;;OAEG;IACH,OAAO,CAAC,YAAY;IAgBpB;;OAEG;YACW,WAAW;IAqEzB;;OAEG;IACH,OAAO,CAAC,aAAa;IA0BrB;;OAEG;IACH,OAAO,CAAC,UAAU;IA4BlB;;OAEG;IACH,OAAO,CAAC,MAAM;CAef"}