@azumag/opencode-rate-limit-fallback 1.31.0 → 1.36.0

package/README.md

@@ -17,10 +17,11 @@ OpenCode plugin that automatically switches to fallback models when rate limited
   - Jitter to prevent thundering herd problem
   - Configurable retry limits and timeouts
   - Retry statistics tracking
-- Toast notifications for user feedback
-- Subagent session support with automatic fallback propagation to parent sessions
-- Configurable maximum subagent nesting depth
-- **Metrics collection** to track rate limits, fallbacks, and model performance
+ - Toast notifications for user feedback
+ - Subagent session support with automatic fallback propagation to parent sessions
+ - Configurable maximum subagent nesting depth
+ - **Circuit breaker pattern** to prevent cascading failures from consistently failing models
+ - **Metrics collection** to track rate limits, fallbacks, and model performance
 
 ## Installation
 
@@ -85,21 +86,29 @@ Create a configuration file at one of these locations:
       "format": "pretty"
     },
     "resetInterval": "daily"
+  },
+  "circuitBreaker": {
+    "enabled": true,
+    "failureThreshold": 5,
+    "recoveryTimeoutMs": 60000,
+    "halfOpenMaxCalls": 1,
+    "successThreshold": 2
   }
 }
 ```
 
 ### Configuration Options
 
-| Option | Type | Default | Description |
-|--------|------|---------|-------------|
-| `enabled` | boolean | `true` | Enable/disable the plugin |
-| `cooldownMs` | number | `60000` | Cooldown period (ms) before retrying a rate-limited model |
-| `fallbackMode` | string | `"cycle"` | Behavior when all models are exhausted (see below) |
-| `fallbackModels` | array | See below | List of fallback models in priority order |
-| `maxSubagentDepth` | number | `10` | Maximum nesting depth for subagent hierarchies |
-| `enableSubagentFallback` | boolean | `true` | Enable/disable fallback for subagent sessions |
-| `retryPolicy` | object | See below | Retry policy configuration (see below) |
+ | Option | Type | Default | Description |
+ |--------|------|---------|-------------|
+ | `enabled` | boolean | `true` | Enable/disable the plugin |
+ | `cooldownMs` | number | `60000` | Cooldown period (ms) before retrying a rate-limited model |
+ | `fallbackMode` | string | `"cycle"` | Behavior when all models are exhausted (see below) |
+ | `fallbackModels` | array | See below | List of fallback models in priority order |
+ | `maxSubagentDepth` | number | `10` | Maximum nesting depth for subagent hierarchies |
+ | `enableSubagentFallback` | boolean | `true` | Enable/disable fallback for subagent sessions |
+ | `retryPolicy` | object | See below | Retry policy configuration (see below) |
+ | `circuitBreaker` | object | See below | Circuit breaker configuration (see below) |
 
 ### Fallback Modes
 
@@ -163,11 +172,56 @@ Example with `baseDelayMs: 1000` and `maxDelayMs: 5000`:
 
 Jitter adds random variation to delay times to prevent the "thundering herd" problem, where multiple clients retry simultaneously and overwhelm the API.
 
-- Recommended for production environments with multiple concurrent users
-- `jitterFactor: 0.1` adds ±10% variance to delay times
-- Example: With base delay of 1000ms and jitterFactor 0.1, actual delay will be 900-1100ms
+ - Recommended for production environments with multiple concurrent users
+ - `jitterFactor: 0.1` adds ±10% variance to delay times
+ - Example: With base delay of 1000ms and jitterFactor 0.1, actual delay will be 900-1100ms
+
+### Circuit Breaker
+
+The circuit breaker pattern prevents cascading failures by temporarily disabling models that are consistently failing (not due to rate limits).
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `circuitBreaker.enabled` | boolean | `false` | Enable/disable circuit breaker |
+| `circuitBreaker.failureThreshold` | number | `5` | Consecutive failures before opening circuit |
+| `circuitBreaker.recoveryTimeoutMs` | number | `60000` | Wait time before attempting recovery (ms) |
+| `circuitBreaker.halfOpenMaxCalls` | number | `1` | Max calls allowed in HALF_OPEN state |
+| `circuitBreaker.successThreshold` | number | `2` | Successes needed to close circuit |
+
+#### How It Works
+
+The circuit breaker maintains three states for each model:
+
+1. **CLOSED State**: Normal operation, requests pass through
+   - Failures are counted until the threshold is reached
+   - On threshold breach, transitions to OPEN state
 
-### Default Fallback Models
+2. **OPEN State**: Model is failing, requests fail fast
+   - The circuit is "open" to prevent unnecessary API calls
+   - No requests are allowed through
+   - After the recovery timeout, transitions to HALF_OPEN state
+
+3. **HALF_OPEN State**: Testing if model recovered after timeout
+   - A limited number of test requests are allowed
+   - On success, transitions back to CLOSED state
+   - On failure, returns to OPEN state
+
+#### Important Notes
+
+- **Rate limit errors are NOT counted as failures**: The circuit breaker only tracks actual failures, not rate limit errors
+- **Disabled by default**: Set `circuitBreaker.enabled: true` to activate this feature
+- **Per-model tracking**: Each model has its own circuit state
+- **Toast notifications**: Users are notified when circuits open/close for awareness
+
+#### Configuration Recommendations
+
+| Environment | failureThreshold | recoveryTimeoutMs | halfOpenMaxCalls |
+|-------------|------------------|-------------------|------------------|
+| Development | 3 | 30000 | 1 |
+| Production | 5 | 60000 | 1 |
+| High Availability | 10 | 30000 | 2 |
+
+ ### Default Fallback Models
 
 If no configuration is provided, the following models are used:
 
@@ -206,11 +260,12 @@ When OpenCode uses subagents (e.g., for complex tasks requiring specialized agen
 
 ## Metrics
 
-The plugin includes a metrics collection feature that tracks:
-- Rate limit events per provider/model
-- Fallback statistics (total, successful, failed, average duration)
-- **Retry statistics** (total attempts, successes, failures, average delay)
-- Model performance (requests, successes, failures, response time)
+ The plugin includes a metrics collection feature that tracks:
+ - Rate limit events per provider/model
+ - Fallback statistics (total, successful, failed, average duration)
+ - **Retry statistics** (total attempts, successes, failures, average delay)
+ - Model performance (requests, successes, failures, response time)
+ - **Circuit breaker statistics** (state transitions, open/closed counts)
 
 ### Metrics Configuration
 
@@ -284,15 +339,28 @@ Retries:
 
 Model Performance:
 ----------------------------------------
-  google/gemini-2.5-pro:
-    Requests: 10
-    Successes: 9
-    Failures: 1
-    Avg Response: 0.85s
-    Success Rate: 90.0%
-```
-
-**JSON** (machine-readable):
+   google/gemini-2.5-pro:
+     Requests: 10
+     Successes: 9
+     Failures: 1
+     Avg Response: 0.85s
+     Success Rate: 90.0%
+
+ Circuit Breaker:
+ ----------------------------------------
+   anthropic/claude-3-5-sonnet-20250514:
+     State: OPEN
+     Failures: 5
+     Successes: 0
+     State Transitions: 2
+   google/gemini-2.5-pro:
+     State: CLOSED
+     Failures: 2
+     Successes: 8
+     State Transitions: 3
+ ```
+
+ **JSON** (machine-readable):
 ```json
 {
   "rateLimits": {
@@ -332,18 +400,32 @@ Model Performance:
       }
     }
   },
-  "modelPerformance": {
-    "google/gemini-2.5-pro": {
-      "requests": 10,
-      "successes": 9,
-      "failures": 1,
-      "averageResponseTime": 850
-    }
-  },
-  "startedAt": 1739148000000,
-  "generatedAt": 1739149800000
-}
-```
+   "modelPerformance": {
+     "google/gemini-2.5-pro": {
+       "requests": 10,
+       "successes": 9,
+       "failures": 1,
+       "averageResponseTime": 850
+     }
+   },
+   "circuitBreaker": {
+     "anthropic/claude-3-5-sonnet-20250514": {
+       "currentState": "OPEN",
+       "failures": 5,
+       "successes": 0,
+       "stateTransitions": 2
+     },
+     "google/gemini-2.5-pro": {
+       "currentState": "CLOSED",
+       "failures": 2,
+       "successes": 8,
+       "stateTransitions": 3
+     }
+   },
+   "startedAt": 1739148000000,
+   "generatedAt": 1739149800000
+ }
+ ```
 
 **CSV** (spreadsheet-friendly):
 ```
@@ -364,10 +446,15 @@ model,attempts,successes,success_rate
 anthropic/claude-3-5-sonnet-20250514,5,3,60.0
 google/gemini-2.5-pro,7,5,71.4
 
-=== MODEL_PERFORMANCE ===
-model,requests,successes,failures,avg_response_time_ms,success_rate
-google/gemini-2.5-pro,10,9,1,850,90.0
-```
+ === MODEL_PERFORMANCE ===
+ model,requests,successes,failures,avg_response_time_ms,success_rate
+ google/gemini-2.5-pro,10,9,1,850,90.0
+
+ === CIRCUIT_BREAKER ===
+ model,current_state,failures,successes,state_transitions
+ anthropic/claude-3-5-sonnet-20250514,OPEN,5,0,2
+ google/gemini-2.5-pro,CLOSED,2,8,3
+ ```
 
 ## License
 

package/dist/index.d.ts

@@ -6,5 +6,6 @@
 import type { Plugin } from "@opencode-ai/plugin";
 export declare const RateLimitFallback: Plugin;
 export default RateLimitFallback;
-export type { PluginConfig, MetricsConfig, FallbackModel, FallbackMode } from "./src/types/index.js";
+export type { PluginConfig, MetricsConfig, FallbackModel, FallbackMode, CircuitBreakerConfig, CircuitBreakerState, CircuitBreakerStateType } from "./src/types/index.js";
 export type { LogConfig, Logger } from "./logger.js";
+export { Logger as LoggerClass } from "./logger.js";

package/dist/index.js

@@ -7,9 +7,12 @@ import { createLogger } from "./logger.js";
 import { MetricsManager } from "./src/metrics/MetricsManager.js";
 import { FallbackHandler } from "./src/fallback/FallbackHandler.js";
 import { loadConfig } from "./src/utils/config.js";
-import { isRateLimitError } from "./src/utils/errorDetection.js";
 import { SubagentTracker } from "./src/session/SubagentTracker.js";
 import { CLEANUP_INTERVAL_MS } from "./src/types/index.js";
+import { ConfigValidator } from "./src/config/Validator.js";
+import { ErrorPatternRegistry } from "./src/errors/PatternRegistry.js";
+import { HealthTracker } from "./src/health/HealthTracker.js";
+import { DiagnosticReporter } from "./src/diagnostics/Reporter.js";
 // ============================================================================
 // Event Type Guards
 // ============================================================================
@@ -53,8 +56,8 @@ function isSubagentSessionCreatedEvent(event) {
 // ============================================================================
 // Main Plugin Export
 // ============================================================================
-export const RateLimitFallback = async ({ client, directory }) => {
-    const config = loadConfig(directory);
+export const RateLimitFallback = async ({ client, directory, worktree }) => {
+    const { config, source: configSource } = loadConfig(directory, worktree);
     // Detect headless mode (no TUI)
     const isHeadless = !client.tui;
     // Auto-adjust log level for headless mode to ensure visibility
@@ -64,38 +67,82 @@ export const RateLimitFallback = async ({ client, directory }) => {
     };
     // Create logger instance
     const logger = createLogger(logConfig, "RateLimitFallback");
+    if (configSource) {
+        logger.info(`Config loaded from ${configSource}`);
+    }
+    else {
+        logger.info("No config file found, using defaults");
+    }
+    // Initialize configuration validator
+    const validator = new ConfigValidator(logger);
+    const validation = configSource
+        ? validator.validateFile(configSource, config.configValidation)
+        : validator.validate(config, config.configValidation);
+    if (!validation.isValid && config.configValidation?.strict) {
+        logger.error("Configuration validation failed in strict mode. Plugin will not load.");
+        logger.error(`Errors: ${validation.errors.map(e => `${e.path}: ${e.message}`).join(', ')}`);
+        return {};
+    }
+    if (validation.errors.length > 0) {
+        logger.warn(`Configuration validation found ${validation.errors.length} error(s)`);
+    }
+    if (validation.warnings.length > 0) {
+        logger.warn(`Configuration validation found ${validation.warnings.length} warning(s)`);
+    }
     if (!config.enabled) {
         return {};
     }
+    // Initialize error pattern registry
+    const errorPatternRegistry = new ErrorPatternRegistry(logger);
+    if (config.errorPatterns?.custom) {
+        errorPatternRegistry.registerMany(config.errorPatterns.custom);
+    }
+    // Initialize health tracker
+    let healthTracker;
+    if (config.enableHealthBasedSelection) {
+        healthTracker = new HealthTracker(config, logger);
+        logger.info("Health-based model selection enabled");
+    }
+    // Initialize diagnostic reporter
+    const diagnostics = new DiagnosticReporter(config, configSource || 'default', healthTracker, undefined, // circuitBreaker will be initialized in FallbackHandler
+    errorPatternRegistry, logger);
+    // Log startup diagnostics if verbose mode
+    if (config.verbose) {
+        logger.debug("Verbose mode enabled - showing diagnostic information");
+        diagnostics.logCurrentConfig();
+    }
     // Initialize components
     const subagentTracker = new SubagentTracker(config);
     const metricsManager = new MetricsManager(config.metrics ?? { enabled: false, output: { console: true, format: "pretty" }, resetInterval: "daily" }, logger);
-    const fallbackHandler = new FallbackHandler(config, client, logger, metricsManager, subagentTracker);
+    const fallbackHandler = new FallbackHandler(config, client, logger, metricsManager, subagentTracker, healthTracker);
     // Cleanup stale entries periodically
     const cleanupInterval = setInterval(() => {
         subagentTracker.cleanupStaleEntries();
         fallbackHandler.cleanupStaleEntries();
+        if (healthTracker) {
+            healthTracker.cleanupOldEntries();
+        }
     }, CLEANUP_INTERVAL_MS);
     return {
         event: async ({ event }) => {
             // Handle session.error events
             if (isSessionErrorEvent(event)) {
                 const { sessionID, error } = event.properties;
-                if (sessionID && error && isRateLimitError(error)) {
+                if (sessionID && error && errorPatternRegistry.isRateLimitError(error)) {
                     await fallbackHandler.handleRateLimitFallback(sessionID, "", "");
                 }
             }
             // Handle message.updated events
             if (isMessageUpdatedEvent(event)) {
                 const info = event.properties.info;
-                if (info?.error && isRateLimitError(info.error)) {
+                if (info?.error && errorPatternRegistry.isRateLimitError(info.error)) {
                     await fallbackHandler.handleRateLimitFallback(info.sessionID, info.providerID || "", info.modelID || "");
                 }
                 else if (info?.status === "completed" && !info?.error && info?.id) {
                     // Record fallback success
                     fallbackHandler.handleMessageUpdated(info.sessionID, info.id, false, false);
                 }
-                else if (info?.error && !isRateLimitError(info.error) && info?.id) {
+                else if (info?.error && !errorPatternRegistry.isRateLimitError(info.error) && info?.id) {
                     // Record non-rate-limit error
                     fallbackHandler.handleMessageUpdated(info.sessionID, info.id, true, false);
                 }
@@ -131,7 +178,11 @@ export const RateLimitFallback = async ({ client, directory }) => {
             subagentTracker.clearAll();
             metricsManager.destroy();
             fallbackHandler.destroy();
+            if (healthTracker) {
+                healthTracker.destroy();
+            }
         },
     };
 };
 export default RateLimitFallback;
+export { Logger as LoggerClass } from "./logger.js";

package/dist/src/circuitbreaker/CircuitBreaker.d.ts

@@ -0,0 +1,60 @@
+/**
+ * Circuit Breaker - Manages circuit breakers for multiple models
+ */
+import type { Logger } from '../../logger.js';
+import type { CircuitBreakerConfig, CircuitBreakerState, OpenCodeClient } from '../types/index.js';
+import type { MetricsManager } from '../metrics/MetricsManager.js';
+/**
+ * CircuitBreaker class - Manages circuit breaker logic for models
+ */
+export declare class CircuitBreaker {
+    private circuits;
+    private config;
+    private logger;
+    private metricsManager?;
+    private client?;
+    constructor(config: CircuitBreakerConfig, logger: Logger, metricsManager?: MetricsManager, client?: OpenCodeClient);
+    /**
+     * Check if a request should be allowed for a model
+     * @param modelKey - The model key (providerID/modelID)
+     * @returns true if request is allowed, false otherwise
+     */
+    canExecute(modelKey: string): boolean;
+    /**
+     * Record a successful request for a model
+     * @param modelKey - The model key (providerID/modelID)
+     */
+    recordSuccess(modelKey: string): void;
+    /**
+     * Record a failed request for a model
+     * @param modelKey - The model key (providerID/modelID)
+     * @param isRateLimit - true if the failure was due to rate limiting
+     */
+    recordFailure(modelKey: string, isRateLimit: boolean): void;
+    /**
+     * Get the current state of a circuit
+     * @param modelKey - The model key (providerID/modelID)
+     * @returns The current circuit state
+     */
+    getState(modelKey: string): CircuitBreakerState;
+    /**
+     * Clean up stale entries from the circuits map
+     */
+    cleanupStaleEntries(): void;
+    /**
+     * Get or create a circuit for a model
+     * @private
+     */
+    private getOrCreateCircuit;
+    /**
+     * Get all circuit states
+     */
+    getAllStates(): {
+        modelKey: string;
+        state: CircuitBreakerState;
+    }[];
+    /**
+     * Destroy circuit breaker and clean up resources
+     */
+    destroy(): void;
+}

package/dist/src/circuitbreaker/CircuitBreaker.js

@@ -0,0 +1,218 @@
+/**
+ * Circuit Breaker - Manages circuit breakers for multiple models
+ */
+import { CircuitState } from './CircuitState.js';
+import { safeShowToast } from '../utils/helpers.js';
+/**
+ * CircuitBreaker class - Manages circuit breaker logic for models
+ */
+export class CircuitBreaker {
+    circuits;
+    config;
+    logger;
+    metricsManager;
+    client;
+    constructor(config, logger, metricsManager, client) {
+        this.config = config;
+        this.logger = logger;
+        this.metricsManager = metricsManager;
+        this.client = client;
+        this.circuits = new Map();
+    }
+    /**
+     * Check if a request should be allowed for a model
+     * @param modelKey - The model key (providerID/modelID)
+     * @returns true if request is allowed, false otherwise
+     */
+    canExecute(modelKey) {
+        if (!this.config.enabled) {
+            return true;
+        }
+        const circuit = this.getOrCreateCircuit(modelKey);
+        const { allowed, transition } = circuit.canExecute();
+        const state = circuit.getState();
+        this.logger.debug(`Circuit breaker check for ${modelKey}`, {
+            state: state.state,
+            allowed,
+            failureCount: state.failureCount,
+        });
+        // Log and record transition if occurred
+        if (transition) {
+            const oldStateType = transition.from;
+            const newStateType = transition.to;
+            this.logger.info(`Circuit breaker state changed for ${modelKey}`, {
+                oldState: oldStateType,
+                newState: newStateType,
+            });
+            // Show toast notification for HALF_OPEN transition (recovery attempt)
+            if (newStateType === 'HALF_OPEN' && this.client) {
+                safeShowToast(this.client, {
+                    body: {
+                        title: "Circuit Recovery Attempt",
+                        message: `Attempting recovery for ${modelKey} after ${this.config.recoveryTimeoutMs}ms`,
+                        variant: "info",
+                        duration: 3000,
+                    },
+                });
+            }
+            // Record metrics
+            if (this.metricsManager) {
+                this.metricsManager.recordCircuitBreakerStateTransition(modelKey, oldStateType, newStateType);
+            }
+        }
+        return allowed;
+    }
+    /**
+     * Record a successful request for a model
+     * @param modelKey - The model key (providerID/modelID)
+     */
+    recordSuccess(modelKey) {
+        if (!this.config.enabled) {
+            return;
+        }
+        const circuit = this.getOrCreateCircuit(modelKey);
+        const oldState = circuit.getState().state;
+        circuit.onSuccess();
+        const newState = circuit.getState().state;
+        // Log state transition
+        if (oldState !== newState) {
+            this.logger.info(`Circuit breaker state changed for ${modelKey}`, {
+                oldState,
+                newState,
+            });
+            // Show toast notification for circuit close
+            if (newState === 'CLOSED' && oldState !== 'CLOSED' && this.client) {
+                safeShowToast(this.client, {
+                    body: {
+                        title: "Circuit Closed",
+                        message: `Circuit breaker closed for ${modelKey} - service recovered`,
+                        variant: "success",
+                        duration: 3000,
+                    },
+                });
+            }
+            // Record metrics
+            if (this.metricsManager) {
+                this.metricsManager.recordCircuitBreakerStateTransition(modelKey, oldState, newState);
+            }
+        }
+    }
+    /**
+     * Record a failed request for a model
+     * @param modelKey - The model key (providerID/modelID)
+     * @param isRateLimit - true if the failure was due to rate limiting
+     */
+    recordFailure(modelKey, isRateLimit) {
+        if (!this.config.enabled) {
+            return;
+        }
+        // Rate limit errors don't count as circuit failures
+        if (isRateLimit) {
+            this.logger.debug(`Rate limit error for ${modelKey}, not counting as circuit failure`);
+            return;
+        }
+        const circuit = this.getOrCreateCircuit(modelKey);
+        const oldState = circuit.getState().state;
+        circuit.onFailure();
+        const newState = circuit.getState().state;
+        // Log state transition
+        if (oldState !== newState) {
+            this.logger.warn(`Circuit breaker state changed for ${modelKey}`, {
+                oldState,
+                newState,
+                failureCount: circuit.getState().failureCount,
+            });
+            // Show toast notification for circuit open
+            if (newState === 'OPEN' && this.client) {
+                safeShowToast(this.client, {
+                    body: {
+                        title: "Circuit Opened",
+                        message: `Circuit breaker opened for ${modelKey} after failure threshold`,
+                        variant: "warning",
+                        duration: 5000,
+                    },
+                });
+            }
+            // Show toast notification for circuit close
+            if (newState === 'CLOSED' && oldState !== 'CLOSED' && this.client) {
+                safeShowToast(this.client, {
+                    body: {
+                        title: "Circuit Closed",
+                        message: `Circuit breaker closed for ${modelKey} - service recovered`,
+                        variant: "success",
+                        duration: 3000,
+                    },
+                });
+            }
+            // Record metrics
+            if (this.metricsManager) {
+                this.metricsManager.recordCircuitBreakerStateTransition(modelKey, oldState, newState);
+            }
+        }
+    }
+    /**
+     * Get the current state of a circuit
+     * @param modelKey - The model key (providerID/modelID)
+     * @returns The current circuit state
+     */
+    getState(modelKey) {
+        const circuit = this.circuits.get(modelKey);
+        if (!circuit) {
+            return {
+                state: 'CLOSED',
+                failureCount: 0,
+                successCount: 0,
+                lastFailureTime: 0,
+                lastSuccessTime: 0,
+                nextAttemptTime: 0,
+            };
+        }
+        return circuit.getState();
+    }
+    /**
+     * Clean up stale entries from the circuits map
+     */
+    cleanupStaleEntries() {
+        const now = Date.now();
+        const cutoffTime = now - (24 * 60 * 60 * 1000); // 24 hours
+        for (const [key, circuit] of this.circuits.entries()) {
+            const state = circuit.getState();
+            const lastActivity = Math.max(state.lastFailureTime, state.lastSuccessTime);
+            // Remove circuits that haven't been active for 24 hours
+            if (lastActivity < cutoffTime) {
+                this.circuits.delete(key);
+                this.logger.debug(`Cleaned up stale circuit for ${key}`);
+            }
+        }
+    }
+    /**
+     * Get or create a circuit for a model
+     * @private
+     */
+    getOrCreateCircuit(modelKey) {
+        let circuit = this.circuits.get(modelKey);
+        if (!circuit) {
+            circuit = new CircuitState(this.config);
+            this.circuits.set(modelKey, circuit);
+            this.logger.debug(`Created new circuit for ${modelKey}`);
+        }
+        return circuit;
+    }
+    /**
+     * Get all circuit states
+     */
+    getAllStates() {
+        const result = [];
+        for (const [modelKey, circuit] of this.circuits.entries()) {
+            result.push({ modelKey, state: circuit.getState() });
+        }
+        return result;
+    }
+    /**
+     * Destroy circuit breaker and clean up resources
+     */
+    destroy() {
+        this.circuits.clear();
+        this.logger.debug('Circuit breaker destroyed');
+    }
+}