ai-speedometer 1.0.1 → 1.2.0

package/README.md

@@ -1,309 +1,58 @@
 # Ai-speedometer
 
-A comprehensive, modern CLI tool for benchmarking AI models across multiple providers with **parallel execution**, **professional tables**, **arrow key navigation**, and **advanced metrics**.
+A CLI tool for benchmarking AI models across multiple providers with parallel execution and performance metrics.
 
-## Quick Start
+## Install
 
 ```bash
-git clone https://github.com/aptdnfapt/Ai-speedometer
-# Install dependencies
-npm install
-# Set up your API keys and providers (see Setup Guide below)
-# Start the CLI
-npm run cli
-
-
+npm install -g ai-speedometer
 ```
-Debug 
-```bash
-# Start with debug logging (for troubleshooting)
-npm run cli:debug
-```
-
-## Setup Guide
-
-### Before You Begin
-
-Before running the benchmark, you need to configure your AI providers with API keys and base URLs. The tool supports two types of providers:
-
-1. **OpenAI-Compatible providers** (OpenAI, local models, custom endpoints)
-2. **Anthropic providers** (Claude models)
-
-### Step 1: Get Your API Keys
-
-#### OpenAI-Compatible Providers
-- **OpenAI**: Get your API key from [OpenAI API Keys](https://platform.openai.com/api-keys)
-- **Other providers**: Check your provider's documentation for API key access
 
-#### Anthropic Providers
-- **Anthropic**: Get your API key from [Anthropic Console](https://console.anthropic.com/settings/keys)
+## What It Measures
 
-### Step 2: Configure Providers
+- **TTFT** (Time to First Token) - How fast the first response token arrives
+- **Total Time** - Complete request duration  
+- **Tokens/Second** - Real-time throughput
+- **Token Counts** - Input, output, and total tokens used
 
-You have two ways to configure providers:
+## Quick Setup
 
-#### Method A: Use the Interactive CLI (Recommended)
-
-1. Run the CLI:
+1. **Set Model**
    ```bash
-   npm run cli
+   ai-speedometer
+   # Select "Set Model" → "Add Verified Provider" → Choose provider (OpenAI, Anthropic, etc.)
+   # Enter your API key when prompted
    ```
 
-2. Select "Set Model" from the main menu
-
-3. Choose "Add New Provider"
-
-4. Select the provider type:
-   - **OpenAI Compatible**: For OpenAI, local models, or custom endpoints
-   - **Anthropic**: For Claude models
+2. **Choose Model Provider**
+   - Verified providers (OpenAI, Anthropic, Google) - auto-configured
+   - Custom providers (Ollama, local models) - add your base URL
 
-5. Enter the required information:
-   - **Provider name**: A friendly name (e.g., "My OpenAI", "Local Ollama")
-   - **Base URL**: The API endpoint (see examples below)
-   - **API Key**: Your secret API key
-   - **Model name**: The specific model you want to test
+3. **Add API Key**
+   - Get API keys from your provider's dashboard
+   - Enter when prompted - stored securely
 
-#### Method B: Manual Configuration
-
-1. Copy the template:
+4. **Run Benchmark**
    ```bash
-   cp ai-benchmark-config.json.template ai-benchmark-config.json
+   ai-speedometer
+   # Select "Run Benchmark (AI SDK)" → Choose models → Press ENTER
    ```
 
-2. Edit `ai-benchmark-config.json` with your provider details:
-
-   ```json
-   {
-     "providers": [
-       {
-         "id": "my_openai",
-         "name": "OpenAI",
-         "type": "openai-compatible",
-         "baseUrl": "https://api.openai.com/v1",
-         "apiKey": "sk-your-openai-key-here",
-         "models": [
-           {
-             "name": "gpt-4",
-             "id": "gpt4_model"
-           }
-         ]
-       },
-       {
-         "id": "my_anthropic",
-         "name": "Anthropic",
-         "type": "anthropic",
-         "baseUrl": "https://api.anthropic.com",
-         "apiKey": "sk-ant-your-anthropic-key-here",
-         "models": [
-           {
-             "name": "claude-3-sonnet-20240229",
-             "id": "claude3_sonnet"
-           }
-         ]
-       }
-     ]
-   }
-   ```
-
-### Step 3: Common Base URL Examples
-
-#### OpenAI-Compatible Providers
-- **OpenAI**: `https://api.openai.com/v1`
-- **Local Ollama**: `http://localhost:11434/v1`
-- **Groq**: `https://api.groq.com/openai/v1`
-- **Together AI**: `https://api.together.xyz/v1`
-- **Anyscale**: `https://api.endpoints.anyscale.com/v1`
-- **Fireworks AI**: `https://api.fireworks.ai/inference/v1`
-
-#### Anthropic Providers
-- **Anthropic Official**: `https://api.anthropic.com`
-- **Custom Anthropic endpoints**: Check with your provider
-
-### Step 4: Security
-
-Your configuration file contains sensitive API keys. The `.gitignore` file already excludes `ai-benchmark-config.json` to prevent accidental commits.
-
-**Never commit your API keys to version control!**
-
-### Step 5: Verify Configuration
-
-After setting up, run the CLI and check that your providers appear in the model selection menu. If you see your providers and models listed, you're ready to benchmark!
-
-### Troubleshooting
-
-- **"Provider not found"**: Check your base URL and API key
-- **"Model not available"**: Verify the model name is correct for your provider
-- **"Connection failed"**: Ensure your base URL is accessible and you have internet access
-- **"Invalid API key"**: Double-check your API key is correct and has proper permissions
-- **Debug Mode**: Use `npm run cli:debug` to enable detailed logging. This creates a `debug.log` file with API request/response details for troubleshooting connection issues.
-
-## Usage Examples
-
-### Main Menu (Modern Arrow Navigation)
-```
-Ai-speedometer
-=============================
-Note: opencode uses ai-sdk
-
-Use ↑↓ arrows to navigate, ENTER to select
-Navigation is circular
-
-● Set Model
-○ Run Benchmark (AI SDK)
-○ Run Benchmark (REST API)
-○ Exit
-```
-
-### Model Selection (Circle-Based UI)
-```
-Select Models for Benchmark
-
-Use ↑↓ arrows to navigate, SPACE to select/deselect, ENTER to confirm
-Navigation is circular - moving past bottom/top wraps around
-Press "A" to select all models, "N" to deselect all
-Circle states: ●=Current+Selected  ○=Current+Unselected  ●=Selected  ○=Unselected
-
-Available Models:
-● gpt-4 (OpenAI)
-○ claude-3-sonnet (Anthropic)
-
-Selected: 1 models
-```
-
-### Provider Management (Vertical Stacking)
-```
-Available Providers
-
-1. chutes (openai-compatible)
-   Models:
-     1. zai-org/GLM-4.5-turbo
-     2. deepseek-ai/DeepSeek-V3.1-turbo
-
-2. zai (openai-compatible)
-   Models:
-     1. glm-4.5
-
-3. zai-anthropic (anthropic)
-   Models:
-     1. claude-3-sonnet-20240229
-```
-
-### Benchmark Results (Professional Tables + Enhanced Charts)
-```
-BENCHMARK RESULTS
-=========================
-Method: AI SDK
-
-COMPREHENSIVE PERFORMANCE SUMMARY
-Note: AI SDK method does not count thinking tokens as first token. REST API method does not use streaming.
-┌─────────────────────────┬─────────────────────┬─────────────────┬────────────┬─────────────────┬─────────────────┬─────────────────┬─────────────────┐
-│ Model                  │ Provider            │ Total Time(s)   │ TTFT(s)    │ Tokens/Sec     │ Output Tokens   │ Prompt Tokens   │ Total Tokens    │
-├─────────────────────────┼─────────────────────┼─────────────────┼────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┤
-│ zai-org/GLM-4.5-turbo  │ chutes              │ 11.47           │ 1.00       │ 81.5           │ 935             │ 14              │ 1205            │
-│ deepseek-ai/DeepSeek-V3 │ chutes              │ 5.21            │ 0.83       │ 178.6          │ 930             │ 14              │ 742             │
-│ glm-4.5                │ zai                 │ 11.30           │ 5.30       │ 72.9           │ 824             │ 14              │ 1087            │
-└─────────────────────────┴─────────────────────┴─────────────────┴────────────┴─────────────────┴─────────────────┴─────────────────┴─────────────────┘
-
-PERFORMANCE COMPARISON CHARTS
-──────────────────────────────────────────────────────────────────────────────────────────
-
-TOTAL TIME COMPARISON (lower is better)
-   5.21s |     178.6 tok/s | deepseek-ai/DeepSeek-V3.1-turbo | ████████████████████████████████████████████████
-  11.30s |      72.9 tok/s | glm-4.5                                | ████████████████████████████████████░░░░░░░░░
-  11.47s |      81.5 tok/s | zai-org/GLM-4.5-turbo                   | ████████████████████████████████████░░░░░░░░░
-
-TOKENS PER SECOND COMPARISON (higher is better)
-     178.6 tok/s |    5.21s | deepseek-ai/DeepSeek-V3.1-turbo | █████████████████████████████████████████████████████
-      81.5 tok/s |   11.47s | zai-org/GLM-4.5-turbo                   | ████████████████████████████████░░░░░░░░░░░░░░░░
-      72.9 tok/s |   11.30s | glm-4.5                                | ████████████████████████████████░░░░░░░░░░░░░░░░
-
-Benchmark completed!
-```
-
-## Configuration
-
-### Adding Providers (Arrow Key Navigation)
-
-#### OpenAI-Compatible Providers
-```
-Add New Provider
+## Usage
 
-Use ↑↓ arrows to navigate, ENTER to select
-Navigation is circular
-
-Select provider type:
+```bash
+# Start CLI
+ai-speedometer
 
-● OpenAI Compatible
-○ Anthropic
-○ Back to main menu
-```
+# Or use short alias  
+aispeed
 
-#### Anthropic Providers (Now Supports Custom Base URLs)
+# Debug mode
+ai-speedometer --debug
 ```
-Enter provider name (e.g., MyAnthropic):
-Enter base URL (e.g., https://api.anthropic.com):
-Enter Anthropic API key: [your-key]
-Enter model name (e.g., claude-3-sonnet-20240229):
-```
-
-**Note**: The system automatically handles `/v1` path requirements for custom Anthropic endpoints. If you encounter issues with custom base URLs, run `npm run cli:debug` to see detailed API request logs.
-
-
-## Performance Metrics Explained
-
-### Core Metrics
-- **Total Time**: Complete request duration (seconds)
-- **Time to First Token (TTFT)**: Latency until first streaming token arrives (0 for REST API since it doesn't use streaming)
-- **Tokens per Second**: Real-time throughput calculation
-- **Output Tokens**: Number of tokens in the AI response
-- **Prompt Tokens**: Number of tokens in the input prompt
-- **Total Tokens**: Combined prompt + output tokens
-
-### Benchmark Methods
-- **AI SDK Method**: Uses streaming with Vercel AI SDK, doesn't count thinking tokens as first token
-- **REST API Method**: Uses direct HTTP calls, no streaming, TTFT is always 0
-
-### Chart Features
-- **Dual Comparison Charts**: Both time and performance perspectives
-- **Left-Side Metrics**: Shows actual values alongside bar charts
-- **Color Coding**: Red bars for time (lower is better), green for performance (higher is better)
-- **Dynamic Scaling**: Bars scale proportionally to the best/worst performers
-
-## Tech Stack
-
-- **AI SDK**: Vercel AI SDK with streaming support (opencode uses it)
-- **Table Rendering**: `cli-table3` for professional tables
-- **Providers**: OpenAI-compatible and Anthropic APIs with custom baseUrl support
-- **Navigation**: Circular arrow key navigation throughout
-- **Colors**: ANSI escape codes for terminal styling
-- **Configuration**: JSON-based persistent storage
-- **Security**: .gitignore protection for sensitive files
-- **Debug Logging**: Built-in debugging system for troubleshooting API connections
 
 ## Requirements
 
 - Node.js 18+
 - API keys for AI providers
-- Terminal that supports ANSI colors and arrow keys
-- Git (for security configuration)
-
-
-## Advanced Features
-
-### Parallel Execution
-- **Speed**: Runs all selected models simultaneously
-- **Efficiency**: No sequential waiting between models
-- **Results**: Comprehensive comparison across all models
-
-### Advanced Navigation
-- **Universal Pattern**: All menus use the same arrow key navigation
-- **Circular Movement**: Navigation wraps at top/bottom for seamless UX
-- **Visual Feedback**: Clear indicators for current selections
-- **Keyboard Shortcuts**: Quick actions like select all ('A') and deselect all ('N')
-
-### Professional Output
-- **Table Format**: Clean, aligned columns with proper spacing
-- **Color Coding**: Different colors for different metric types
-- **Comprehensive Data**: All relevant metrics in one view
-- **Visual Charts**: Bar charts for quick visual comparison
-
+- Terminal with arrow keys and ANSI colors

package/cli.js

@@ -21,7 +21,10 @@ import {
   getVerifiedProvidersFromConfig,
   addCustomProvider,
   addModelToCustomProvider,
-  getAIConfigDebugPaths
+  getAIConfigDebugPaths,
+  addToRecentModels,
+  getRecentModels,
+  cleanupRecentModelsFromConfig
 } from './ai-config.js';
 import 'dotenv/config';
 import Table from 'cli-table3';
@@ -203,9 +206,12 @@ async function selectModelsCircular() {
   showHeader();
   console.log(colorText('Select Models for Benchmark', 'magenta'));
   console.log('');
-  
+   
   const config = await loadConfig();
-  
+   
+  // Clean up recent models from main config and migrate to cache
+  await cleanupRecentModelsFromConfig();
+   
   if (config.providers.length === 0) {
     console.log(colorText('No providers available. Please add a provider first.', 'red'));
     await question(colorText('Press Enter to continue...', 'yellow'));
@@ -230,16 +236,38 @@ async function selectModelsCircular() {
     });
   });
   
+  // Load recent models
+  const recentModelsData = await getRecentModels();
+  
+  // Create a mapping of recent models to actual model objects
+  const recentModelObjects = [];
+  recentModelsData.forEach(recentModel => {
+    const modelObj = allModels.find(model => 
+      model.id === recentModel.modelId && 
+      model.providerName === recentModel.providerName
+    );
+    if (modelObj) {
+      recentModelObjects.push({
+        ...modelObj,
+        isRecent: true
+      });
+    }
+  });
+  
   let currentIndex = 0;
   let currentPage = 0;
   let searchQuery = '';
-  let filteredModels = [...allModels];
   
   // Create a reusable filter function to avoid code duplication
   const filterModels = (query) => {
     if (!query.trim()) {
-      return [...allModels];
+      // When search is empty, return the combined list with recent models at top
+      const recentModelIds = new Set(recentModelObjects.map(m => m.id));
+      const nonRecentModels = allModels.filter(model => !recentModelIds.has(model.id));
+      return [...recentModelObjects, ...nonRecentModels];
     }
+    
+    // When searching, search through all models (no recent section)
     const lowercaseQuery = query.toLowerCase();
     return allModels.filter(model => {
       const modelNameMatch = model.name.toLowerCase().includes(lowercaseQuery);
@@ -251,6 +279,9 @@ async function selectModelsCircular() {
     });
   };
   
+  // Initialize filtered models using the filter function
+  let filteredModels = filterModels('');
+  
   // Debounce function to reduce filtering frequency
   let searchTimeout;
   const debouncedFilter = (query, callback) => {
@@ -275,7 +306,7 @@ async function selectModelsCircular() {
     screenContent += colorText('Type to search (real-time filtering)', 'cyan') + '\n';
     screenContent += colorText('Press "A" to select all models, "N" to deselect all', 'cyan') + '\n';
     screenContent += colorText('Circle states: ●=Current+Selected  ○=Current+Unselected  ●=Selected  ○=Unselected', 'dim') + '\n';
-    screenContent += colorText('Quick run: ENTER on any model | Multi-select: TAB then ENTER', 'dim') + '\n';
+    screenContent += colorText('Quick run: ENTER on any model | Multi-select: TAB then ENTER | Recent: R', 'dim') + '\n';
     screenContent += '\n';
     
     // Search interface - always visible
@@ -294,13 +325,51 @@ async function selectModelsCircular() {
     const endIndex = Math.min(startIndex + visibleItemsCount, filteredModels.length);
     
     // Display models in a vertical layout with pagination
-    screenContent += colorText('Available Models:', 'yellow') + '\n';
-    screenContent += '\n';
+    let hasRecentModelsInCurrentPage = false;
+    let recentSectionDisplayed = false;
+    let nonRecentSectionDisplayed = false;
+    
+    // Only show recent section when search is empty and we have recent models
+    const showRecentSection = searchQuery.length === 0 && recentModelObjects.length > 0;
+    
+    // Check if current page contains any recent models (only when search is empty)
+    if (showRecentSection) {
+      for (let i = startIndex; i < endIndex; i++) {
+        if (filteredModels[i].isRecent) {
+          hasRecentModelsInCurrentPage = true;
+          break;
+        }
+      }
+    }
     
+    // Display models with proper section headers
     for (let i = startIndex; i < endIndex; i++) {
       const model = filteredModels[i];
       const isCurrent = i === currentIndex;
-      const isSelected = model.selected;
+      // For recent models, check selection state from the original model
+      let isSelected;
+      if (model.isRecent) {
+        const originalModelIndex = allModels.findIndex(originalModel => 
+          originalModel.id === model.id && 
+          originalModel.providerName === model.providerName &&
+          !originalModel.isRecent
+        );
+        isSelected = originalModelIndex !== -1 ? allModels[originalModelIndex].selected : false;
+      } else {
+        isSelected = model.selected;
+      }
+      
+      // Show recent section header if we encounter a recent model and haven't shown the header yet
+      if (model.isRecent && !recentSectionDisplayed && hasRecentModelsInCurrentPage && showRecentSection) {
+        screenContent += colorText('-------recent--------', 'dim') + '\n';
+        recentSectionDisplayed = true;
+      }
+      
+      // Show separator between recent and non-recent models
+      if (!model.isRecent && recentSectionDisplayed && !nonRecentSectionDisplayed && showRecentSection) {
+        screenContent += colorText('-------recent--------', 'dim') + '\n';
+        nonRecentSectionDisplayed = true;
+      }
       
       // Single circle that shows both current state and selection
       let circle;
@@ -378,10 +447,28 @@ async function selectModelsCircular() {
       }
     } else if (key === '\t') {
       // Tab - select/deselect current model
-      const actualModelIndex = allModels.indexOf(filteredModels[currentIndex]);
+      const currentModel = filteredModels[currentIndex];
+      let actualModelIndex;
+      
+      if (currentModel.isRecent) {
+        // For recent models, find by matching the original model ID and provider name
+        actualModelIndex = allModels.findIndex(model => 
+          model.id === currentModel.id && 
+          model.providerName === currentModel.providerName &&
+          !model.isRecent // Don't match the recent copy, match the original
+        );
+      } else {
+        // For regular models, use the standard matching
+        actualModelIndex = allModels.findIndex(model => 
+          model.id === currentModel.id && model.providerName === currentModel.providerName
+        );
+      }
+      
       if (actualModelIndex !== -1) {
         allModels[actualModelIndex].selected = !allModels[actualModelIndex].selected;
       }
+      // Force immediate screen redraw by continuing to next iteration
+      continue;
     } else if (key === '\r') {
       // Enter - run benchmark on selected models
       const currentModel = filteredModels[currentIndex];
@@ -448,6 +535,35 @@ async function selectModelsCircular() {
           currentPage = 0;
         });
       }
+    } else if (key === 'R' || key === 'r') {
+      // Run recent models - only when search is empty and we have recent models
+      if (searchQuery.length === 0 && recentModelObjects.length > 0) {
+        // Deselect all models first
+        allModels.forEach(model => model.selected = false);
+        
+        // Select all recent models by finding the original models
+        recentModelObjects.forEach(recentModel => {
+          const actualModelIndex = allModels.findIndex(model => 
+            model.id === recentModel.id && 
+            model.providerName === recentModel.providerName &&
+            !model.isRecent // Match the original, not the recent copy
+          );
+          if (actualModelIndex !== -1) {
+            allModels[actualModelIndex].selected = true;
+          }
+        });
+        
+        // Break out of loop to run benchmark
+        break;
+      } else {
+        // If search is active or no recent models, add 'R' to search query
+        searchQuery += key;
+        debouncedFilter(searchQuery, (newFilteredModels) => {
+          filteredModels = newFilteredModels;
+          currentIndex = 0;
+          currentPage = 0;
+        });
+      }
     } else if (key === 'a' || key === 'n') {
       // Lowercase 'a' and 'n' go to search field (not select all/none)
       searchQuery += key;
@@ -653,11 +769,11 @@ async function runStreamingBenchmark(models) {
   console.log('');
   console.log(colorText('All benchmarks completed!', 'green'));
   
-  await displayColorfulResults(results, 'AI SDK');
+  await displayColorfulResults(results, 'AI SDK', models);
 }
 
 // Colorful results display with comprehensive table and enhanced bars
-async function displayColorfulResults(results, method = 'AI SDK') {
+async function displayColorfulResults(results, method = 'AI SDK', models = []) {
   clearScreen();
   showHeader();
   console.log(colorText('BENCHMARK RESULTS', 'magenta'));
@@ -820,6 +936,26 @@ async function displayColorfulResults(results, method = 'AI SDK') {
     console.log('');
   }
   
+  // Add successful models to recent models list
+  const successfulModels = results
+    .filter(r => r.success)
+    .map(r => {
+      // Find the actual model object that matches this benchmark result
+      const modelObj = models.find(model => 
+        model.name === r.model && model.providerName === r.provider
+      );
+      
+      return {
+        modelId: modelObj ? modelObj.id : r.model, // Use actual ID if found, fallback to name
+        modelName: r.model,
+        providerName: r.provider
+      };
+    });
+  
+  if (successfulModels.length > 0) {
+    await addToRecentModels(successfulModels);
+  }
+  
   console.log(colorText('Benchmark completed!', 'green'));
   await question(colorText('Press Enter to continue...', 'yellow'));
 }
@@ -1695,7 +1831,27 @@ async function runRestApiBenchmark(models) {
   console.log('');
   console.log(colorText('All REST API benchmarks completed!', 'green'));
   
-  await displayColorfulResults(results, 'REST API');
+  await displayColorfulResults(results, 'REST API', models);
+  
+  // Add successful models to recent models list
+  const successfulModels = results
+    .filter(r => r.success)
+    .map(r => {
+      // Find the actual model object that matches this benchmark result
+      const modelObj = models.find(model => 
+        model.name === r.model && model.providerName === r.provider
+      );
+      
+      return {
+        modelId: modelObj ? modelObj.id : r.model, // Use actual ID if found, fallback to name
+        modelName: r.model,
+        providerName: r.provider
+      };
+    });
+  
+  if (successfulModels.length > 0) {
+    await addToRecentModels(successfulModels);
+  }
 }
 
 // Main menu with arrow key navigation
@@ -1840,7 +1996,13 @@ process.on('SIGINT', () => {
 if (import.meta.url === `file://${process.argv[1]}` || 
     process.argv.length === 2 || 
     (process.argv.length === 3 && process.argv[2] === '--debug')) {
-  showMainMenu();
+  
+  // Clean up recent models from main config and migrate to cache on startup
+  cleanupRecentModelsFromConfig().then(() => {
+    showMainMenu();
+  }).catch(() => {
+    showMainMenu();
+  });
 }
 
 export { showMainMenu, listProviders, selectModelsCircular, runStreamingBenchmark, loadConfig, saveConfig };