modelmix 2.6.0 → 2.6.2

package/README.md

@@ -1,11 +1,11 @@
 # 🧬 ModelMix: Unified API for Diverse AI LLM
 
-**ModelMix** is a versatile module that enables seamless integration of various language models from different providers through a unified interface. With ModelMix, you can effortlessly manage and utilize multiple AI models while controlling parallel requests to avoid provider restrictions.
+**ModelMix** is a versatile module that enables seamless integration of various language models from different providers through a unified interface. With ModelMix, you can effortlessly manage and utilize multiple AI models while controlling request rates to avoid provider restrictions.
 
 ## ✨ Features
 
 - **Unified Interface**: Interact with multiple AI models through a single, coherent API.
-- **Request Control**: Manage the number of parallel requests to adhere to provider limitations with `max_request`.
+- **Request Rate Control**: Manage the rate of requests to adhere to provider limitations using Bottleneck.
 - **Flexible Integration**: Easily integrate popular models like OpenAI, Anthropic, Perplexity, Groq, Ollama, LM Studio or custom models.
 - **History Tracking**: Automatically logs the conversation history with model responses, allowing you to limit the number of historical messages with `max_history`.
 
@@ -50,7 +50,7 @@ Here's a quick example to get you started:
         config: {
             system: "You are {name} from Melmac.",
             max_history: 2,
-            max_request: 1,
+            bottleneck: { maxConcurrent: 2 },
             debug: true
         }
     });
@@ -198,6 +198,34 @@ When you run your script this way, you'll see detailed information about the req
 
 This information is valuable for debugging and understanding how ModelMix is processing your requests.
 
+## 🚦 Bottleneck Integration
+
+ModelMix now uses Bottleneck for efficient rate limiting of API requests. This integration helps prevent exceeding API rate limits and ensures smooth operation when working with multiple models or high request volumes.
+
+### How it works:
+
+1. **Configuration**: Bottleneck is configured in the ModelMix constructor. You can customize the settings or use the default configuration:
+
+    ```javascript
+    const mmix = new ModelMix({
+        config: {
+            bottleneck: {
+                maxConcurrent: 5,     // Maximum number of concurrent requests
+                minTime: 200,         // Minimum time between requests (in ms)
+                reservoir: 60,        // Number of requests allowed in the reservoir period
+                reservoirRefreshAmount: 60, // How many requests are added when the reservoir refreshes
+                reservoirRefreshInterval: 60 * 1000 // Reservoir refresh interval (60 seconds)
+            }
+        }
+    });
+    ```
+
+2. **Rate Limiting**: When you make a request using any of the attached models, Bottleneck automatically manages the request flow based on the configured settings.
+
+3. **Automatic Queueing**: If the rate limit is reached, Bottleneck will automatically queue subsequent requests and process them as capacity becomes available.
+
+This integration ensures that your application respects API rate limits while maximizing throughput, providing a robust solution for managing multiple AI model interactions.
+
 ## 📚 ModelMix Class Overview
 
 ```javascript
@@ -213,7 +241,12 @@ new ModelMix(args = { options: {}, config: {} })
   - **config**: This object contains configuration settings that control the behavior of the `ModelMix` instance. These settings can also be overridden for specific model instances. Examples of configuration settings include:
     - `system`: Sets the default system message for the model, e.g., "You are an assistant."
     - `max_history`: Limits the number of historical messages to retain, e.g., 5.
-    - `max_request`: Limits the number of parallel requests, e.g., 1.
+    - `bottleneck`: Configures the rate limiting behavior using Bottleneck. For example:
+      - `maxConcurrent`: 5 Maximum number of concurrent requests
+      - `minTime`: 200 Minimum time between requests (in ms)
+      - `reservoir`: 60 Number of requests allowed in the reservoir period
+      - `reservoirRefreshAmount`: 60 How many requests are added when the reservoir refreshes
+      - `reservoirRefreshInterval`: 60 * 1000 // Reservoir refresh interval (60 seconds)
     - ...(Additional configuration parameters can be added as needed)
 
 **Methods**

package/demo/demo.mjs

@@ -8,7 +8,7 @@ const mmix = new ModelMix({
     config: {
         system: 'You are {name} from Melmac.',
         max_history: 2,
-        max_request: 1,
+        bottleneck: { maxConcurrent: 1 },
         debug: true,
     }
 });

package/demo/lmstudio.mjs

@@ -6,7 +6,7 @@ const mmix = new ModelMix({
     },
     config: {
         max_history: 2,
-        max_request: 1,
+        bottleneck: { maxConcurrent: 1 },
     }
 });
 

package/demo/parallel.mjs

@@ -1,50 +1,51 @@
 import 'dotenv/config';
-import { ModelMix, MixOpenAI, MixAnthropic, MixPerplexity, MixOllama } from '../index.js';
+import { ModelMix, MixOpenAI } from '../index.js';
 
 const mix = new ModelMix({
     options: {
         max_tokens: 200,
     },
     config: {
-        system: 'You are {name} from Melmac.',
         max_history: 2,
-        max_request: 3,
+        bottleneck: {
+            maxConcurrent: 1,     // Maximum number of concurrent requests
+        },
         debug: true,
     }
 });
 
 mix.attach(new MixOpenAI());
 
-// Función para crear una promesa que se resuelve después de un tiempo aleatorio
+// Function to create a promise that resolves after a random time
 const randomDelay = () => new Promise(resolve => setTimeout(resolve, Math.random() * 2000 + 1000));
 
-// Función para realizar una solicitud al modelo
+// Function to make a request to the model
 async function makeRequest(id) {
     const start = Date.now();
-    console.log(`Iniciando solicitud ${id}`);
-    
+    console.log(`Starting request ${id}`);
+
     const message = await mix.create('gpt-4o-mini')
-        .addText(`Genera un hecho interesante sobre el número ${id}.`)
+        .addText(`Generate an interesting fact about the number ${id}.`)
         .message();
-    
-    // await randomDelay(); // Simula algún procesamiento adicional
-    
+
+    await randomDelay(); // Simulates some additional processing
+
     const duration = Date.now() - start;
-    console.log(`Solicitud ${id} completada en ${duration}ms: ${message}`);
+    console.log(`Request ${id} finished in ${duration}ms: ${message}`);
 }
 
-// Función principal para ejecutar el ejemplo
+// Main function to run the example
 async function runExample() {
-    console.log("Iniciando ejemplo de concurrencia...");
-    
-    // Crear un array de promesas para 5 solicitudes
+    console.log("Starting concurrency example...");
+
+    // Create a promise array for 5 requests
     const requests = Array.from({ length: 5 }, (_, i) => makeRequest(i + 1));
-    
-    // Ejecutar todas las solicitudes y esperar a que se completen
+
+    // Execute all requests and wait for them to complete
     await Promise.all(requests);
-    
-    console.log("Ejemplo de concurrencia completado.");
+
+    console.log("Completed concurrency example.");
 }
 
-// Ejecutar el ejemplo
+// Run the example
 runExample().catch(console.error);

package/index.js

@@ -2,7 +2,7 @@ const axios = require('axios');
 const fs = require('fs');
 const mime = require('mime-types');
 const log = require('lemonlog')('ModelMix');
-const pLimit = require('p-limit');
+const Bottleneck = require('bottleneck');
 
 class ModelMix {
     constructor(args = { options: {}, config: {} }) {
@@ -14,15 +14,24 @@ class ModelMix {
             ...args.options
         };
 
+        // Standard Bottleneck configuration
+        const defaultBottleneckConfig = {
+            maxConcurrent: 5,     // Maximum number of concurrent requests
+            minTime: 200,         // Minimum time between requests (in ms)
+            reservoir: 60,        // Number of requests allowed in the reservoir period
+            reservoirRefreshAmount: 60, // How many requests are added when the reservoir refreshes
+            reservoirRefreshInterval: 60 * 1000 // Reservoir refresh interval (60 seconds)
+        };        
+
         this.config = {
             system: 'You are an assistant.',
-            max_request: 1,
             max_history: 5, // Default max history
             debug: false,
+            bottleneck: defaultBottleneckConfig,
             ...args.config
         }
 
-        this.limit = pLimit(this.config.max_request);
+        this.limiter = new Bottleneck(this.config.bottleneck);
     }
 
     replace(keyValues) {
@@ -60,6 +69,21 @@ class ModelMix {
 
         return new MessageHandler(this, modelEntry, options, config);
     }
+
+    setSystem(text) {
+        this.config.system = text;
+        return this;
+    }
+
+    setSystemFromFile(filePath) {
+        try {
+            const content = fs.readFileSync(filePath, { encoding: 'utf8' });
+            this.setSystem(content);
+        } catch (error) {
+            console.error(`Error reading system message file ${filePath}:`, error);
+        }
+        return this;
+    }    
 }
 
 class MessageHandler {
@@ -255,7 +279,7 @@ class MessageHandler {
     }
 
     async execute() {
-        return this.mix.limit(() => new Promise(async (resolve, reject) => {
+        return this.mix.limiter.schedule(async () => {
             await this.processImageUrls();
 
             this.applyTemplate();
@@ -271,11 +295,11 @@ class MessageHandler {
             try {
                 const result = await this.modelEntry.create({ options: this.options, config: this.config });
                 this.messages.push({ role: "assistant", content: result.message });
-                resolve(result);
+                return result;
             } catch (error) {
-                reject(error);
+                throw error;
             }
-        }));
+        });
     }
 }
 class MixCustom {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "modelmix",
-  "version": "2.6.0",
+  "version": "2.6.2",
   "description": "🧬 ModelMix - Unified API for Diverse AI LLM.",
   "main": "index.js",
   "repository": {
@@ -27,7 +27,8 @@
     "gpt-4o-mini",
     "4o",
     "ollama",
-    "lmstudio"
+    "lmstudio",
+    "bottleneck"
   ],
   "author": "Martin Clasen",
   "license": "MIT",
@@ -37,7 +38,7 @@
   "homepage": "https://github.com/clasen/ModelMix#readme",
   "dependencies": {
     "axios": "^1.7.4",
-    "lemonlog": "^1.1.2",
-    "p-limit": "^3.1.0"
+    "bottleneck": "^2.19.5",
+    "lemonlog": "^1.1.2"
   }
 }