llm-speed-bench 1.0.0

package/README.md

@@ -0,0 +1,198 @@
+# LLM Speed Bench
+
+`llm-speed-bench` is a command-line interface (CLI) tool for benchmarking the performance of Large Language Model (LLM) providers that offer an OpenAI-compatible API.
+
+It is designed to provide detailed, actionable data on the output speed and latency characteristics of different models and providers. It measures key performance indicators from the moment a request is sent until the final token of the response is received, with a focus on streaming APIs.
+
+## Features
+
+*   **OpenAI-Compatible:** Works with any API that adheres to the OpenAI specification for streaming chat completions.
+*   **Streaming First:** Benchmarks performance by leveraging the provider's streaming API to get detailed timing data.
+*   **Detailed Performance Metrics:** Collects and calculates a comprehensive set of metrics, including token counts, time to first token, inter-token latency, and overall throughput.
+*   **Flexible Configuration:** Manage inputs via both command-line arguments and environment variables.
+*   **Multiple Output Formats:** Presents results in a clean, human-readable format, with an option for machine-readable JSON.
+
+## Installation
+
+1.  **Clone the repository:**
+    ```bash
+    git clone https://github.com/your-username/llm-speed-bench.git
+    cd llm-speed-bench
+    ```
+
+2.  **Install dependencies:**
+    ```bash
+    npm install
+    ```
+
+3.  **Build the project:**
+    ```bash
+    npm run build
+    ```
+    This will compile the TypeScript code into JavaScript and place the executable in the `dist/` directory.
+
+## Usage
+
+You can run the tool using the compiled executable located at `dist/index.js`. Configuration can be provided through command-line arguments or environment variables.
+
+### Configuration
+
+The tool requires four pieces of information to run:
+
+| Parameter | CLI Argument | Environment Variable | Required | Description |
+| :--- | :--- | :--- | :--- | :--- |
+| API Base URL | `--api-base-url <url>` | `LLM_API_BASE_URL` | Yes | The base URL for the OpenAI-compatible API. |
+| API Key | `--api-key <key>` | `LLM_API_KEY` | Yes | The authentication key for the API. |
+| Model Name | `--model <name>` | `LLM_MODEL_NAME` | Yes | The specific model to be benchmarked (e.g., `gpt-4o`). |
+| Prompt | `--prompt <text>` | `LLM_PROMPT` | Yes | The input text to send to the model. |
+
+### Examples
+
+#### Using Command-Line Arguments
+
+```bash
+./dist/index.js \
+  --api-base-url "https://api.openai.com/v1" \
+  --api-key "sk-..." \
+  --model "gpt-4o" \
+  --prompt "Tell me a short story about a robot who discovers music."
+```
+
+#### Using Environment Variables
+
+You can create a `.env` file in the project root or export the variables in your shell:
+
+**`.env` file:**
+```
+LLM_API_BASE_URL="https://api.openai.com/v1"
+LLM_API_KEY="sk-..."
+LLM_MODEL_NAME="gpt-4o"
+LLM_PROMPT="Tell me a short story about a robot who discovers music."
+```
+
+Then, run the tool:
+```bash
+./dist/index.js
+```
+
+#### Getting JSON Output
+
+To get the results in a machine-readable JSON format, use the `--json` flag:
+
+```bash
+./dist/index.js --json > results.json
+```
+
+## Output Format
+
+### Standard Output
+
+The default output is a human-readable summary:
+
+```
+LLM Benchmark Results
+=======================
+
+Configuration
+-----------------------
+Provider API Base:   https://api.groq.com/openai
+Model:               llama3-70b-8192
+
+Metrics
+-----------------------
+Time to First Token:   152 ms
+Total Wall Clock Time: 2,130 ms
+Overall Output Rate:   234.7 tokens/sec
+
+Token Counts
+-----------------------
+Prompt Tokens:         35 (estimated)
+Output Tokens:         450
+
+Inter-Token Latency (ms)
+-----------------------
+Min:                 2 ms
+Mean:                4.1 ms
+Median:              4 ms
+Max:                 15 ms
+p90:                 6 ms
+p95:                 8 ms
+p99:                 12 ms
+```
+
+### JSON Output (`--json`)
+
+The JSON output includes all the calculated metrics and configuration details.
+
+```json
+{
+  "configuration": {
+    "apiBaseUrl": "https://api.groq.com/openai",
+    "model": "llama3-70b-8192"
+  },
+  "metrics": {
+    "timeToFirstTokenMs": 152,
+    "totalWallClockTimeMs": 2130,
+    "overallOutputRateTps": 234.7
+  },
+  "tokenCounts": {
+    "promptTokens": 35,
+    "outputTokens": 450
+  },
+  "interTokenLatencyMs": {
+    "min": 2,
+    "mean": 4.1,
+    "median": 4,
+    "max": 15,
+    "p90": 6,
+    "p95": 8,
+    "p99": 12
+  }
+}
+```
+
+## Development
+
+### Running with ts-node
+
+To run the tool in development mode without building, you can use `ts-node`:
+
+```bash
+npx ts-node src/index.ts --api-base-url ...
+```
+
+### Local Installation and Testing
+
+To test the CLI locally as if it were globally installed, you can use `npm link`. This is the best way to test the final command-line experience before publishing.
+
+1.  **Build the project:**
+    Make sure your latest changes are compiled.
+    ```bash
+    npm run build
+    ```
+
+2.  **Link the package:**
+    This creates a global symbolic link to your local project.
+    ```bash
+    npm link
+    ```
+
+3.  **Run the command globally:**
+    You can now run the command from any directory.
+    ```bash
+    llm-speed-bench --api-base-url "..." --api-key "..."
+    ```
+
+4.  **Rebuild after changes:**
+    Whenever you change the source code, just re-run the build command. The symbolic link will ensure your global command always uses the latest compiled code.
+    ```bash
+    npm run build
+    ```
+
+5.  **Unlink the package:**
+    When you're done with local testing, you can remove the global link.
+    ```bash
+    npm unlink llm-speed-bench
+    ```
+
+```

package/dist/index.d.ts

@@ -0,0 +1,3 @@
+#!/usr/bin/env node
+export {};
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":""}

package/dist/index.js

@@ -0,0 +1,339 @@
+#!/usr/bin/env node
+process.env.DOTENV_CONFIG_SILENT = 'true';
+import yargs from 'yargs';
+import { hideBin } from 'yargs/helpers';
+import dotenv from 'dotenv';
+import { get_encoding } from 'tiktoken';
+// Load environment variables from .env file
+dotenv.config({ quiet: true });
+// --- Utility Functions ---
+/**
+ * Calculates percentile values from a sorted array of numbers.
+ * @param arr - A sorted array of numbers.
+ * @param p - The percentile to calculate (0-100).
+ * @returns The value at the given percentile.
+ */
+const percentile = (arr, p) => {
+    if (arr.length === 0)
+        return 0;
+    const k = (p / 100) * (arr.length - 1);
+    const f = Math.floor(k);
+    const c = Math.ceil(k);
+    if (f === c) {
+        return arr[f];
+    }
+    const fVal = arr[f];
+    const cVal = arr[c];
+    return fVal * (c - k) + cVal * (k - f);
+};
+/**
+ * Calculates statistical metrics for a list of latencies.
+ * @param latencies - An array of numbers.
+ * @returns An object with min, max, mean, median, and percentile stats.
+ */
+const calculateLatencyStats = (latencies) => {
+    if (latencies.length === 0) {
+        return { min: 0, max: 0, mean: 0, median: 0, p90: 0, p95: 0, p99: 0 };
+    }
+    const sorted = [...latencies].sort((a, b) => a - b);
+    const sum = sorted.reduce((a, b) => a + b, 0);
+    const mean = sum / sorted.length;
+    const min = sorted[0];
+    const max = sorted[sorted.length - 1];
+    if (min === undefined || max === undefined) {
+        // This path should be unreachable due to the length check above,
+        // but it satisfies the strict type checker.
+        return { min: 0, max: 0, mean: 0, median: 0, p90: 0, p95: 0, p99: 0 };
+    }
+    const median = percentile(sorted, 50);
+    const p90 = percentile(sorted, 90);
+    const p95 = percentile(sorted, 95);
+    const p99 = percentile(sorted, 99);
+    return {
+        min: parseFloat(min.toFixed(2)),
+        max: parseFloat(max.toFixed(2)),
+        mean: parseFloat(mean.toFixed(2)),
+        median: parseFloat(median.toFixed(2)),
+        p90: parseFloat(p90.toFixed(2)),
+        p95: parseFloat(p95.toFixed(2)),
+        p99: parseFloat(p99.toFixed(2)),
+    };
+};
+// --- Main Application Logic ---
+class ProgressIndicator {
+    constructor(isJson) {
+        this.isJson = isJson;
+        this.stream = process.stderr;
+        this.spinner = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
+        this.frame = 0;
+        this.message = '';
+    }
+    render() {
+        if (this.isJson)
+            return;
+        this.stream.clearLine(0);
+        this.stream.cursorTo(0);
+        const spinnerFrame = this.spinner[this.frame];
+        this.stream.write(`${spinnerFrame} ${this.message}`);
+        this.frame = (this.frame + 1) % this.spinner.length;
+    }
+    start(message) {
+        if (this.isJson)
+            return;
+        this.message = message;
+        this.timer = setInterval(() => this.render(), 80);
+    }
+    update(message) {
+        if (this.isJson)
+            return;
+        this.message = message;
+    }
+    stop(finalMessage) {
+        if (this.isJson)
+            return;
+        if (this.timer) {
+            clearInterval(this.timer);
+            this.timer = undefined;
+        }
+        this.stream.clearLine(0);
+        this.stream.cursorTo(0);
+        this.stream.write(`✅ ${finalMessage}\n`);
+    }
+    clear() {
+        if (this.isJson)
+            return;
+        this.stream.clearLine(0);
+        this.stream.cursorTo(0);
+    }
+}
+/**
+ * Parses command-line arguments and environment variables.
+ * @returns The configuration object.
+ */
+async function getConfig() {
+    const argv = await yargs(hideBin(process.argv))
+        .option('api-base-url', {
+        alias: 'u',
+        type: 'string',
+        description: 'The base URL for the OpenAI-compatible API endpoint.',
+        default: process.env.LLM_API_URL,
+    })
+        .option('api-key', {
+        alias: 'k',
+        type: 'string',
+        description: 'The authentication key for the API.',
+        default: process.env.LLM_API_KEY,
+    })
+        .option('model', {
+        alias: 'm',
+        type: 'string',
+        description: 'The specific model to be benchmarked.',
+        default: process.env.LLM_MODEL_NAME,
+    })
+        .option('prompt', {
+        alias: 'p',
+        type: 'string',
+        description: 'The input text to send to the model.',
+        default: process.env.LLM_PROMPT,
+    })
+        .option('json', {
+        type: 'boolean',
+        description: 'Output the results in JSON format.',
+        default: false,
+    })
+        .help()
+        .alias('help', 'h')
+        .parse();
+    if (!argv.apiBaseUrl || !argv.apiKey || !argv.model || !argv.prompt) {
+        console.error('Error: Missing required arguments.');
+        console.error('Please provide api-base-url, api-key, model, and prompt via CLI arguments or environment variables.');
+        // Throwing an error is cleaner and more explicit for type checkers
+        throw new Error('Missing required arguments.');
+    }
+    const config = {
+        apiBaseUrl: argv.apiBaseUrl,
+        apiKey: argv.apiKey,
+        model: argv.model,
+        prompt: argv.prompt,
+        json: argv.json ?? false,
+    };
+    return config;
+}
+/**
+ * Runs the benchmark against the specified LLM provider.
+ * @param config - The configuration object.
+ */
+async function runBenchmark(config) {
+    const { apiBaseUrl, apiKey, model, prompt, json } = config;
+    const progress = new ProgressIndicator(json);
+    progress.start('Connecting...');
+    const T_start = performance.now();
+    let T_first_token = null;
+    const interTokenLatencies = [];
+    let lastChunkTime = 0;
+    let outputTokens = 0;
+    try {
+        const url = new URL(apiBaseUrl);
+        if (!url.pathname.endsWith('/')) {
+            url.pathname += '/';
+        }
+        let finalUrl;
+        if (url.pathname.endsWith('/v1/')) {
+            finalUrl = new URL('chat/completions', url);
+        }
+        else {
+            finalUrl = new URL('v1/chat/completions', url);
+        }
+        progress.update('Sending Request...');
+        const response = await fetch(finalUrl, {
+            method: 'POST',
+            headers: {
+                'Content-Type': 'application/json',
+                Authorization: `Bearer ${apiKey}`,
+            },
+            body: JSON.stringify({
+                model,
+                messages: [{ role: 'user', content: prompt }],
+                stream: true,
+            }),
+        });
+        if (!response.ok || !response.body) {
+            const errorBody = await response.text();
+            throw new Error(`API Error: ${response.status} ${response.statusText} - ${errorBody}`);
+        }
+        progress.update('Waiting for First Token...');
+        const reader = response.body.getReader();
+        const decoder = new TextDecoder();
+        while (true) {
+            const { done, value } = await reader.read();
+            if (done) {
+                break;
+            }
+            const chunkTime = performance.now();
+            if (T_first_token === null) {
+                T_first_token = chunkTime;
+            }
+            if (lastChunkTime > 0) {
+                interTokenLatencies.push(chunkTime - lastChunkTime);
+            }
+            lastChunkTime = chunkTime;
+            const chunk = decoder.decode(value, { stream: true });
+            const lines = chunk.split('\n').filter((line) => line.trim().startsWith('data:'));
+            for (const line of lines) {
+                const data = line.substring(5).trim();
+                if (data === '[DONE]') {
+                    break;
+                }
+                try {
+                    const parsed = JSON.parse(data);
+                    const content = parsed.choices[0]?.delta?.content;
+                    if (content) {
+                        outputTokens++;
+                        const elapsedS = (performance.now() - T_first_token) / 1000;
+                        const tokensPerSecond = elapsedS > 0 ? outputTokens / elapsedS : 0;
+                        progress.update(`Receiving Tokens... (${outputTokens} tokens received @ ${tokensPerSecond.toFixed(2)} tokens/s)`);
+                    }
+                }
+                catch (e) {
+                    // Ignore parsing errors for non-JSON lines
+                }
+            }
+        }
+        const T_end = performance.now();
+        progress.stop('Finished receiving tokens.');
+        const totalWallClockTimeMs = T_end - T_start;
+        const timeToFirstTokenMs = T_first_token ? T_first_token - T_start : 0;
+        // Estimate prompt tokens using tiktoken
+        const encoding = get_encoding('cl100k_base');
+        const promptTokens = encoding.encode(prompt).length;
+        encoding.free();
+        const outputDurationS = (totalWallClockTimeMs - timeToFirstTokenMs) / 1000;
+        const overallOutputRateTps = outputDurationS > 0 ? outputTokens / outputDurationS : 0;
+        const latencyStats = calculateLatencyStats(interTokenLatencies);
+        const results = {
+            configuration: {
+                apiBaseUrl,
+                model,
+            },
+            metrics: {
+                timeToFirstTokenMs: parseFloat(timeToFirstTokenMs.toFixed(2)),
+                totalWallClockTimeMs: parseFloat(totalWallClockTimeMs.toFixed(2)),
+                overallOutputRateTps: parseFloat(overallOutputRateTps.toFixed(2)),
+            },
+            tokenCounts: {
+                promptTokens,
+                outputTokens,
+            },
+            interTokenLatencyMs: {
+                ...latencyStats,
+                latencies: interTokenLatencies,
+            },
+        };
+        printResults(results, config.json);
+    }
+    catch (error) {
+        console.error('An error occurred during the benchmark:', error);
+        process.exit(1);
+    }
+}
+/**
+ * Prints the benchmark results to the console.
+ * @param results - The benchmark results object.
+ * @param asJson - Whether to print in JSON format.
+ */
+function printResults(results, asJson) {
+    if (asJson) {
+        // Create a new object for JSON output that doesn't include the full latencies array
+        const jsonOutput = {
+            ...results,
+            interTokenLatencyMs: {
+                ...results.interTokenLatencyMs,
+                latencies: undefined, // Remove the detailed list for the final JSON output
+            },
+        };
+        // delete jsonOutput.interTokenLatencyMs.latencies;
+        console.log(JSON.stringify(jsonOutput, null, 2));
+        return;
+    }
+    console.log('LLM Benchmark Results');
+    console.log('=======================');
+    console.log();
+    console.log('Configuration');
+    console.log('-----------------------');
+    console.log(`Provider API Base:   ${results.configuration.apiBaseUrl}`);
+    console.log(`Model:               ${results.configuration.model}`);
+    console.log();
+    console.log('Metrics');
+    console.log('-----------------------');
+    console.log(`Time to First Token:   ${results.metrics.timeToFirstTokenMs} ms`);
+    console.log(`Total Wall Clock Time: ${results.metrics.totalWallClockTimeMs} ms`);
+    console.log(`Overall Output Rate:   ${results.metrics.overallOutputRateTps} tokens/sec`);
+    console.log();
+    console.log('Token Counts');
+    console.log('-----------------------');
+    console.log(`Prompt Tokens:         ${results.tokenCounts.promptTokens} (estimated)`);
+    console.log(`Output Tokens:         ${results.tokenCounts.outputTokens}`);
+    console.log();
+    console.log('Inter-Token Latency (ms)');
+    console.log('-----------------------');
+    console.log(`Min:                 ${results.interTokenLatencyMs.min} ms`);
+    console.log(`Mean:                ${results.interTokenLatencyMs.mean} ms`);
+    console.log(`Median:              ${results.interTokenLatencyMs.median} ms`);
+    console.log(`Max:                 ${results.interTokenLatencyMs.max} ms`);
+    console.log(`p90:                 ${results.interTokenLatencyMs.p90} ms`);
+    console.log(`p95:                 ${results.interTokenLatencyMs.p95} ms`);
+    console.log(`p99:                 ${results.interTokenLatencyMs.p99} ms`);
+}
+// --- Entry Point ---
+(async () => {
+    try {
+        const config = await getConfig();
+        await runBenchmark(config);
+    }
+    catch (error) {
+        // Catch errors from getConfig and exit gracefully
+        // The error message is already printed inside getConfig
+        process.exit(1);
+    }
+})();
+//# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";AACA,OAAO,CAAC,GAAG,CAAC,oBAAoB,GAAG,MAAM,CAAC;AAE1C,OAAO,KAAK,MAAM,OAAO,CAAC;AAC1B,OAAO,EAAE,OAAO,EAAE,MAAM,eAAe,CAAC;AACxC,OAAO,MAAM,MAAM,QAAQ,CAAC;AAC5B,OAAO,EAAE,YAAY,EAAE,MAAM,UAAU,CAAC;AAExC,4CAA4C;AAC5C,MAAM,CAAC,MAAM,CAAC,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC;AA+C/B,4BAA4B;AAE5B;;;;;GAKG;AACH,MAAM,UAAU,GAAG,CAAC,GAAa,EAAE,CAAS,EAAU,EAAE;IACtD,IAAI,GAAG,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,CAAC,CAAC;IAC/B,MAAM,CAAC,GAAG,CAAC,CAAC,GAAG,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IACvC,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;IACxB,MAAM,CAAC,GAAG,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IACvB,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;QACZ,OAAO,GAAG,CAAC,CAAC,CAAE,CAAC;IACjB,CAAC;IACD,MAAM,IAAI,GAAG,GAAG,CAAC,CAAC,CAAE,CAAC;IACrB,MAAM,IAAI,GAAG,GAAG,CAAC,CAAC,CAAE,CAAC;IACrB,OAAO,IAAI,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,IAAI,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;AACzC,CAAC,CAAC;AAEF;;;;GAIG;AACH,MAAM,qBAAqB,GAAG,CAAC,SAAmB,EAAgB,EAAE;IAClE,IAAI,SAAS,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC3B,OAAO,EAAE,GAAG,EAAE,CAAC,EAAE,GAAG,EAAE,CAAC,EAAE,IAAI,EAAE,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,GAAG,EAAE,CAAC,EAAE,GAAG,EAAE,CAAC,EAAE,GAAG,EAAE,CAAC,EAAE,CAAC;IACxE,CAAC;IAED,MAAM,MAAM,GAAG,CAAC,GAAG,SAAS,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;IACpD,MAAM,GAAG,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC;IAC9C,MAAM,IAAI,GAAG,GAAG,GAAG,MAAM,CAAC,MAAM,CAAC;IACjC,MAAM,GAAG,GAAG,MAAM,CAAC,CAAC,CAAC,CAAC;IACtB,MAAM,GAAG,GAAG,MAAM,CAAC,MAAM,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IAEtC,IAAI,GAAG,KAAK,SAAS,IAAI,GAAG,KAAK,SAAS,EAAE,CAAC;QAC3C,iEAAiE;QACjE,4CAA4C;QAC5C,OAAO,EAAE,GAAG,EAAE,CAAC,EAAE,GAAG,EAAE,CAAC,EAAE,IAAI,EAAE,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,GAAG,EAAE,CAAC,EAAE,GAAG,EAAE,CAAC,EAAE,GAAG,EAAE,CAAC,EAAE,CAAC;IACxE,CAAC;IAED,MAAM,MAAM,GAAG,UAAU,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;IACtC,MAAM,GAAG,GAAG,UAAU,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;IACnC,MAAM,GAAG,GAAG,UAAU,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;IACnC,MAAM,GAAG,GAAG,UAAU,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;IAEnC,OAAO;QACL,GAAG,EAAE,UAAU,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;QAC/B,GAAG,EAAE,UAAU,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;QAC/B,IAAI,EAAE,UAAU,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;QACjC,MAAM,EAAE,UAAU,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;QACrC,GAAG,EAAE,UAAU,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;QAC/B,GAAG,EAAE,UAAU,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;QAC/B,GAAG,EAAE,UAAU,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;KAChC,CAAC;AACJ,CAAC,CAAC;AAEF,iCAAiC;AAEjC,MAAM,iBAAiB;IAOrB,YAAoB,MAAe;QAAf,WAAM,GAAN,MAAM,CAAS;QAN3B,WAAM,GAAuB,OAAO,CAAC,MAAM,CAAC;QAE5C,YAAO,GAAG,CAAC,GAAG,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,CAAC,CAAC;QAC7D,UAAK,GAAG,CAAC,CAAC;QACV,YAAO,GAAG,EAAE,CAAC;IAEiB,CAAC;IAE/B,MAAM;QACZ,IAAI,IAAI,CAAC,MAAM;YAAE,OAAO;QACxB,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC;QACzB,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC;QACxB,MAAM,YAAY,GAAG,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;QAC9C,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,YAAY,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC;QACrD,IAAI,CAAC,KAAK,GAAG,CAAC,IAAI,CAAC,KAAK,GAAG,CAAC,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC;IACtD,CAAC;IAED,KAAK,CAAC,OAAe;QACnB,IAAI,IAAI,CAAC,MAAM;YAAE,OAAO;QACxB,IAAI,CAAC,OAAO,GAAG,OAAO,CAAC;QACvB,IAAI,CAAC,KAAK,GAAG,WAAW,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,MAAM,EAAE,EAAE,EAAE,CAAC,CAAC;IACpD,CAAC;IAED,MAAM,CAAC,OAAe;QACpB,IAAI,IAAI,CAAC,MAAM;YAAE,OAAO;QACxB,IAAI,CAAC,OAAO,GAAG,OAAO,CAAC;IACzB,CAAC;IAED,IAAI,CAAC,YAAoB;QACvB,IAAI,IAAI,CAAC,MAAM;YAAE,OAAO;QACxB,IAAI,IAAI,CAAC,KAAK,EAAE,CAAC;YACf,aAAa,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;YAC1B,IAAI,CAAC,KAAK,GAAG,SAAS,CAAC;QACzB,CAAC;QACD,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC;QACzB,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC;QACxB,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,KAAK,YAAY,IAAI,CAAC,CAAC;IAC3C,CAAC;IAED,KAAK;QACH,IAAI,IAAI,CAAC,MAAM;YAAE,OAAO;QACxB,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC;QACzB,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC;IAC1B,CAAC;CACF;AAED;;;GAGG;AACH,KAAK,UAAU,SAAS;IACtB,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC;SAC5C,MAAM,CAAC,cAAc,EAAE;QACtB,KAAK,EAAE,GAAG;QACV,IAAI,EAAE,QAAQ;QACd,WAAW,EAAE,sDAAsD;QACnE,OAAO,EAAE,OAAO,CAAC,GAAG,CAAC,WAAW;KACjC,CAAC;SACD,MAAM,CAAC,SAAS,EAAE;QACjB,KAAK,EAAE,GAAG;QACV,IAAI,EAAE,QAAQ;QACd,WAAW,EAAE,qCAAqC;QAClD,OAAO,EAAE,OAAO,CAAC,GAAG,CAAC,WAAW;KACjC,CAAC;SACD,MAAM,CAAC,OAAO,EAAE;QACf,KAAK,EAAE,GAAG;QACV,IAAI,EAAE,QAAQ;QACd,WAAW,EAAE,uCAAuC;QACpD,OAAO,EAAE,OAAO,CAAC,GAAG,CAAC,cAAc;KACpC,CAAC;SACD,MAAM,CAAC,QAAQ,EAAE;QAChB,KAAK,EAAE,GAAG;QACV,IAAI,EAAE,QAAQ;QACd,WAAW,EAAE,sCAAsC;QACnD,OAAO,EAAE,OAAO,CAAC,GAAG,CAAC,UAAU;KAChC,CAAC;SACD,MAAM,CAAC,MAAM,EAAE;QACd,IAAI,EAAE,SAAS;QACf,WAAW,EAAE,oCAAoC;QACjD,OAAO,EAAE,KAAK;KACf,CAAC;SACD,IAAI,EAAE;SACN,KAAK,CAAC,MAAM,EAAE,GAAG,CAAC;SAClB,KAAK,EAAE,CAAC;IAEX,IAAI,CAAC,IAAI,CAAC,UAAU,IAAI,CAAC,IAAI,CAAC,MAAM,IAAI,CAAC,IAAI,CAAC,KAAK,IAAI,CAAC,IAAI,CAAC,MAAM,EAAE,CAAC;QACpE,OAAO,CAAC,KAAK,CAAC,oCAAoC,CAAC,CAAC;QACpD,OAAO,CAAC,KAAK,CAAC,qGAAqG,CAAC,CAAC;QACrH,mEAAmE;QACnE,MAAM,IAAI,KAAK,CAAC,6BAA6B,CAAC,CAAC;IACjD,CAAC;IAED,MAAM,MAAM,GAAW;QACrB,UAAU,EAAE,IAAI,CAAC,UAAoB;QACrC,MAAM,EAAE,IAAI,CAAC,MAAgB;QAC7B,KAAK,EAAE,IAAI,CAAC,KAAe;QAC3B,MAAM,EAAE,IAAI,CAAC,MAAgB;QAC7B,IAAI,EAAE,IAAI,CAAC,IAAI,IAAI,KAAK;KACzB,CAAC;IACF,OAAO,MAAM,CAAC;AAChB,CAAC;AAED;;;GAGG;AACH,KAAK,UAAU,YAAY,CAAC,MAAc;IACxC,MAAM,EAAE,UAAU,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,GAAG,MAAM,CAAC;IAC3D,MAAM,QAAQ,GAAG,IAAI,iBAAiB,CAAC,IAAI,CAAC,CAAC;IAE7C,QAAQ,CAAC,KAAK,CAAC,eAAe,CAAC,CAAC;IAChC,MAAM,OAAO,GAAG,WAAW,CAAC,GAAG,EAAE,CAAC;IAClC,IAAI,aAAa,GAAkB,IAAI,CAAC;IAExC,MAAM,mBAAmB,GAAa,EAAE,CAAC;IACzC,IAAI,aAAa,GAAG,CAAC,CAAC;IACtB,IAAI,YAAY,GAAG,CAAC,CAAC;IAErB,IAAI,CAAC;QACH,MAAM,GAAG,GAAG,IAAI,GAAG,CAAC,UAAU,CAAC,CAAC;QAChC,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC;YAChC,GAAG,CAAC,QAAQ,IAAI,GAAG,CAAC;QACtB,CAAC;QAED,IAAI,QAAQ,CAAC;QACb,IAAI,GAAG,CAAC,QAAQ,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;YAChC,QAAQ,GAAG,IAAI,GAAG,CAAC,kBAAkB,EAAE,GAAG,CAAC,CAAC;QAChD,CAAC;aAAM,CAAC;YACJ,QAAQ,GAAG,IAAI,GAAG,CAAC,qBAAqB,EAAE,GAAG,CAAC,CAAC;QACnD,CAAC;QAED,QAAQ,CAAC,MAAM,CAAC,oBAAoB,CAAC,CAAC;QACtC,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,QAAQ,EAAE;YACrC,MAAM,EAAE,MAAM;YACd,OAAO,EAAE;gBACP,cAAc,EAAE,kBAAkB;gBAClC,aAAa,EAAE,UAAU,MAAM,EAAE;aAClC;YACD,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;gBACnB,KAAK;gBACL,QAAQ,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,CAAC;gBAC7C,MAAM,EAAE,IAAI;aACb,CAAC;SACH,CAAC,CAAC;QAEH,IAAI,CAAC,QAAQ,CAAC,EAAE,IAAI,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAC;YACnC,MAAM,SAAS,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;YACxC,MAAM,IAAI,KAAK,CAAC,cAAc,QAAQ,CAAC,MAAM,IAAI,QAAQ,CAAC,UAAU,MAAM,SAAS,EAAE,CAAC,CAAC;QACzF,CAAC;QAED,QAAQ,CAAC,MAAM,CAAC,4BAA4B,CAAC,CAAC;QAC9C,MAAM,MAAM,GAAG,QAAQ,CAAC,IAAI,CAAC,SAAS,EAAE,CAAC;QACzC,MAAM,OAAO,GAAG,IAAI,WAAW,EAAE,CAAC;QAElC,OAAO,IAAI,EAAE,CAAC;YACZ,MAAM,EAAE,IAAI,EAAE,KAAK,EAAE,GAAG,MAAM,MAAM,CAAC,IAAI,EAAE,CAAC;YAC5C,IAAI,IAAI,EAAE,CAAC;gBACT,MAAM;YACR,CAAC;YAED,MAAM,SAAS,GAAG,WAAW,CAAC,GAAG,EAAE,CAAC;YACpC,IAAI,aAAa,KAAK,IAAI,EAAE,CAAC;gBAC3B,aAAa,GAAG,SAAS,CAAC;YAC5B,CAAC;YAED,IAAI,aAAa,GAAG,CAAC,EAAE,CAAC;gBACtB,mBAAmB,CAAC,IAAI,CAAC,SAAS,GAAG,aAAa,CAAC,CAAC;YACtD,CAAC;YACD,aAAa,GAAG,SAAS,CAAC;YAE1B,MAAM,KAAK,GAAG,OAAO,CAAC,MAAM,CAAC,KAAK,EAAE,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC;YACtD,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC,UAAU,CAAC,OAAO,CAAC,CAAC,CAAC;YAElF,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;gBACzB,MAAM,IAAI,GAAG,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;gBACtC,IAAI,IAAI,KAAK,QAAQ,EAAE,CAAC;oBACtB,MAAM;gBACR,CAAC;gBACD,IAAI,CAAC;oBACH,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;oBAChC,MAAM,OAAO,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC;oBAClD,IAAI,OAAO,EAAE,CAAC;wBACZ,YAAY,EAAE,CAAC;wBACf,MAAM,QAAQ,GAAG,CAAC,WAAW,CAAC,GAAG,EAAE,GAAG,aAAa,CAAC,GAAG,IAAI,CAAC;wBAC5D,MAAM,eAAe,GAAG,QAAQ,GAAG,CAAC,CAAC,CAAC,CAAC,YAAY,GAAG,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC;wBACnE,QAAQ,CAAC,MAAM,CAAC,wBAAwB,YAAY,sBAAsB,eAAe,CAAC,OAAO,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC;oBACpH,CAAC;gBACH,CAAC;gBAAC,OAAO,CAAC,EAAE,CAAC;oBACX,2CAA2C;gBAC7C,CAAC;YACH,CAAC;QACH,CAAC;QAED,MAAM,KAAK,GAAG,WAAW,CAAC,GAAG,EAAE,CAAC;QAChC,QAAQ,CAAC,IAAI,CAAC,4BAA4B,CAAC,CAAC;QAC5C,MAAM,oBAAoB,GAAG,KAAK,GAAG,OAAO,CAAC;QAC7C,MAAM,kBAAkB,GAAG,aAAa,CAAC,CAAC,CAAC,aAAa,GAAG,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC;QAEvE,wCAAwC;QACxC,MAAM,QAAQ,GAAG,YAAY,CAAC,aAAa,CAAC,CAAC;QAC7C,MAAM,YAAY,GAAG,QAAQ,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,MAAM,CAAC;QACpD,QAAQ,CAAC,IAAI,EAAE,CAAC;QAEhB,MAAM,eAAe,GAAG,CAAC,oBAAoB,GAAG,kBAAkB,CAAC,GAAG,IAAI,CAAC;QAC3E,MAAM,oBAAoB,GAAG,eAAe,GAAG,CAAC,CAAC,CAAC,CAAC,YAAY,GAAG,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;QAEtF,MAAM,YAAY,GAAG,qBAAqB,CAAC,mBAAmB,CAAC,CAAC;QAEhE,MAAM,OAAO,GAAqB;YAChC,aAAa,EAAE;gBACb,UAAU;gBACV,KAAK;aACN;YACD,OAAO,EAAE;gBACP,kBAAkB,EAAE,UAAU,CAAC,kBAAkB,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;gBAC7D,oBAAoB,EAAE,UAAU,CAAC,oBAAoB,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;gBACjE,oBAAoB,EAAE,UAAU,CAAC,oBAAoB,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;aAClE;YACD,WAAW,EAAE;gBACX,YAAY;gBACZ,YAAY;aACb;YACD,mBAAmB,EAAE;gBACnB,GAAG,YAAY;gBACf,SAAS,EAAE,mBAAmB;aAC/B;SACF,CAAC;QAEF,YAAY,CAAC,OAAO,EAAE,MAAM,CAAC,IAAI,CAAC,CAAC;IAErC,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,OAAO,CAAC,KAAK,CAAC,yCAAyC,EAAE,KAAK,CAAC,CAAC;QAChE,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;AACH,CAAC;AAGD;;;;GAIG;AACH,SAAS,YAAY,CAAC,OAAyB,EAAE,MAAe;IAC9D,IAAI,MAAM,EAAE,CAAC;QACX,oFAAoF;QACpF,MAAM,UAAU,GAAG;YACjB,GAAG,OAAO;YACV,mBAAmB,EAAE;gBACnB,GAAG,OAAO,CAAC,mBAAmB;gBAC9B,SAAS,EAAE,SAAS,EAAE,qDAAqD;aAC5E;SACF,CAAC;QACF,mDAAmD;QACnD,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,UAAU,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;QACjD,OAAO;IACT,CAAC;IAED,OAAO,CAAC,GAAG,CAAC,uBAAuB,CAAC,CAAC;IACrC,OAAO,CAAC,GAAG,CAAC,yBAAyB,CAAC,CAAC;IACvC,OAAO,CAAC,GAAG,EAAE,CAAC;IAEd,OAAO,CAAC,GAAG,CAAC,eAAe,CAAC,CAAC;IAC7B,OAAO,CAAC,GAAG,CAAC,yBAAyB,CAAC,CAAC;IACvC,OAAO,CAAC,GAAG,CAAC,wBAAwB,OAAO,CAAC,aAAa,CAAC,UAAU,EAAE,CAAC,CAAC;IACxE,OAAO,CAAC,GAAG,CAAC,wBAAwB,OAAO,CAAC,aAAa,CAAC,KAAK,EAAE,CAAC,CAAC;IACnE,OAAO,CAAC,GAAG,EAAE,CAAC;IAEd,OAAO,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;IACvB,OAAO,CAAC,GAAG,CAAC,yBAAyB,CAAC,CAAC;IACvC,OAAO,CAAC,GAAG,CAAC,0BAA0B,OAAO,CAAC,OAAO,CAAC,kBAAkB,KAAK,CAAC,CAAC;IAC/E,OAAO,CAAC,GAAG,CAAC,0BAA0B,OAAO,CAAC,OAAO,CAAC,oBAAoB,KAAK,CAAC,CAAC;IACjF,OAAO,CAAC,GAAG,CAAC,0BAA0B,OAAO,CAAC,OAAO,CAAC,oBAAoB,aAAa,CAAC,CAAC;IACzF,OAAO,CAAC,GAAG,EAAE,CAAC;IAEd,OAAO,CAAC,GAAG,CAAC,cAAc,CAAC,CAAC;IAC5B,OAAO,CAAC,GAAG,CAAC,yBAAyB,CAAC,CAAC;IACvC,OAAO,CAAC,GAAG,CAAC,0BAA0B,OAAO,CAAC,WAAW,CAAC,YAAY,cAAc,CAAC,CAAC;IACtF,OAAO,CAAC,GAAG,CAAC,0BAA0B,OAAO,CAAC,WAAW,CAAC,YAAY,EAAE,CAAC,CAAC;IAC1E,OAAO,CAAC,GAAG,EAAE,CAAC;IAEd,OAAO,CAAC,GAAG,CAAC,0BAA0B,CAAC,CAAC;IACxC,OAAO,CAAC,GAAG,CAAC,yBAAyB,CAAC,CAAC;IACvC,OAAO,CAAC,GAAG,CAAC,wBAAwB,OAAO,CAAC,mBAAmB,CAAC,GAAG,KAAK,CAAC,CAAC;IAC1E,OAAO,CAAC,GAAG,CAAC,wBAAwB,OAAO,CAAC,mBAAmB,CAAC,IAAI,KAAK,CAAC,CAAC;IAC3E,OAAO,CAAC,GAAG,CAAC,wBAAwB,OAAO,CAAC,mBAAmB,CAAC,MAAM,KAAK,CAAC,CAAC;IAC7E,OAAO,CAAC,GAAG,CAAC,wBAAwB,OAAO,CAAC,mBAAmB,CAAC,GAAG,KAAK,CAAC,CAAC;IAC1E,OAAO,CAAC,GAAG,CAAC,wBAAwB,OAAO,CAAC,mBAAmB,CAAC,GAAG,KAAK,CAAC,CAAC;IAC1E,OAAO,CAAC,GAAG,CAAC,wBAAwB,OAAO,CAAC,mBAAmB,CAAC,GAAG,KAAK,CAAC,CAAC;IAC1E,OAAO,CAAC,GAAG,CAAC,wBAAwB,OAAO,CAAC,mBAAmB,CAAC,GAAG,KAAK,CAAC,CAAC;AAC5E,CAAC;AAED,sBAAsB;AAEtB,CAAC,KAAK,IAAI,EAAE;IACV,IAAI,CAAC;QACH,MAAM,MAAM,GAAG,MAAM,SAAS,EAAE,CAAC;QACjC,MAAM,YAAY,CAAC,MAAM,CAAC,CAAC;IAC7B,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,kDAAkD;QAClD,wDAAwD;QACxD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;AACH,CAAC,CAAC,EAAE,CAAC"}

package/package.json

@@ -0,0 +1,41 @@
+{
+  "name": "llm-speed-bench",
+  "version": "1.0.0",
+  "description": "A CLI tool to benchmark the performance of OpenAI-compatible LLM providers.",
+  "main": "dist/index.js",
+  "type": "module",
+  "bin": {
+    "llm-speed-bench": "./dist/index.js"
+  },
+  "files": [
+    "dist"
+  ],
+  "scripts": {
+    "start": "ts-node src/index.ts",
+    "build": "tsc",
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "keywords": [
+    "llm",
+    "benchmark",
+    "performance",
+    "openai",
+    "speed"
+  ],
+  "author": "",
+  "license": "ISC",
+  "dependencies": {
+    "@types/node": "^24.3.0",
+    "@types/yargs": "^17.0.33",
+    "dotenv": "^17.2.2",
+    "tiktoken": "^1.0.22",
+    "ts-node": "^10.9.2",
+    "typescript": "^5.9.2",
+    "yargs": "^18.0.0"
+  },
+  "devDependencies": {
+    "@typescript-eslint/eslint-plugin": "^8.42.0",
+    "@typescript-eslint/parser": "^8.42.0",
+    "eslint": "^9.34.0"
+  }
+}