@aistupidlevel/mcp 0.0.0-stage → 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Studio Platforms (AI Stupid Level)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,224 @@
-# Temporary Holding Version
+# @aistupidlevel/mcp
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+An [MCP](https://modelcontextprotocol.io) server for [AI Stupid Level](https://aistupidlevel.info) (ASL).
+It lets Claude, Cursor, VS Code or any other MCP client read ASL's LLM leaderboard, score history,
+drift status, incidents and provider status. With a router key it can also ask the
+[ASL Smart Router](https://aistupidlevel.info/router) which model it would pick for a prompt, or
+send the prompt through the router.
+
+Ask your assistant things like:
+
+- "Which model is best for coding right now, according to AI Stupid Level?"
+- "Has Claude Opus 5.5 got worse this week?"
+- "Were there any provider incidents in the last few days?"
+- "Which model would the Smart Router pick for this prompt, and why?"
+
+It runs locally over stdio and needs Node.js 18 or later.
+
+## Keys
+
+The tools use two different keys, and you only need the one for the tools you want.
+
+| Key | Used by | Where to get it |
+|---|---|---|
+| `ASL_DATA_KEY` (`asl_live_…`) | leaderboard, model, history, drift, incidents, providers, recommendations, global index | <https://aistupidlevel.info/account/data-keys> (free tier available) |
+| `ASL_ROUTER_KEY` (`aism_…`) | `preview_route`, `ask_router`, router usage | <https://aistupidlevel.info/router/keys>, plus at least one provider key in your account |
+
+A tool whose key is missing does not fail the server. It answers with a message that says which
+key it needs and where to create it.
+
+`ASL_API_KEY` is also accepted: it is used as the data key if it starts with `asl_`, and as the router
+key if it starts with `aism_`.
+
+### Free-tier limits
+
+A free Data API key allows **10 requests a day and 1 a minute**. Paid plans allow 10,000 a day or more
+([pricing](https://aistupidlevel.info/pricing)). The free tier is meant for trying the API, so the
+server is careful with it:
+
+- Every answer is cached in memory for 15 minutes (`ASL_CACHE_TTL`). Asking the same question again,
+  or for another vendor or a shorter list from the same leaderboard, makes no request, and model
+  names are looked up in the cached leaderboard.
+- One tool call makes at most one Data API request. The exception is a model given by name before
+  any leaderboard has been fetched, which takes one extra request to look up the model ids.
+- Once the daily quota is used up, the server stops sending requests until 00:00 UTC. A request
+  refused for quota still counts against it.
+- If a fresh answer cannot be fetched, the server returns the last cached answer (up to 24 hours old)
+  and says how old it is.
+- Every answer ends with the number of requests left today.
+
+On a free key, calling several different data tools within a minute hits the per-minute limit. The
+tool then says when to try again. If the wait is 15 seconds or less (`ASL_MAX_WAIT`), the server waits
+and retries once by itself.
+
+The newest-run view (`period: "latest"`) needs a paid key. Free keys read the 24-hour, 7-day and
+30-day views.
+
+`preview_route` and `get_usage` are not counted against your Smart Router plan. `ask_router` is a
+real routed request: the router calls the model with your own provider key, so the provider bills
+you and the request counts against your plan. Pass `sandbox: true`, or set `ASL_SANDBOX=true`, to get
+mock answers from the real routing at no cost.
+
+## Install
+
+The package runs with `npx`, so there is nothing to install first. Put your keys in place of the
+placeholders, and leave out any key you do not have.
+
+### Claude Code
+
+```sh
+claude mcp add aistupidlevel -e ASL_DATA_KEY=asl_live_... -e ASL_ROUTER_KEY=aism_... -- npx -y @aistupidlevel/mcp
+```
+
+Add `-s user` to make it available in every project.
+
+### Claude Desktop
+
+Settings → Developer → Edit Config, then add the server to `claude_desktop_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "aistupidlevel": {
+      "command": "npx",
+      "args": ["-y", "@aistupidlevel/mcp"],
+      "env": {
+        "ASL_DATA_KEY": "asl_live_...",
+        "ASL_ROUTER_KEY": "aism_..."
+      }
+    }
+  }
+}
+```
+
+Restart Claude Desktop afterwards.
+
+### Cursor
+
+Add the following to `~/.cursor/mcp.json` (for all projects) or `.cursor/mcp.json` (for one project):
+
+```json
+{
+  "mcpServers": {
+    "aistupidlevel": {
+      "command": "npx",
+      "args": ["-y", "@aistupidlevel/mcp"],
+      "env": {
+        "ASL_DATA_KEY": "asl_live_...",
+        "ASL_ROUTER_KEY": "aism_..."
+      }
+    }
+  }
+}
+```
+
+### VS Code
+
+Add the following to `.vscode/mcp.json` (or to your user configuration with **MCP: Open User
+Configuration**). VS Code prompts for the keys and stores them securely:
+
+```json
+{
+  "inputs": [
+    { "type": "promptString", "id": "asl-data-key", "description": "AI Stupid Level Data API key (asl_live_...)", "password": true },
+    { "type": "promptString", "id": "asl-router-key", "description": "AI Stupid Level router key (aism_...)", "password": true }
+  ],
+  "servers": {
+    "aistupidlevel": {
+      "type": "stdio",
+      "command": "npx",
+      "args": ["-y", "@aistupidlevel/mcp"],
+      "env": {
+        "ASL_DATA_KEY": "${input:asl-data-key}",
+        "ASL_ROUTER_KEY": "${input:asl-router-key}"
+      }
+    }
+  }
+}
+```
+
+### Other clients
+
+Any client that can start a stdio server can run it: the command is `npx -y @aistupidlevel/mcp`,
+and the keys go in the environment variables listed under [Configuration](#configuration).
+
+## Tools
+
+All tools except `ask_router` are read-only.
+
+| Tool | Key | What it returns |
+|---|---|---|
+| `get_leaderboard` | data | Tracked models ranked by combined, coding, reasoning or tool-use score, or by score per list-price dollar. Each row has the score, trend, status and stability, and the model ids the other tools take. Filter by vendor; choose the view (24h, 7d, 1m, or latest on a paid key). |
+| `get_model` | data | One model's details: vendor, version, notes, tool-calling support and days tracked. |
+| `get_model_history` | data | One model's score for each run in the window, with a summary and the per-axis results of the newest run. |
+| `get_drift_status` | data | One model's drift signature: regime, current score against its baseline, the change-point detector for each suite, the last significant change, and which axes moved. |
+| `get_incidents` | data | Recent incidents (provider-wide and per model) with severity, the models affected, when each was detected and when it was resolved. |
+| `get_provider_status` | data | Each provider's API status from ASL's health probes, with the probe's response time and the last error. |
+| `get_recommendations` | data | ASL's current picks (best for coding, most reliable, fastest, best value, poor value, avoid now), each with its reason and the evidence behind it. |
+| `get_global_index` | data | ASL's global index across the tracked models, now and over the last 24 hours. |
+| `preview_route` | router | The model the Smart Router would pick for a prompt, its score and estimated cost, the reason, and the fallback chain. Nothing is called or counted. |
+| `ask_router` | router | Sends a prompt through the Smart Router and returns the answer, the model and provider that answered, why it was chosen, the latency and the tokens used. Set `sandbox: true` for a mock answer at no cost. |
+| `get_usage` | either | This month's router usage and budget, and the Data API requests left today. |
+
+Models can be given by numeric id, by name (`claude-opus-5-5`) or by display name (`Claude Opus 5.5`).
+
+`preview_route` and `ask_router` take a routing `mode` (`auto`, `auto-best`, `auto-coding`,
+`auto-reasoning`, `auto-tooling`, `auto-task`, `auto-value`, `auto-value-coding`,
+`auto-value-reasoning`, `auto-value-tooling`, `auto-fastest-quality`, `auto-consistent`,
+`auto-fastest`, `auto-cheapest`), or `pin_model` to name a model directly. The
+[router documentation](https://aistupidlevel.info/router/docs#strategies) explains each mode.
+
+The server also provides one prompt, `choose_model`, which walks the assistant through choosing a
+model for a task you describe.
+
+Answers from the Data API end with a source line. The data is licensed with attribution required:
+cite aistupidlevel.info when you republish it.
+
+## Configuration
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `ASL_DATA_KEY` | none | Data API key (`asl_live_…`) |
+| `ASL_ROUTER_KEY` | none | Smart Router key (`aism_…`) |
+| `ASL_SANDBOX` | `false` | `true` puts every Smart Router call in sandbox mode: real routing, mock answer, nothing spent or counted |
+| `ASL_CACHE_TTL` | `900` | Seconds a Data API answer is reused |
+| `ASL_MAX_WAIT` | `15` | When the per-minute limit asks for a wait this short (seconds) or shorter, wait and retry once. `0` turns this off |
+| `ASL_TIMEOUT_MS` | `300000` | Timeout for a Smart Router request |
+
+Keys are never logged. The server writes its diagnostics to stderr, because stdout carries the MCP
+protocol.
+
+## Use as a library
+
+```ts
+import { createServer } from '@aistupidlevel/mcp';
+import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
+
+const server = createServer({ dataKey: process.env.ASL_DATA_KEY, cacheTtlSeconds: 600 });
+await server.connect(new StdioServerTransport());
+```
+
+`createServer` accepts the same settings as the environment variables, plus a custom `fetch`.
+
+## Development
+
+```sh
+npm ci && npm run build && npm test
+```
+
+The unit tests start the server in-process, connect an MCP client to it over an in-memory transport,
+and replay real API responses from `test/fixtures`. The live tests are skipped unless keys are set:
+
+```sh
+ASL_ROUTER_KEY=aism_... npm test    # Smart Router, sandbox only: nothing is spent or counted
+ASL_DATA_KEY=asl_live_... npm test  # Data API: one request
+```
+
+## Links
+
+- [Data API documentation](https://aistupidlevel.info/api-docs)
+- [Smart Router documentation](https://aistupidlevel.info/router/docs)
+- [TypeScript SDK for the router](https://www.npmjs.com/package/@aistupidlevel/router)
+- [Leaderboard](https://aistupidlevel.info)
+
+MIT licence.

package/dist/cli.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/dist/cli.js

@@ -0,0 +1,40 @@
+#!/usr/bin/env node
+/**
+ * aistupidlevel-mcp: the AI Stupid Level MCP server over stdio.
+ * stdout carries the protocol, so every diagnostic goes to stderr, and keys are never printed.
+ */
+import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
+import { createServer } from './server.js';
+import { VERSION, resolveConfig } from './config.js';
+const HELP = `aistupidlevel-mcp ${VERSION}: MCP server for AI Stupid Level (https://aistupidlevel.info)
+
+Usage: npx -y @aistupidlevel/mcp      (speaks MCP over stdio; add it to your MCP client)
+
+Environment:
+  ASL_DATA_KEY     Data API key (asl_live_...) for rankings, history, drift, incidents.
+                   Free key: https://aistupidlevel.info/account/data-keys (10 requests/day, 1/minute)
+  ASL_ROUTER_KEY   Smart Router key (aism_...) for preview_route, ask_router and usage.
+                   https://aistupidlevel.info/router/keys
+  ASL_SANDBOX      true: every Smart Router call returns a mock answer (nothing spent)
+  ASL_CACHE_TTL    seconds a Data API answer is reused (default 900)
+  ASL_MAX_WAIT     wait and retry once when the per-minute limit asks for at most this many seconds (default 15)
+  ASL_TIMEOUT_MS   Smart Router request timeout in ms (default 300000)
+`;
+const args = process.argv.slice(2);
+if (args.includes('--help') || args.includes('-h')) {
+    process.stderr.write(HELP);
+    process.exit(0);
+}
+if (args.includes('--version') || args.includes('-v')) {
+    process.stderr.write(`${VERSION}\n`);
+    process.exit(0);
+}
+const log = (m) => process.stderr.write(`[aistupidlevel-mcp] ${m}\n`);
+const cfg = resolveConfig();
+const server = createServer({ log });
+const transport = new StdioServerTransport();
+await server.connect(transport);
+log(`v${VERSION} running on stdio; Data API key ${cfg.dataKey ? 'set' : 'not set (ASL_DATA_KEY)'}, router key ${cfg.routerKey ? 'set' : 'not set (ASL_ROUTER_KEY)'}${cfg.forceSandbox ? ', sandbox forced' : ''}.`);
+const shutdown = () => { server.close().finally(() => process.exit(0)); };
+process.on('SIGINT', shutdown);
+process.on('SIGTERM', shutdown);

package/dist/config.d.ts

@@ -0,0 +1,59 @@
+/** Settings, read from the environment by default. Keys are never logged or echoed back. */
+export declare const VERSION = "0.1.0";
+export declare const DEFAULT_DATA_BASE_URL = "https://aistupidlevel.info/api/v1";
+export declare const DEFAULT_ROUTER_BASE_URL = "https://aistupidlevel.info/v1";
+export declare const LINKS: {
+    readonly dataKeys: "https://aistupidlevel.info/account/data-keys";
+    readonly dataDocs: "https://aistupidlevel.info/api-docs";
+    readonly routerKeys: "https://aistupidlevel.info/router/keys";
+    readonly routerDocs: "https://aistupidlevel.info/router/docs";
+    readonly providerKeys: "https://aistupidlevel.info/router/providers";
+    readonly pricing: "https://aistupidlevel.info/pricing";
+    readonly billing: "https://aistupidlevel.info/account/billing";
+    readonly site: "https://aistupidlevel.info";
+};
+export interface ServerOptions {
+    /** Data API key (asl_live_…). Default: ASL_DATA_KEY, or ASL_API_KEY when it is an asl_ key. */
+    dataKey?: string;
+    /** Smart Router key (aism_…). Default: ASL_ROUTER_KEY, or ASL_API_KEY when it is an aism_ key. */
+    routerKey?: string;
+    /** Default: ASL_DATA_BASE_URL or https://aistupidlevel.info/api/v1 */
+    dataBaseURL?: string;
+    /** Default: ASL_BASE_URL or https://aistupidlevel.info/v1 */
+    routerBaseURL?: string;
+    /** How long a Data API response is reused, in seconds. Default: ASL_CACHE_TTL or 900 (15 minutes). */
+    cacheTtlSeconds?: number;
+    /**
+     * When the Data API's per-minute limit answers 429 with a Retry-After of at most this many
+     * seconds, wait and retry once. Default: ASL_MAX_WAIT or 15. 0 never waits.
+     */
+    maxWaitSeconds?: number;
+    /** Force sandbox mode for every Smart Router call (mock answers, nothing spent). Default: ASL_SANDBOX. */
+    forceSandbox?: boolean;
+    /** Per-request timeout for Smart Router calls, ms. Default: ASL_TIMEOUT_MS or 300000. */
+    routerTimeoutMs?: number;
+    /** A fetch implementation (tests, proxies). Default: the global fetch. */
+    fetch?: typeof fetch;
+    /** Read settings from this object instead of process.env. */
+    env?: Record<string, string | undefined>;
+    /** Where diagnostics go. Default: stderr (stdout belongs to the stdio transport). */
+    log?: (message: string) => void;
+}
+export interface ResolvedConfig {
+    dataKey: string | null;
+    routerKey: string | null;
+    dataBaseURL: string;
+    routerBaseURL: string;
+    cacheTtlMs: number;
+    maxWaitMs: number;
+    forceSandbox: boolean;
+    routerTimeoutMs: number;
+    fetch: typeof fetch;
+    log: (message: string) => void;
+    /** Problems with the configuration, reported by the tools that are affected. */
+    warnings: {
+        data?: string;
+        router?: string;
+    };
+}
+export declare function resolveConfig(o?: ServerOptions): ResolvedConfig;

package/dist/config.js

@@ -0,0 +1,61 @@
+/** Settings, read from the environment by default. Keys are never logged or echoed back. */
+export const VERSION = '0.1.0';
+export const DEFAULT_DATA_BASE_URL = 'https://aistupidlevel.info/api/v1';
+export const DEFAULT_ROUTER_BASE_URL = 'https://aistupidlevel.info/v1';
+export const LINKS = {
+    dataKeys: 'https://aistupidlevel.info/account/data-keys',
+    dataDocs: 'https://aistupidlevel.info/api-docs',
+    routerKeys: 'https://aistupidlevel.info/router/keys',
+    routerDocs: 'https://aistupidlevel.info/router/docs',
+    providerKeys: 'https://aistupidlevel.info/router/providers',
+    pricing: 'https://aistupidlevel.info/pricing',
+    billing: 'https://aistupidlevel.info/account/billing',
+    site: 'https://aistupidlevel.info',
+};
+const truthy = (v) => !!v && /^(1|true|yes|on)$/i.test(v.trim());
+const num = (v, fallback) => {
+    if (v === undefined || v.trim() === '')
+        return fallback;
+    const n = Number(v);
+    return Number.isFinite(n) && n >= 0 ? n : fallback;
+};
+const clean = (v) => (v && v.trim() ? v.trim() : null);
+export function resolveConfig(o = {}) {
+    const env = o.env ?? globalThis.process?.env ?? {};
+    const generic = clean(env.ASL_API_KEY);
+    let dataKey = clean(o.dataKey) ?? clean(env.ASL_DATA_KEY) ?? (generic?.startsWith('asl_') ? generic : null);
+    let routerKey = clean(o.routerKey) ?? clean(env.ASL_ROUTER_KEY) ?? (generic?.startsWith('aism_') ? generic : null);
+    const warnings = {};
+    // The two products use different keys; a swapped pair is the most likely set-up mistake.
+    if (dataKey?.startsWith('aism_')) {
+        warnings.data = 'ASL_DATA_KEY holds a Smart Router key (aism_…). The Data API needs a Data API key (asl_live_…).';
+        if (!routerKey)
+            routerKey = dataKey;
+        dataKey = null;
+    }
+    if (routerKey?.startsWith('asl_')) {
+        warnings.router = 'ASL_ROUTER_KEY holds a Data API key (asl_…). The Smart Router needs a router key (aism_…).';
+        if (!dataKey)
+            dataKey = routerKey;
+        routerKey = null;
+    }
+    const f = o.fetch ?? globalThis.fetch;
+    if (typeof f !== 'function')
+        throw new Error('No fetch implementation is available: use Node 18 or later.');
+    return {
+        dataKey,
+        routerKey,
+        dataBaseURL: (o.dataBaseURL ?? clean(env.ASL_DATA_BASE_URL) ?? DEFAULT_DATA_BASE_URL).replace(/\/+$/, ''),
+        routerBaseURL: (o.routerBaseURL ?? clean(env.ASL_BASE_URL) ?? DEFAULT_ROUTER_BASE_URL).replace(/\/+$/, ''),
+        cacheTtlMs: (o.cacheTtlSeconds ?? num(env.ASL_CACHE_TTL, 900)) * 1000,
+        maxWaitMs: (o.maxWaitSeconds ?? num(env.ASL_MAX_WAIT, 15)) * 1000,
+        forceSandbox: o.forceSandbox ?? truthy(env.ASL_SANDBOX),
+        routerTimeoutMs: o.routerTimeoutMs ?? num(env.ASL_TIMEOUT_MS, 300_000),
+        fetch: o.fetch ? f : f.bind(globalThis),
+        log: o.log ?? ((m) => { try {
+            globalThis.process?.stderr?.write(`[aistupidlevel-mcp] ${m}\n`);
+        }
+        catch { /* ignore */ } }),
+        warnings,
+    };
+}

package/dist/data-api.d.ts

@@ -0,0 +1,76 @@
+/**
+ * Client for the ASL Data API (https://aistupidlevel.info/api/v1).
+ *
+ * A free key allows 10 requests a day and 1 a minute, so every response is cached in memory
+ * (default 15 minutes, keyed by URL), identical requests in flight are shared, refusals that will
+ * not change (403, 404) are cached too, and once the daily quota is spent no further request is
+ * sent until it resets. When a request cannot be made, an older cached answer is served with a
+ * note saying how old it is.
+ */
+import { type ResolvedConfig } from './config.js';
+import { ToolError } from './errors.js';
+/** The envelope every Data API response is wrapped in. */
+export interface Envelope<T = unknown> {
+    success: boolean;
+    version?: string;
+    generated_at?: string;
+    license?: string;
+    period?: string;
+    sort_by?: string;
+    count?: number;
+    limit?: number;
+    data: T;
+    [extra: string]: unknown;
+}
+export interface DataResult<T = unknown> {
+    body: Envelope<T>;
+    /** When this answer was fetched from the API. */
+    fetchedAt: number;
+    cached: boolean;
+    /** Set when an expired answer is served because a fresh one could not be fetched. */
+    staleReason?: string;
+}
+export interface Quota {
+    limit: number;
+    remaining: number;
+    /** Epoch ms of the next reset (00:00 UTC). */
+    resetAt: number | null;
+}
+export interface ModelRef {
+    id: number;
+    name: string;
+    displayName: string | null;
+}
+export declare class DataApi {
+    private readonly cfg;
+    private readonly cache;
+    private readonly inflight;
+    private readonly models;
+    private modelsSeenAt;
+    /** The newest leaderboard (/models) answer, for showing a model's row without a request. */
+    lastBoard: Envelope | null;
+    private dailyBlockedUntil;
+    private dailyBlockedMessage;
+    private lastQuota;
+    /** Network requests actually sent (for tests and diagnostics). */
+    requestsSent: number;
+    constructor(cfg: ResolvedConfig);
+    get hasKey(): boolean;
+    /** The quota from the newest response, or null once its day is over. */
+    get quota(): Quota | null;
+    /** The error a data tool returns when no Data API key is configured. */
+    missingKeyError(): ToolError;
+    /**
+     * GET a Data API path. `query` values that are undefined are left out, so "let the API choose"
+     * and an explicit value are cached separately, exactly as the API would answer them.
+     */
+    get<T = unknown>(path: string, query?: Record<string, string | number | undefined>): Promise<DataResult<T>>;
+    /** Resolve a model given as a numeric id, a model name (claude-opus-5-5) or a display name. */
+    resolveModel(input: string): Promise<ModelRef>;
+    /** Every model the last /models answer listed (no request is made). */
+    knownModel(id: number): ModelRef | undefined;
+    private learnModels;
+    private staleOr;
+    private readQuota;
+    private fetchUrl;
+}