@aggc/or-info 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 or-info contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,294 @@
+# or-info
+
+> CLI + MCP server to query OpenRouter model info: prices, ELO rankings, context and comparisons.
+
+Any person or AI agent (Claude Code, Cursor, pi, etc.) can install it and use it
+to make informed decisions about which model to use.
+
+[![npm version](https://img.shields.io/npm/v/@aggc/or-info.svg)](https://www.npmjs.com/package/@aggc/or-info)
+[![CI](https://github.com/jmtrs/or-info/actions/workflows/ci.yml/badge.svg)](https://github.com/jmtrs/or-info/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+## Install
+
+```bash
+npm install -g @aggc/or-info
+```
+
+Requires Node.js 22 or later.
+
+Supported runtimes and platforms:
+
+- Node.js 22+
+- macOS, Linux, Windows
+
+## Config and cache paths
+
+`or-info` resolves config and cache natively per platform:
+
+| Platform | Config directory | Cache directory |
+|----------|------------------|-----------------|
+| macOS / Linux | `$XDG_CONFIG_HOME/or-info` or `~/.config/or-info` | `$XDG_CACHE_HOME/or-info` or `~/.cache/or-info` |
+| Windows | `%APPDATA%\\or-info` | `%LOCALAPPDATA%\\or-info` |
+| Any platform | `OR_INFO_CONFIG_DIR` override | `OR_INFO_CACHE_DIR` override |
+
+Files of interest:
+
+- Config file: `<config-dir>/.env`
+- Model cache: `<cache-dir>/models.json`
+- LMArena cache: `<cache-dir>/benchmarks.json`
+
+## API key (optional)
+
+Without an API key the CLI works with OpenRouter's public catalog.
+With a key you also see private/pay-gated models.
+
+### Bash / Zsh
+
+```bash
+export OPENROUTER_API_KEY=sk-or-...
+```
+
+### PowerShell
+
+```powershell
+$env:OPENROUTER_API_KEY = "sk-or-..."
+```
+
+### CMD
+
+```cmd
+set OPENROUTER_API_KEY=sk-or-...
+```
+
+### Config file
+
+macOS / Linux:
+
+```bash
+mkdir -p ~/.config/or-info
+echo 'OPENROUTER_API_KEY=sk-or-...' >> ~/.config/or-info/.env
+```
+
+Windows PowerShell:
+
+```powershell
+New-Item -ItemType Directory -Force "$env:APPDATA\or-info" | Out-Null
+Add-Content -Path "$env:APPDATA\or-info\.env" -Value "OPENROUTER_API_KEY=sk-or-..."
+```
+
+Windows CMD:
+
+```cmd
+if not exist "%APPDATA%\or-info" mkdir "%APPDATA%\or-info"
+echo OPENROUTER_API_KEY=sk-or-...>> "%APPDATA%\or-info\.env"
+```
+
+For tests and debugging you can redirect storage without touching your real machine state:
+
+```bash
+OR_INFO_CONFIG_DIR=/tmp/or-info-config OR_INFO_CACHE_DIR=/tmp/or-info-cache or-info status
+```
+
+## CLI usage
+
+### List models
+
+```bash
+or-info models                        # All models sorted by name
+or-info models --sort price           # Cheapest output first
+or-info models --sort context         # Largest context first
+or-info models --filter coding        # Models whose ID/name contains "coding"
+or-info models --free                 # Free models only
+or-info models --limit 20             # Limit the number of results
+or-info models --tags                 # Show feature tags (vision, tools, reasoning…)
+or-info models --json                 # Raw JSON
+```
+
+### Pricing and details
+
+```bash
+or-info price anthropic/claude-sonnet-4.5
+or-info price google/gemini-2.5-flash --json
+```
+
+### ELO ranking (LMArena)
+
+```bash
+or-info benchmark openai/gpt-4o
+or-info benchmark deepseek/deepseek-r1 --json
+```
+
+Shows the model's ELO score from [LMArena](https://lmarena.ai) (human preference votes),
+confidence interval, global rank and vote count. Data is fetched live from HuggingFace
+and cached locally for 24 hours — no API key required.
+
+### Compare two models
+
+```bash
+or-info compare anthropic/claude-sonnet-4.5 google/gemini-2.5-flash
+or-info compare openai/gpt-4o deepseek/deepseek-chat-v3-0324 --json
+```
+
+### Top models for a task
+
+```bash
+or-info top --task coding             # Best coding models
+or-info top --task reasoning          # Best reasoning models
+or-info top --task general            # Best all-rounders
+or-info top --task vision             # Best vision models
+or-info top --task cheap              # Best value for money
+or-info top --task coding --budget 2  # Best coders under $2/M output
+or-info top --task general --limit 10
+```
+
+Ranking combines LMArena ELO with price. `--task vision` and `--task coding` additionally
+filter for models that support the required capability (image input / tool use).
+
+### Cache management
+
+```bash
+or-info status          # Show cache age and TTL for each data source
+or-info refresh         # Force-refresh OpenRouter catalog + LMArena ELO
+```
+
+## MCP server
+
+`or-info` can run as an MCP server for AI agents.
+
+### Tools available
+
+| Tool | Description |
+|------|-------------|
+| `get_model_info` | Pricing, context, architecture, features and LMArena ELO for a model |
+| `list_models` | List models with optional filter, sort and limit |
+| `get_benchmarks` | LMArena ELO, rank, votes and confidence interval for a model |
+| `compare_models` | Side-by-side comparison of two models |
+| `best_for_task` | Ranked top models for coding/reasoning/general/vision/cheap |
+| `refresh_cache` | Force-refresh OpenRouter catalog + LMArena ELO |
+
+### Register in Claude Code
+
+Add to `~/.claude/settings.json`:
+
+```json
+{
+  "mcpServers": {
+    "or-info": {
+      "command": "or-info",
+      "args": ["--mcp"],
+      "env": {
+        "OPENROUTER_API_KEY": "sk-or-..."
+      }
+    }
+  }
+}
+```
+
+Windows global install:
+
+```json
+{
+  "mcpServers": {
+    "or-info": {
+      "command": "or-info.cmd",
+      "args": ["--mcp"],
+      "env": {
+        "OPENROUTER_API_KEY": "sk-or-..."
+      }
+    }
+  }
+}
+```
+
+If you installed via `npx` (without global install) on macOS/Linux:
+
+```json
+{
+  "mcpServers": {
+    "or-info": {
+      "command": "npx",
+      "args": ["-y", "@aggc/or-info", "--mcp"]
+    }
+  }
+}
+```
+
+If you installed via `npx` on Windows:
+
+```json
+{
+  "mcpServers": {
+    "or-info": {
+      "command": "npx.cmd",
+      "args": ["-y", "@aggc/or-info", "--mcp"]
+    }
+  }
+}
+```
+
+Then verify:
+
+```bash
+claude mcp list
+```
+
+### Use from Pi
+
+Pi does not use an `mcpServers` settings schema. The recommended integration is a Pi skill
+that calls the installed `or-info` CLI, for example `~/.pi/agent/skills/or-info/SKILL.md`.
+
+### Test the MCP server
+
+macOS / Linux:
+
+```bash
+echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | or-info --mcp
+```
+
+Windows PowerShell:
+
+```powershell
+'{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | or-info.cmd --mcp
+```
+
+Windows CMD:
+
+```cmd
+echo {"jsonrpc":"2.0","id":1,"method":"tools/list"} | or-info.cmd --mcp
+```
+
+## Data sources
+
+| Data | Source | Refresh |
+|------|--------|---------|
+| Model catalog and pricing | [OpenRouter API](https://openrouter.ai/api/v1/models) | Every 30 min |
+| ELO rankings | [LMArena](https://lmarena.ai) via [HuggingFace dataset](https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset) | Every 24 h |
+
+ELO data is fetched directly from the `lmarena-ai/leaderboard-dataset` dataset on HuggingFace
+using their public Datasets Server API — no API key required. Coverage: ~350 models
+including all major commercial and open-source models tracked by LMArena.
+
+## Testing
+
+```bash
+npm test
+```
+
+`npm test` runs the deterministic local suite and is the release gate used before publishing.
+Live integration tests are available separately because they depend on OpenRouter and
+HuggingFace availability and can occasionally hit third-party rate limits.
+
+Additional entry points:
+
+- `npm run test:local` for deterministic no-network coverage
+- `npm run test:online:smoke` for the live smoke subset used by CI as a non-blocking signal
+- `npm run test:online` for the full live CLI/MCP suite, including edge cases
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) — adding new CLI commands or new MCP tools.
+
+## License
+
+[MIT](LICENSE)

package/bin/or-info.mjs

@@ -0,0 +1,240 @@
+#!/usr/bin/env node
+import { InvalidArgumentError, program } from 'commander';
+import chalk from 'chalk';
+import { fetchModels, findModel, pricePerMillion, contextLength } from '../lib/openrouter.mjs';
+import { getElo, getAllElo, loadLeaderboard } from '../lib/lmarena.mjs';
+import { rankModels } from '../lib/scorer.mjs';
+import { clearAll, status } from '../lib/cache.mjs';
+import { getApiKey } from '../lib/secrets.mjs';
+import {
+  modelTable,
+  modelDetail,
+  comparison,
+  topList,
+  statusReport,
+} from '../lib/formatter.mjs';
+
+function die(msg) {
+  console.error(chalk.red('Error: ') + msg);
+  process.exit(1);
+}
+
+const TOP_TASKS = new Set(['coding', 'reasoning', 'general', 'vision', 'cheap']);
+
+function parsePositiveInteger(value) {
+  const n = Number.parseInt(value, 10);
+  if (!Number.isInteger(n) || n < 1 || String(n) !== String(value).trim()) {
+    throw new InvalidArgumentError('must be a positive integer');
+  }
+  return n;
+}
+
+function parseLimit(value, max) {
+  return Math.min(max, parsePositiveInteger(value));
+}
+
+async function apiKey() {
+  return getApiKey();
+}
+
+program
+  .name('or-info')
+  .description('OpenRouter model info: prices, benchmarks, context and comparisons')
+  .version('0.1.0')
+  .option('--mcp', 'Start MCP server (stdio transport)');
+
+// ── models ─────────────────────────────────────────────────────────────────
+program
+  .command('models')
+  .description('List all models with pricing')
+  .option('--sort <field>', 'Sort by: price, context, name', 'name')
+  .option('--filter <text>', 'Filter by model ID or name (case-insensitive)')
+  .option('--free', 'Show only free models')
+  .option('--limit <n>', 'Maximum number of models to return (max 200)', (v) => parseLimit(v, 200))
+  .option('--tags', 'Show feature tags')
+  .option('--json', 'Output raw JSON')
+  .action(async (opts) => {
+    const key = await apiKey();
+    let models = await fetchModels({ apiKey: key });
+
+    if (opts.filter) {
+      const q = opts.filter.toLowerCase();
+      models = models.filter(
+        (m) => m.id.toLowerCase().includes(q) || (m.name ?? '').toLowerCase().includes(q)
+      );
+    }
+    if (opts.free) {
+      models = models.filter((m) => {
+        const p = pricePerMillion(m);
+        return p.input === 0 && p.output === 0;
+      });
+    }
+
+    if (opts.sort === 'price') {
+      models.sort((a, b) => {
+        const pa = pricePerMillion(a).output ?? Infinity;
+        const pb = pricePerMillion(b).output ?? Infinity;
+        return pa - pb;
+      });
+    } else if (opts.sort === 'context') {
+      models.sort((a, b) => (contextLength(b) ?? 0) - (contextLength(a) ?? 0));
+    } else {
+      models.sort((a, b) => a.id.localeCompare(b.id));
+    }
+
+    if (opts.limit) models = models.slice(0, opts.limit);
+
+    if (opts.json) {
+      console.log(JSON.stringify(models, null, 2));
+      return;
+    }
+    console.log(modelTable(models, { showTags: opts.tags }));
+    console.log(chalk.dim(`  ${models.length} models`));
+  });
+
+// ── price ──────────────────────────────────────────────────────────────────
+program
+  .command('price <model-id>')
+  .description('Detailed pricing and context for a model')
+  .option('--json', 'Output raw JSON')
+  .action(async (modelId, opts) => {
+    const key = await apiKey();
+    const models = await fetchModels({ apiKey: key });
+    const model = findModel(models, modelId);
+    if (!model) die(`Model not found: ${modelId}`);
+
+    if (opts.json) {
+      console.log(JSON.stringify({
+        model: model.id,
+        pricing: model.pricing,
+        context_length: contextLength(model),
+      }, null, 2));
+      return;
+    }
+    const elo = await getElo(modelId);
+    console.log(modelDetail(model, elo));
+  });
+
+// ── benchmark ──────────────────────────────────────────────────────────────
+program
+  .command('benchmark <model-id>')
+  .description('LMArena ELO and pricing details for a model')
+  .option('--json', 'Output raw JSON')
+  .action(async (modelId, opts) => {
+    const [key, elo] = await Promise.all([apiKey(), getElo(modelId)]);
+    if (opts.json) {
+      console.log(JSON.stringify({ model: modelId, elo: elo ?? null }, null, 2));
+      return;
+    }
+    const models = await fetchModels({ apiKey: key });
+    const model = findModel(models, modelId);
+    if (!model) die(`Model not found: ${modelId}`);
+    if (!elo) {
+      console.log(modelDetail(model, null));
+      console.log(chalk.yellow('  No LMArena ELO data for this model.'));
+      console.log(chalk.dim('  Data refreshes every 24h from lmarena.ai via HuggingFace.'));
+      return;
+    }
+    console.log(modelDetail(model, elo));
+  });
+
+// ── compare ────────────────────────────────────────────────────────────────
+program
+  .command('compare <model-a> <model-b>')
+  .description('Side-by-side comparison of two models')
+  .option('--json', 'Output raw JSON')
+  .action(async (idA, idB, opts) => {
+    const key = await apiKey();
+    const [models, eloA, eloB] = await Promise.all([
+      fetchModels({ apiKey: key }),
+      getElo(idA),
+      getElo(idB),
+    ]);
+    const mA = findModel(models, idA);
+    const mB = findModel(models, idB);
+    if (!mA) die(`Model not found: ${idA}`);
+    if (!mB) die(`Model not found: ${idB}`);
+
+    if (opts.json) {
+      console.log(JSON.stringify({ a: { model: mA, elo: eloA }, b: { model: mB, elo: eloB } }, null, 2));
+      return;
+    }
+    console.log(comparison(mA, eloA, mB, eloB));
+  });
+
+// ── top ────────────────────────────────────────────────────────────────────
+program
+  .command('top')
+  .description('Best models for a task')
+  .option('--task <task>', 'Task: coding, reasoning, general, vision, cheap', 'general')
+  .option('--budget <usd>', 'Max price per 1M output tokens (e.g. 1.00)', parseFloat)
+  .option('--limit <n>', 'Number of results', parsePositiveInteger, 5)
+  .option('--json', 'Output raw JSON')
+  .action(async (opts) => {
+    if (!TOP_TASKS.has(opts.task)) {
+      die(`Invalid task: ${opts.task}. Expected one of: ${[...TOP_TASKS].join(', ')}`);
+    }
+
+    const key = await apiKey();
+    const [models, allElo] = await Promise.all([
+      fetchModels({ apiKey: key }),
+      getAllElo(),
+    ]);
+
+    const ranked = rankModels(models, allElo, {
+      task: opts.task,
+      maxPricePerMOutput: opts.budget,
+      limit: opts.limit,
+    });
+
+    if (!ranked.length) {
+      console.log(chalk.yellow('No results. Try a different task or increase --budget.'));
+      return;
+    }
+    if (opts.json) {
+      console.log(JSON.stringify(ranked.map((r) => ({ id: r.model.id, score: r.score, elo: r.eloEntry }))));
+      return;
+    }
+    console.log(topList(ranked));
+  });
+
+// ── refresh ────────────────────────────────────────────────────────────────
+program
+  .command('refresh')
+  .description('Force-refresh the cache (models + LMArena ELO)')
+  .action(async () => {
+    const key = await apiKey();
+    process.stdout.write(chalk.dim('Refreshing models…'));
+    const models = await fetchModels({ apiKey: key, force: true });
+    process.stdout.write(chalk.green(` ✓  ${models.length} models\n`));
+
+    process.stdout.write(chalk.dim('Refreshing LMArena ELO…'));
+    const elo = await loadLeaderboard({ force: true });
+    console.log(chalk.green(` ✓  ${elo.length} entries`));
+  });
+
+// ── status ─────────────────────────────────────────────────────────────────
+program
+  .command('status')
+  .description('Show cache status')
+  .action(async () => {
+    const items = await status();
+    console.log(statusReport(items));
+  });
+
+// ── root --mcp ─────────────────────────────────────────────────────────────
+// Must be checked before parseAsync: Commander exits (shows help, code 1)
+// when no subcommand is given, so the --mcp branch would never be reached.
+if (process.argv.includes('--mcp')) {
+  const { startMcp } = await import('../mcp/server.mjs');
+  await startMcp();
+  process.exit(0);
+}
+
+program.hook('preAction', async () => {});
+
+program.parseAsync(process.argv).catch((err) => {
+  // Avoid leaking API keys or file paths from raw errors
+  const safe = err.message?.replace(/sk-[a-zA-Z0-9-]+/g, '[REDACTED]');
+  die(safe ?? 'Unexpected error');
+});

package/lib/cache.mjs

@@ -0,0 +1,117 @@
+import fs from 'node:fs/promises';
+import { randomUUID } from 'node:crypto';
+import { basename, dirname, join } from 'node:path';
+import { CACHE_DIR, MODELS_CACHE, BENCHMARKS_CACHE } from './paths.mjs';
+
+export const TTL = {
+  MODELS: 30 * 60 * 1000,       // 30 min
+  BENCHMARKS: 24 * 60 * 60 * 1000, // 24 h
+};
+
+const IS_WINDOWS = process.platform === 'win32';
+const PERMISSION_ERRORS = new Set(['EINVAL', 'ENOSYS', 'EPERM']);
+
+async function chmodBestEffort(path, mode) {
+  if (IS_WINDOWS) return;
+  try {
+    await fs.chmod(path, mode);
+  } catch (err) {
+    if (!PERMISSION_ERRORS.has(err?.code)) throw err;
+  }
+}
+
+async function ensureDir(dir) {
+  await fs.mkdir(dir, { recursive: true, mode: 0o700 });
+  await chmodBestEffort(dir, 0o700);
+}
+
+async function readJson(file) {
+  try {
+    const text = await fs.readFile(file, 'utf8');
+    return JSON.parse(text);
+  } catch {
+    return null;
+  }
+}
+
+function tempFileFor(file) {
+  return join(
+    dirname(file),
+    `.${basename(file)}.${process.pid}.${Date.now()}.${randomUUID()}.tmp`
+  );
+}
+
+async function writeJson(file, data) {
+  await ensureDir(dirname(file));
+  const tmp = tempFileFor(file);
+
+  try {
+    await fs.writeFile(tmp, JSON.stringify(data), { mode: 0o600 });
+    await fs.rename(tmp, file);
+    await chmodBestEffort(file, 0o600);
+  } catch (err) {
+    await fs.unlink(tmp).catch(() => {});
+    throw err;
+  }
+}
+
+async function mtimeMs(file) {
+  try {
+    const stat = await fs.stat(file);
+    return stat.mtimeMs;
+  } catch {
+    return 0;
+  }
+}
+
+export async function get(file, ttlMs) {
+  const age = Date.now() - (await mtimeMs(file));
+  if (age > ttlMs) return null;
+  return readJson(file);
+}
+
+export async function set(file, data) {
+  await writeJson(file, data);
+}
+
+export async function clear(file) {
+  try {
+    await fs.unlink(file);
+  } catch (err) {
+    if (err.code !== 'ENOENT') throw err;
+  }
+}
+
+export async function clearAll() {
+  try {
+    const entries = await fs.readdir(CACHE_DIR);
+    await Promise.all(
+      entries
+        .filter((e) => e.endsWith('.json'))
+        .map((e) => fs.unlink(join(CACHE_DIR, e)).catch(() => {}))
+    );
+  } catch (err) {
+    if (err.code !== 'ENOENT') throw err;
+  }
+}
+
+export async function status() {
+  const files = [
+    { key: 'models', file: MODELS_CACHE, ttl: TTL.MODELS },
+    { key: 'benchmarks', file: BENCHMARKS_CACHE, ttl: TTL.BENCHMARKS },
+  ];
+
+  return Promise.all(
+    files.map(async ({ key, file, ttl }) => {
+      const mtime = await mtimeMs(file);
+      const age = mtime ? Date.now() - mtime : null;
+      return {
+        key,
+        exists: mtime > 0,
+        ageMs: age,
+        fresh: age !== null && age < ttl,
+        ttlMs: ttl,
+      };
+    })
+  );
+}