metrillm 0.1.0 → 0.2.0

package/README.md

@@ -16,11 +16,18 @@
 npx metrillm@latest bench
 ```
 
+<p align="center">
+  <img src="docs/images/cli1.png" width="48%" alt="MetriLLM CLI — interactive menu" />
+  <img src="docs/images/cli2.png" width="48%" alt="MetriLLM CLI — hardware detection" />
+</p>
+
+[![MetriLLM Leaderboard](docs/images/leaderboard.png)](https://metrillm.dev)
+
 ## What You Get
 
 - **Performance metrics**: tokens/sec, time to first token, memory usage, load time
 - **Quality evaluation**: reasoning, coding, math, instruction following, structured output, multilingual (14 prompts, 6 categories)
-- **Global score** (0-100): 40% hardware fit + 60% quality
+- **Global score** (0-100): 30% hardware fit + 70% quality
 - **Verdict**: EXCELLENT / GOOD / MARGINAL / NOT RECOMMENDED
 - **One-click share**: `--share` uploads your result and gives you a public URL + leaderboard rank
 
@@ -45,7 +52,8 @@ npx metrillm@latest bench
 
 ## Install
 
-> Requires [Node 20+](https://nodejs.org/) and [Ollama](https://ollama.com/) running.
+> Requires [Node 20+](https://nodejs.org/) and a local runtime:
+> [Ollama](https://ollama.com/) or [LM Studio](https://lmstudio.ai/).
 
 ```bash
 # Run directly (no install)
@@ -55,6 +63,16 @@ npx metrillm@latest bench
 npm i -g metrillm
 metrillm bench
 
+# Homebrew (no global npm install)
+# One-liner install (without pre-tapping):
+brew install MetriLLM/metrillm/metrillm
+
+# Or one-time tap for short install command:
+brew tap MetriLLM/metrillm
+# Then:
+brew install metrillm
+metrillm bench
+
 # Alternative package managers
 pnpm dlx metrillm@latest bench
 bunx metrillm@latest bench
@@ -69,6 +87,9 @@ metrillm bench
 # Benchmark a specific model
 metrillm bench --model gemma3:4b
 
+# Benchmark with LM Studio backend
+metrillm bench --backend lm-studio --model qwen3-8b
+
 # Benchmark all installed models
 metrillm bench --all
 
@@ -86,38 +107,138 @@ metrillm bench --export json
 metrillm bench --export csv
 ```
 
+## Upload Configuration (CLI + MCP)
+
+By default, production builds upload shared results to the official MetriLLM leaderboard (`https://metrillm.dev`).
+
+- No CI secret injection is required for standard releases.
+- Local/dev runs use the same default behavior.
+- Self-hosted or staging deployments can override endpoints with:
+  - `METRILLM_SUPABASE_URL`
+  - `METRILLM_SUPABASE_ANON_KEY`
+  - `METRILLM_PUBLIC_RESULT_BASE_URL`
+
+If these variables are set to placeholder values (from templates), MetriLLM falls back to official defaults.
+
+## Runtime Backends
+
+| Backend | Flag | Default URL | Required env |
+|---|---|---|---|
+| Ollama | `--backend ollama` | `http://127.0.0.1:11434` | `OLLAMA_HOST` (optional) |
+| LM Studio | `--backend lm-studio` | `http://127.0.0.1:1234` | `LM_STUDIO_BASE_URL` (optional), `LM_STUDIO_API_KEY` (optional), `LM_STUDIO_STREAM_STALL_TIMEOUT_MS` (optional) |
+
+For very large models, tune timeout flags:
+- `--perf-warmup-timeout-ms` (default `300000`)
+- `--perf-prompt-timeout-ms` (default `120000`)
+- `--quality-timeout-ms` (default `120000`)
+- `--coding-timeout-ms` (default `240000`)
+- `--lm-studio-stream-stall-timeout-ms` (default `180000`, `0` disables stall timeout)
+
 ## How Scoring Works
 
 **Hardware Fit Score** (0-100) — how well the model runs on your machine:
-- Speed: 40% (tokens/sec relative to your hardware tier)
-- TTFT: 30% (time to first token)
+- Speed: 50% (tokens/sec relative to your hardware tier)
+- TTFT: 20% (time to first token)
 - Memory: 30% (RAM efficiency)
 
 **Quality Score** (0-100) — how well the model answers:
 - Reasoning: 20pts | Coding: 20pts | Instruction Following: 20pts
 - Structured Output: 15pts | Math: 15pts | Multilingual: 10pts
 
-**Global Score** = 40% Hardware Fit + 60% Quality
+**Global Score** = 30% Hardware Fit + 70% Quality
 
 Hardware is auto-detected and scoring adapts to your tier (Entry/Balanced/High-End). A model hitting 10 tok/s on a 8GB machine scores differently than on a 64GB rig.
 
 [Full methodology &rarr;](https://metrillm.dev/methodology)
 
-## Submit Your Result
+## Share Your Results
 
-Every benchmark you share enriches the public leaderboard. No account needed.
+Every benchmark you share enriches the [public leaderboard](https://metrillm.dev). No account needed — pick the method that fits your workflow:
+
+| Method | Command / Action | Best for |
+|--------|-----------------|----------|
+| **CLI** | `metrillm bench --share` | Terminal users |
+| **MCP** | Call `share_result` tool | AI coding assistants |
+| **Plugin** | `/benchmark` skill with share option | Claude Code / Cursor |
+
+All methods produce the same result:
+- A **public URL** for your benchmark
+- Your **rank**: "Top X% globally, Top Y% on [your CPU]"
+- A **share card** for social media
+- A **challenge link** to send to friends
+
+[Compare your results on the leaderboard &rarr;](https://metrillm.dev)
+
+## MCP Server
+
+Use MetriLLM from Claude Code, Cursor, Windsurf, or any MCP client — no CLI needed.
 
 ```bash
-metrillm bench --share
+# Claude Code
+claude mcp add metrillm -- npx metrillm-mcp@latest
+
+# Claude Desktop / Cursor / Windsurf — add to MCP config:
+# { "command": "npx", "args": ["metrillm-mcp@latest"] }
 ```
 
-You'll get:
-- A public URL for your result
-- Your rank: "Top X% globally, Top Y% on [your CPU]"
-- A share card for social media
-- A challenge link to send to friends
+| Tool | Description |
+|------|-------------|
+| `list_models` | List locally available LLM models |
+| `run_benchmark` | Run full benchmark (performance + quality) on a model |
+| `get_results` | Retrieve previous benchmark results |
+| `share_result` | Upload a result to the public leaderboard |
 
-[Compare your results on the leaderboard &rarr;](https://metrillm.dev)
+[Full MCP documentation &rarr;](mcp/README.md)
+
+## Skills
+
+Slash commands that work inside AI coding assistants — no server needed, just a Markdown file.
+
+| Skill | Trigger | Description |
+|-------|---------|-------------|
+| `/benchmark` | User-invoked | Run a full benchmark interactively |
+| `metrillm-guide` | Auto-invoked | Contextual guidance on model selection and results |
+
+Skills are included in the [plugins](#plugins) below, or can be installed standalone:
+
+```bash
+# Claude Code
+cp -r plugins/claude-code/skills/* ~/.claude/skills/
+
+# Cursor
+cp -r plugins/cursor/skills/* ~/.cursor/skills/
+```
+
+## Plugins
+
+Pre-built bundles (MCP + skills + agents) for deeper IDE integration.
+
+| Component | Description |
+|-----------|-------------|
+| MCP config | Auto-connects to `metrillm-mcp` server |
+| Skills | `/benchmark` + `metrillm-guide` |
+| Agent | `benchmark-advisor` — analyzes your hardware and recommends models |
+
+**Install:**
+```bash
+# Claude Code
+cp -r plugins/claude-code/.claude/* ~/.claude/
+
+# Cursor
+cp -r plugins/cursor/.cursor/* ~/.cursor/
+```
+
+See [Claude Code plugin](plugins/claude-code/README.md) and [Cursor plugin](plugins/cursor/README.md) for details.
+
+## Integrations
+
+| Integration | Package | Status | Docs |
+|-------------|---------|--------|------|
+| CLI | [`metrillm`](https://www.npmjs.com/package/metrillm) | Stable | [Usage](#usage) |
+| MCP Server | [`metrillm-mcp`](https://www.npmjs.com/package/metrillm-mcp) | Stable | [MCP docs](mcp/README.md) |
+| Skills | — | Stable | [Skills](#skills) |
+| Claude Code plugin | — | Stable | [Plugin docs](plugins/claude-code/README.md) |
+| Cursor plugin | — | Stable | [Plugin docs](plugins/cursor/README.md) |
 
 ## Development
 
@@ -128,6 +249,26 @@ npm run dev           # run from source
 npm run test:watch    # vitest watch mode
 ```
 
+## Homebrew Formula Maintenance
+
+The tap formula lives in `Formula/metrillm.rb`.
+
+```bash
+# Refresh Formula/metrillm.rb with latest npm tarball + sha256
+./scripts/update-homebrew-formula.sh
+
+# Or pin a specific version
+./scripts/update-homebrew-formula.sh 0.2.0
+```
+
+After updating the formula, commit and push so users can install/update with:
+
+```bash
+brew tap MetriLLM/metrillm
+brew install metrillm
+brew upgrade metrillm
+```
+
 ## Contributing
 
 Contributions are welcome! Please read the [Contributing Guide](CONTRIBUTING.md) before submitting a pull request. All commits must include a DCO sign-off.