metrillm 0.1.1 → 0.2.1

package/README.md

@@ -13,7 +13,8 @@
 > Think Geekbench, but for local LLMs on your actual hardware.
 
 ```bash
-npx metrillm@latest bench
+npm install -g metrillm
+metrillm bench
 ```
 
 <p align="center">
@@ -27,7 +28,7 @@ npx metrillm@latest bench
 
 - **Performance metrics**: tokens/sec, time to first token, memory usage, load time
 - **Quality evaluation**: reasoning, coding, math, instruction following, structured output, multilingual (14 prompts, 6 categories)
-- **Global score** (0-100): 40% hardware fit + 60% quality
+- **Global score** (0-100): 30% hardware fit + 70% quality
 - **Verdict**: EXCELLENT / GOOD / MARGINAL / NOT RECOMMENDED
 - **One-click share**: `--share` uploads your result and gives you a public URL + leaderboard rank
 
@@ -52,19 +53,23 @@ npx metrillm@latest bench
 
 ## Install
 
-> Requires [Node 20+](https://nodejs.org/) and [Ollama](https://ollama.com/) running.
+> Requires [Node 20+](https://nodejs.org/) and a local runtime:
+> [Ollama](https://ollama.com/) or [LM Studio](https://lmstudio.ai/).
 
 ```bash
-# Run directly (no install)
-npx metrillm@latest bench
-
-# Or install globally
-npm i -g metrillm
+# Install globally
+npm install -g metrillm
 metrillm bench
 
 # Alternative package managers
-pnpm dlx metrillm@latest bench
-bunx metrillm@latest bench
+pnpm add -g metrillm
+bun add -g metrillm
+
+# Homebrew
+brew install MetriLLM/metrillm/metrillm
+
+# Or run without installing
+npx metrillm@latest bench
 ```
 
 ## Usage
@@ -76,6 +81,9 @@ metrillm bench
 # Benchmark a specific model
 metrillm bench --model gemma3:4b
 
+# Benchmark with LM Studio backend
+metrillm bench --backend lm-studio --model qwen3-8b
+
 # Benchmark all installed models
 metrillm bench --all
 
@@ -93,18 +101,56 @@ metrillm bench --export json
 metrillm bench --export csv
 ```
 
+## Upload Configuration (CLI + MCP)
+
+By default, production builds upload shared results to the official MetriLLM leaderboard (`https://metrillm.dev`).
+
+- No CI secret injection is required for standard releases.
+- Local/dev runs use the same default behavior.
+- Self-hosted or staging deployments can override endpoints with:
+  - `METRILLM_SUPABASE_URL`
+  - `METRILLM_SUPABASE_ANON_KEY`
+  - `METRILLM_PUBLIC_RESULT_BASE_URL`
+
+If these variables are set to placeholder values (from templates), MetriLLM falls back to official defaults.
+
+## Runtime Backends
+
+| Backend | Flag | Default URL | Required env |
+|---|---|---|---|
+| Ollama | `--backend ollama` | `http://127.0.0.1:11434` | `OLLAMA_HOST` (optional) |
+| LM Studio | `--backend lm-studio` | `http://127.0.0.1:1234` | `LM_STUDIO_BASE_URL` (optional), `LM_STUDIO_API_KEY` (optional), `LM_STUDIO_STREAM_STALL_TIMEOUT_MS` (optional) |
+
+For very large models, tune timeout flags:
+- `--perf-warmup-timeout-ms` (default `300000`)
+- `--perf-prompt-timeout-ms` (default `120000`)
+- `--quality-timeout-ms` (default `120000`)
+- `--coding-timeout-ms` (default `240000`)
+- `--lm-studio-stream-stall-timeout-ms` (default `180000`, `0` disables stall timeout)
+
+Benchmark Profile v1 (applied to all benchmark prompts):
+- `temperature=0`
+- `top_p=1`
+- `seed=42`
+- `thinking` follows your benchmark mode (`--thinking` / `--no-thinking`)
+- Context window stays runtime default (`context=runtime-default`) and is recorded as such in metadata.
+
+LM Studio non-thinking guard:
+- When benchmark mode requests non-thinking (`--no-thinking` or default), MetriLLM now aborts if the model still emits reasoning traces (for result comparability).
+- To disable it in LM Studio for affected models, put this at the top of the model chat template: `{%- set enable_thinking = false %}` then eject/reload the model.
+
 ## How Scoring Works
 
 **Hardware Fit Score** (0-100) — how well the model runs on your machine:
-- Speed: 40% (tokens/sec relative to your hardware tier)
-- TTFT: 30% (time to first token)
+- Speed: 50% (tokens/sec relative to your hardware tier)
+- TTFT: 20% (time to first token)
 - Memory: 30% (RAM efficiency)
 
 **Quality Score** (0-100) — how well the model answers:
 - Reasoning: 20pts | Coding: 20pts | Instruction Following: 20pts
 - Structured Output: 15pts | Math: 15pts | Multilingual: 10pts
 
-**Global Score** = 40% Hardware Fit + 60% Quality
+**Global Score** = 30% Hardware Fit + 70% Quality
 
 Hardware is auto-detected and scoring adapts to your tier (Entry/Balanced/High-End). A model hitting 10 tok/s on a 8GB machine scores differently than on a 64GB rig.
 
@@ -208,6 +254,26 @@ npm run dev           # run from source
 npm run test:watch    # vitest watch mode
 ```
 
+## Homebrew Formula Maintenance
+
+The tap formula lives in `Formula/metrillm.rb`.
+
+```bash
+# Refresh Formula/metrillm.rb with latest npm tarball + sha256
+./scripts/update-homebrew-formula.sh
+
+# Or pin a specific version
+./scripts/update-homebrew-formula.sh 0.2.1
+```
+
+After updating the formula, commit and push so users can install/update with:
+
+```bash
+brew tap MetriLLM/metrillm
+brew install metrillm
+brew upgrade metrillm
+```
+
 ## Contributing
 
 Contributions are welcome! Please read the [Contributing Guide](CONTRIBUTING.md) before submitting a pull request. All commits must include a DCO sign-off.