metrillm 0.2.0 → 0.2.2

package/README.md

@@ -13,7 +13,8 @@
 > Think Geekbench, but for local LLMs on your actual hardware.
 
 ```bash
-npx metrillm@latest bench
+npm install -g metrillm
+metrillm bench
 ```
 
 <p align="center">
@@ -56,26 +57,19 @@ npx metrillm@latest bench
 > [Ollama](https://ollama.com/) or [LM Studio](https://lmstudio.ai/).
 
 ```bash
-# Run directly (no install)
-npx metrillm@latest bench
-
-# Or install globally
-npm i -g metrillm
+# Install globally
+npm install -g metrillm
 metrillm bench
 
-# Homebrew (no global npm install)
-# One-liner install (without pre-tapping):
-brew install MetriLLM/metrillm/metrillm
+# Alternative package managers
+pnpm add -g metrillm
+bun add -g metrillm
 
-# Or one-time tap for short install command:
-brew tap MetriLLM/metrillm
-# Then:
-brew install metrillm
-metrillm bench
+# Homebrew
+brew install MetriLLM/metrillm/metrillm
 
-# Alternative package managers
-pnpm dlx metrillm@latest bench
-bunx metrillm@latest bench
+# Or run without installing
+npx metrillm@latest bench
 ```
 
 ## Usage
@@ -120,19 +114,47 @@ By default, production builds upload shared results to the official MetriLLM lea
 
 If these variables are set to placeholder values (from templates), MetriLLM falls back to official defaults.
 
+## Windows Users
+
+PowerShell's default execution policy blocks npm global scripts. If you see `PSSecurityException` or `UnauthorizedAccess` when running `metrillm`, run this once:
+
+```powershell
+Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned
+```
+
+Alternatively, use `npx metrillm` which bypasses the issue entirely.
+
 ## Runtime Backends
 
 | Backend | Flag | Default URL | Required env |
 |---|---|---|---|
 | Ollama | `--backend ollama` | `http://127.0.0.1:11434` | `OLLAMA_HOST` (optional) |
-| LM Studio | `--backend lm-studio` | `http://127.0.0.1:1234` | `LM_STUDIO_BASE_URL` (optional), `LM_STUDIO_API_KEY` (optional), `LM_STUDIO_STREAM_STALL_TIMEOUT_MS` (optional) |
+| LM Studio | `--backend lm-studio` | `http://127.0.0.1:1234` | `LM_STUDIO_BASE_URL` (optional), `LM_STUDIO_API_KEY` (optional) |
+
+Shared runtime env:
+- `METRILLM_STREAM_STALL_TIMEOUT_MS` (optional): stream watchdog for all backends, default `30000`, `0` disables it
+
+LM Studio benchmark runs now use the native REST inference endpoint (`/api/v1/chat`) for both streaming and non-streaming generation.
+The previous OpenAI-compatible inference path (`/v1/chat/completions`) has been retired from MetriLLM so tok/s and TTFT can rely on native LM Studio stats when available.
+If a LM Studio response omits native token stats, MetriLLM still computes a score and shows the throughput as `estimated`.
 
 For very large models, tune timeout flags:
 - `--perf-warmup-timeout-ms` (default `300000`)
 - `--perf-prompt-timeout-ms` (default `120000`)
 - `--quality-timeout-ms` (default `120000`)
 - `--coding-timeout-ms` (default `240000`)
-- `--lm-studio-stream-stall-timeout-ms` (default `180000`, `0` disables stall timeout)
+- `--stream-stall-timeout-ms` (default `30000`, `0` disables stall timeout for any backend)
+
+Benchmark Profile v1 (applied to all benchmark prompts):
+- `temperature=0`
+- `top_p=1`
+- `seed=42`
+- `thinking` follows your benchmark mode (`--thinking` / `--no-thinking`)
+- Context window stays runtime default (`context=runtime-default`) and is recorded as such in metadata.
+
+LM Studio non-thinking guard:
+- When benchmark mode requests non-thinking (`--no-thinking` or default), MetriLLM now aborts if the model still emits reasoning traces (for result comparability).
+- To disable it in LM Studio for affected models, put this at the top of the model chat template: `{%- set enable_thinking = false %}` then eject/reload the model.
 
 ## How Scoring Works
 
@@ -258,7 +280,7 @@ The tap formula lives in `Formula/metrillm.rb`.
 ./scripts/update-homebrew-formula.sh
 
 # Or pin a specific version
-./scripts/update-homebrew-formula.sh 0.2.0
+./scripts/update-homebrew-formula.sh 0.2.1
 ```
 
 After updating the formula, commit and push so users can install/update with: