llm-checker 3.2.8 → 3.4.0

package/README.md

@@ -22,8 +22,11 @@
 </p>
 
 <p align="center">
+  <a href="#start-here-2-minutes">Start Here</a> &bull;
   <a href="#installation">Installation</a> &bull;
   <a href="#quick-start">Quick Start</a> &bull;
+  <a href="#calibration-quick-start-10-minutes">Calibration Quick Start</a> &bull;
+  <a href="https://github.com/Pavelevich/llm-checker/tree/main/docs">Docs</a> &bull;
   <a href="#claude-code-mcp">Claude MCP</a> &bull;
   <a href="#commands">Commands</a> &bull;
   <a href="#scoring-system">Scoring</a> &bull;
@@ -54,6 +57,17 @@ Choosing the right LLM for your hardware is complex. With thousands of model var
 
 ---
 
+## Documentation
+
+- [Docs Hub](https://github.com/Pavelevich/llm-checker/tree/main/docs)
+- [Usage Guide](https://github.com/Pavelevich/llm-checker/blob/main/docs/guides/usage-guide.md)
+- [Advanced Usage](https://github.com/Pavelevich/llm-checker/blob/main/docs/guides/advanced-usage.md)
+- [Technical Reference](https://github.com/Pavelevich/llm-checker/blob/main/docs/reference/technical-docs.md)
+- [Changelog](https://github.com/Pavelevich/llm-checker/blob/main/docs/reference/changelog.md)
+- [Calibration Fixtures](https://github.com/Pavelevich/llm-checker/tree/main/docs/fixtures/calibration)
+
+---
+
 ## Comparison with Other Tooling (e.g. `llmfit`)
 
 LLM Checker and `llmfit` solve related but different problems:
@@ -89,6 +103,32 @@ npm install sql.js
 
 ---
 
+## Start Here (2 Minutes)
+
+If you are new, use this exact flow:
+
+```bash
+# 1) Install
+npm install -g llm-checker
+
+# 2) Detect your hardware
+llm-checker hw-detect
+
+# 3) Get recommendations by category
+llm-checker recommend --category coding
+
+# 4) Run with auto-selection
+llm-checker ai-run --category coding --prompt "Write a hello world in Python"
+```
+
+If you already calibrated routing:
+
+```bash
+llm-checker ai-run --calibrated --category coding --prompt "Refactor this function"
+```
+
+---
+
 ## Distribution
 
 LLM Checker is published in all primary channels:
@@ -97,23 +137,16 @@ LLM Checker is published in all primary channels:
 - GitHub Releases: [Release history](https://github.com/Pavelevich/llm-checker/releases)
 - GitHub Packages: [`@pavelevich/llm-checker`](https://github.com/users/Pavelevich/packages/npm/package/llm-checker)
 
-### v3.2.8 Highlights
-
-- Fixed multimodal recommendation false positives from noisy metadata.
-- Coding-only models with incidental `input_types: image` flags are no longer treated as vision models.
-- Added regression tests to keep multimodal category picks aligned with true vision-capable models.
-
-### v3.2.7 Highlights
+### v3.3.0 Highlights
 
-- License updated to **NPDL-1.0**: paid redistribution and paid hosted/API delivery now require a separate commercial license.
-
-- Recommendation engine now enforces feasible 30B-class coverage on high-capacity discrete multi-GPU setups (for non-speed objectives).
-- Heterogeneous GPU inventories are preserved in output summaries and downstream recommendation inputs.
-- Added and validated fallback mappings/paths for:
-  - AMD Radeon AI PRO R9700 (PCI ID `7551`)
-  - NVIDIA GTX 1070 Ti (device `1b82`)
-  - Linux RX 7900 XTX detection via non-ROCm fallbacks (`lspci`/`sysfs`)
-- Expanded deterministic and hardware regression coverage for multi-GPU and unified-memory edge cases.
+- Calibrated routing is now first-class in `recommend` and `ai-run`:
+  - `--calibrated [file]` support with default discovery path.
+  - clear precedence: `--policy` > `--calibrated` > deterministic fallback.
+  - routing provenance output (source, route, selected model).
+- New calibration fixtures and end-to-end tests for:
+  - `calibrate --policy-out ...` → `recommend --calibrated ...`
+- Hardened Jetson CUDA detection to avoid false CPU-only fallback.
+- Documentation reorganized under `docs/` with clearer onboarding paths.
 
 ### Optional: Install from GitHub Packages
 
@@ -147,6 +180,51 @@ llm-checker search qwen --use-case coding
 
 ---
 
+## Calibration Quick Start (10 Minutes)
+
+This path produces both calibration artifacts and verifies calibrated routing in one pass.
+
+### 1) Use the sample prompt suite
+
+```bash
+cp ./docs/fixtures/calibration/sample-suite.jsonl ./sample-suite.jsonl
+```
+
+### 2) Generate calibration artifacts (dry-run)
+
+```bash
+mkdir -p ./artifacts
+llm-checker calibrate \
+  --suite ./sample-suite.jsonl \
+  --models qwen2.5-coder:7b llama3.2:3b \
+  --runtime ollama \
+  --objective balanced \
+  --dry-run \
+  --output ./artifacts/calibration-result.json \
+  --policy-out ./artifacts/calibration-policy.yaml
+```
+
+Artifacts created:
+
+- `./artifacts/calibration-result.json` (calibration contract)
+- `./artifacts/calibration-policy.yaml` (routing policy for runtime commands)
+
+### 3) Apply calibrated routing
+
+```bash
+llm-checker recommend --calibrated ./artifacts/calibration-policy.yaml --category coding
+llm-checker ai-run --calibrated ./artifacts/calibration-policy.yaml --category coding --prompt "Refactor this function"
+```
+
+Notes:
+
+- `--policy <file>` has precedence over `--calibrated [file]`.
+- If `--calibrated` has no path, discovery uses `~/.llm-checker/calibration-policy.{yaml,yml,json}`.
+- `--mode full` currently requires `--runtime ollama`.
+- `./docs/fixtures/calibration/sample-generated-policy.yaml` shows the expected policy structure.
+
+---
+
 ## Claude Code MCP
 
 LLM Checker includes a built-in [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) server, allowing **Claude Code** and other MCP-compatible AI assistants to analyze your hardware and manage local models directly.
@@ -229,7 +307,9 @@ Claude will automatically call the right tools and give you actionable results.
 | `hw-detect` | Detect GPU/CPU capabilities, memory, backends |
 | `check` | Full system analysis with compatible models and recommendations |
 | `recommend` | Intelligent recommendations by category (coding, reasoning, multimodal, etc.) |
+| `calibrate` | Generate calibration result + routing policy artifacts from a JSONL prompt suite |
 | `installed` | Rank your installed Ollama models by compatibility |
+| `ollama-plan` | Compute safe Ollama runtime env vars (`NUM_CTX`, `NUM_PARALLEL`, `MAX_LOADED_MODELS`) for selected local models |
 
 ### Advanced Commands (require `sql.js`)
 
@@ -263,6 +343,28 @@ llm-checker check --policy ./policy.yaml --use-case coding --runtime vllm
 llm-checker recommend --policy ./policy.yaml --category coding
 ```
 
+### Calibrated Routing in `recommend` and `ai-run`
+
+`recommend` and `ai-run` now support calibration routing policies generated by `calibrate --policy-out`.
+
+- `--calibrated [file]`:
+  - If `file` is omitted, discovery defaults to `~/.llm-checker/calibration-policy.{yaml,yml,json}`.
+- `--policy <file>` takes precedence over `--calibrated` for routing resolution.
+- Resolution precedence:
+  - `--policy` (explicit)
+  - `--calibrated` (explicit file or default discovery)
+  - deterministic selector fallback
+- CLI output includes routing provenance (`--policy`, `--calibrated`, or default discovery) and the selected route/model.
+
+Examples:
+
+```bash
+llm-checker recommend --calibrated --category coding
+llm-checker recommend --calibrated ./calibration-policy.yaml --category reasoning
+llm-checker ai-run --calibrated --category coding --prompt "Refactor this function"
+llm-checker ai-run --policy ./calibration-policy.yaml --prompt "Summarize this report"
+```
+
 ### Policy Audit Export
 
 Use `audit export` when you need machine-readable compliance evidence for CI/CD gates, governance reviews, or security tooling.
@@ -722,7 +824,7 @@ LLM Checker is licensed under **NPDL-1.0** (No Paid Distribution License).
 - Free use, modification, and redistribution are allowed.
 - Selling the software or offering it as a paid hosted/API service is not allowed without a separate commercial license.
 
-See [LICENSE](LICENSE) for full terms.
+See [LICENSE](https://github.com/Pavelevich/llm-checker/blob/main/LICENSE) for full terms.
 
 ---