@houtini/lm 2.9.0 → 2.11.0

package/README.md

@@ -12,11 +12,11 @@
 
 > **Quick Navigation**
 >
-> [How it works](#how-it-works) | [Quick start](#quick-start) | [What gets offloaded](#what-gets-offloaded) | [Tools](#tools) | [Model routing](#model-routing) | [Configuration](#configuration) | [Compatible endpoints](#compatible-endpoints)
+> [How it works](#how-it-works) | [Quick start](#quick-start) | [What gets offloaded](#what-gets-offloaded) | [Tools](#tools) | [Performance tracking](#performance-tracking) | [Structured JSON output](#structured-json-output) | [Model routing](#model-routing) | [Self-test (shakedown)](#self-test-shakedown) | [Configuration](#configuration) | [Compatible endpoints](#compatible-endpoints) | [Developer guide](./DEVELOPER.md)
 
 I built this because I kept leaving Claude Code running overnight on big refactors and the token bill was painful. A huge chunk of that spend goes on bounded tasks any decent model handles fine - generating boilerplate, code review, commit messages, format conversion. Stuff that doesn't need Claude's reasoning or tool access.
 
-Houtini LM connects Claude Code to a local LLM on your network - or any OpenAI-compatible API. Claude keeps doing the hard work - architecture, planning, multi-file changes - and offloads the grunt work to whatever cheaper model you've got running. Free. No rate limits. Private.
+Houtini LM connects Claude Code to a local LLM on your network - or any OpenAI-compatible API. Claude keeps doing the hard work - architecture, planning, multi-file changes - and offloads the grunt work to whatever cheaper model you've got running. No Claude quota burn. No rate limits. Private. The trade is wall-clock time: local inference is typically 3-30× slower than frontier models, so delegation wins on bounded, self-contained tasks rather than everything.
 
 I wrote a [full walkthrough of why I built this and how I use it day to day](https://houtini.com/how-to-cut-your-claude-code-bill-with-houtini-lm/).
 
@@ -144,19 +144,34 @@ The tool descriptions are written to nudge Claude into planning delegation at th
 
 ## Performance tracking
 
-Every response includes a footer with real performance data - computed from the SSE stream, not from any proprietary API:
+Every response includes a footer with real performance data — computed from the SSE stream, not from any proprietary API:
 
 ```
-Model: zai-org/glm-4.7-flash | 125->430 tokens | TTFT: 678ms, 48.7 tok/s, 12.5s
-Session: 8,450 tokens offloaded across 14 calls
+---
+Model: nvidia/nemotron-3-nano | 279→303 tokens (12 reasoning / 291 visible) | TTFT: 485ms, 58.0 tok/s, 5.2s
+📊 First measured call on nvidia/nemotron-3-nano: 58.0 tok/s, 485ms to first token — use this to gauge whether to delegate longer tasks.
+💰 Claude quota saved — this session: 4,283 tokens / 7 calls · lifetime: 147,432 tokens / 213 calls
 ```
 
-The `discover` tool shows per-model averages across the session:
+The 📊 line only appears on the first measured call per model per session — it's a real benchmark from a genuine task, not a synthetic warmup. The 💰 line updates every call.
+
+When the active model returns `completion_tokens_details.reasoning_tokens` (DeepSeek R1, LM Studio with "Separate reasoning_content" enabled, OpenAI reasoning models), the token block splits into `reasoning / visible` so you can see when a thinking model is burning its output budget on hidden reasoning.
+
+### Lifetime persistence
+
+Per-model performance and token counts persist across Claude Desktop restarts in `~/.houtini-lm/model-cache.db`. This means:
+
+- From call 1 of a new session, `discover` shows **historical** tok/s and TTFT for the loaded model — not "not yet benchmarked".
+- The 💰 counter shows both session and lifetime totals.
+- The `code_task_files` pre-flight estimator uses measured per-model prefill rate to refuse obviously-too-large inputs with a clear diagnostic, instead of letting them silently hang against the MCP client timeout.
+
+The data is workstation-specific — that's intentional. Routing decisions should reflect your actual hardware, not a synthetic benchmark.
+
+The `discover` tool shows per-model averages across both scopes:
 
 ```
-Performance (this session):
-  nvidia/nemotron-3-nano: 6 calls, avg TTFT 234ms, avg 45.2 tok/s
-  zai-org/glm-4.7-flash: 8 calls, avg TTFT 678ms, avg 48.7 tok/s
+Measured speed (session):  58.0 tok/s · TTFT 485ms (1 call)
+Measured speed (lifetime on this workstation): 46.9 tok/s · TTFT 2641ms (214 calls, last used 2026-04-20)
 ```
 
 In practice, Claude delegates more aggressively the longer a session runs. After about 5,000 offloaded tokens, it starts hunting for more work to push over. Reinforcing loop.
@@ -180,7 +195,7 @@ The workhorse. Send a task, get an answer. The description includes planning tri
 | `message` | yes | - | The task. Be specific about output format. |
 | `system` | no | - | Persona - "Senior TypeScript dev" not "helpful assistant" |
 | `temperature` | no | 0.3 | 0.1 for code, 0.3 for analysis, 0.7 for creative |
-| `max_tokens` | no | 2048 | Lower for quick answers, higher for generation |
+| `max_tokens` | no | *auto* | Defaults to 25% of the loaded model's context window (fallback 16,384). Pass a number to cap it. |
 | `json_schema` | no | - | Force structured JSON output conforming to a schema |
 
 ### `custom_prompt`
@@ -193,7 +208,7 @@ Three-part prompt: system, context, instruction. Keeping them separate prevents
 | `system` | no | - | Persona + constraints, under 30 words |
 | `context` | no | - | Complete data to analyse. Never truncate. |
 | `temperature` | no | 0.3 | 0.1 for review, 0.3 for analysis |
-| `max_tokens` | no | 2048 | Match to expected output length |
+| `max_tokens` | no | *auto* | Defaults to 25% of the loaded model's context window (fallback 16,384). |
 | `json_schema` | no | - | Force structured JSON output |
 
 ### `code_task`
@@ -205,7 +220,20 @@ Built for code analysis. Pre-configured system prompt with temperature and outpu
 | `code` | yes | - | Complete source code. Never truncate. |
 | `task` | yes | - | "Find bugs", "Explain this", "Write tests" |
 | `language` | no | - | "typescript", "python", "rust", etc. |
-| `max_tokens` | no | 2048 | Match to expected output length |
+| `max_tokens` | no | *auto* | Defaults to 25% of the loaded model's context window (fallback 16,384). |
+
+### `code_task_files`
+
+Like `code_task`, but the local LLM reads files directly from disk — source never passes through the MCP client's context window. Use this when reviewing multiple related files, or a single large file that's awkward to paste. Files are read in parallel with `Promise.allSettled`, so one unreadable file doesn't sink the call; failures are surfaced inline with the reason.
+
+Includes a **pre-flight prefill estimator**: if measured per-model data from the SQLite cache shows the input would exceed the MCP client's ~60s request-timeout during prompt processing, the call is refused early with a concrete diagnostic (estimated prefill seconds, tokens, and sample-count) instead of letting it silently hang. First-time callers are never refused — the estimator only fires after ≥2 measured samples.
+
+| Parameter | Required | Default | What it does |
+|-----------|----------|---------|-------------|
+| `paths` | yes | - | Array of absolute file paths. Relative paths are rejected. |
+| `task` | yes | - | "Find bugs across these files", "Audit this module" |
+| `language` | no | - | "typescript", "python", "rust", etc. |
+| `max_tokens` | no | *auto* | Defaults to 25% of the loaded model's context window (fallback 16,384). |
 
 ### `embed`
 
@@ -218,12 +246,43 @@ Generate text embeddings via the OpenAI-compatible `/v1/embeddings` endpoint. Re
 
 ### `discover`
 
-Health check. Returns model name, context window, latency, capability profile, and cumulative session stats including per-model performance averages. Call before delegating if you're not sure the LLM's available.
+Health check and speed readout. Returns model name, context window, capability profile, connection latency (labelled explicitly — this is the `/v1/models` fetch round-trip, *not* inference speed), and the active model's measured tok/s and TTFT averaged over the session. Before any real call has run, measured speed shows as "not yet benchmarked — will be captured on the first real call" rather than inventing a number from a synthetic probe. Call before delegating if you're not sure the LLM's available, or when deciding whether a longer task is worth offloading.
 
 ### `list_models`
 
 Lists everything on the LLM server - loaded and downloaded - with full metadata: architecture, quantisation, context window, capabilities, and HuggingFace enrichment data. Shows capability profiles describing what each model is best at, so Claude can make informed delegation decisions.
 
+### `stats`
+
+Compact markdown dump of your offload stats — session and lifetime totals, per-model performance history, reasoning-token overhead — without the model catalog that `discover` prints. Cheap to call repeatedly to watch the 💰 counter climb.
+
+| Parameter | Required | Default | What it does |
+|-----------|----------|---------|-------------|
+| `model` | no | - | Filter output to a single model ID. Omit for all models ever used on this workstation. |
+
+Example output:
+
+```
+## Houtini LM stats
+**Endpoint**: http://hopper:1234 (LM Studio)
+**First call on this workstation**: 2026-04-14
+
+### Totals
+| Scope    | Calls | Prompt tokens | Completion tokens | Total tokens |
+| Session  |     7 |         3,100 |             1,183 |        4,283 |
+| Lifetime |   213 |             — |                 — |      147,432 |
+
+### Per-model performance
+| Model                    | Scope    | Calls | Avg TTFT | Avg tok/s | Prompt tokens | Last used  |
+| nvidia/nemotron-3-nano   | session  |     7 |    485   |      58.0 | —             | —          |
+| nvidia/nemotron-3-nano   | lifetime |   213 |   2641   |      46.9 |       89,320  | 2026-04-20 |
+
+### Reasoning-token overhead (lifetime)
+124 / 47,183 completion tokens spent on hidden reasoning (0.3%). Low — reasoning is effectively suppressed.
+```
+
+The reasoning-token overhead line is the canary for "is `reasoning_effort` actually being honoured on this model and this backend?" — above ~30% is a signal to investigate.
+
 ## Structured JSON output
 
 Both `chat` and `custom_prompt` accept a `json_schema` parameter that forces the response to conform to a JSON Schema. LM Studio uses grammar-based sampling to guarantee valid output - no hoping the model remembers to close its brackets.
@@ -270,6 +329,35 @@ Qwen, Llama, Nemotron, GLM - they score brilliantly on coding benchmarks now. Th
 
 **One call at a time.** As of v2.8.0, houtini-lm enforces this automatically with a request semaphore. Parallel calls queue up and run one at a time, so each gets the full timeout budget instead of stacking.
 
+## Self-test (shakedown)
+
+The canonical way to verify an install and get an honest read on what the loaded model can do on your hardware:
+
+```bash
+npm run shakedown
+```
+
+This runs [`shakedown.mjs`](./shakedown.mjs) — an end-to-end test that exercises all seven tools (`discover` → `list_models` → `chat` → `custom_prompt` → `code_task` → `code_task_files` → `embed`) and prints a summary table with real TTFT, tok/s, token counts, and reasoning-token split for each call. Takes under a minute on a decent rig.
+
+Sample output tail:
+
+```
+Summary
+
+   7/7 steps passed on LM Studio, model=nvidia/nemotron-3-nano
+
+| Tool              | OK  | TTFT (ms) | tok/s  | Tokens in→out        | Reasoning | Notes
+| chat              | ✅  |      891  |   36.9 | 48→104               |        —  | answered
+| custom_prompt     | ✅  |      872  |   43.9 | 170→333              |        —  | 5 valid items
+| code_task         | ✅  |      857  |   41.6 | 180→189              |        —  | tests generated
+| code_task_files   | ✅  |   11028   |   39.5 | 6891→3000            |        —  | cross-referenced
+| embed             | ✅  |      —    |     —  | —                    |        —  | 768-dim vector
+
+   Tokens offloaded: 10,915 (prompt: 7,289, completion: 3,626, reasoning: 0)
+```
+
+Want a human-readable quality review rather than just latency numbers? Paste [SHAKEDOWN.md](./SHAKEDOWN.md) into a Claude session that has houtini-lm attached — Claude will drive the seven steps and write you a report on output quality as well as performance.
+
 ## Think-block handling
 
 Some models emit `<think>...</think>` reasoning blocks before the actual answer. Houtini-lm handles this in two ways:
@@ -285,9 +373,9 @@ The quality footer flags `think-blocks-stripped` when stripping occurred, so you
 Every response includes structured quality signals in the footer so Claude (or any orchestrator) can make informed trust decisions:
 
 ```
-Model: qwen3-coder-30b-a3b | 413→81 tokens | TTFT: 2355ms, 15.0 tok/s, 5.4s
-Quality: think-blocks-stripped, tokens-estimated
-Session: 494 tokens offloaded across 1 call
+---
+Model: qwen3-coder-30b-a3b | 413→81 tokens | TTFT: 2355ms, 15.0 tok/s, 5.4s | Quality: think-blocks-stripped, tokens-estimated
+💰 Claude quota saved this session: 494 tokens across 1 offloaded call
 ```
 
 Flags include: `TRUNCATED` (partial result), `think-blocks-stripped`, `tokens-estimated` (usage data was missing, estimated from content length), `hit-max-tokens`. When no flags fire, the quality line is omitted — clean output, nothing to report.
@@ -344,7 +432,7 @@ Works with anything that speaks the OpenAI `/v1/chat/completions` API:
 
 ## Streaming and timeouts
 
-All inference uses Server-Sent Events streaming. Tokens arrive incrementally. As of v2.8.0, houtini-lm sends MCP progress notifications on every streamed chunk, which resets the SDK's 60-second client timeout. This means generation can run as long as the model needs — there's no hard ceiling as long as tokens keep flowing.
+All inference uses Server-Sent Events streaming. Tokens arrive incrementally. Since v2.9.0, houtini-lm sends MCP progress notifications on every streamed chunk — including during the thinking phase for reasoning models — which resets the SDK's 60-second client timeout. A 5-minute soft timeout acts as a safety net so a genuinely wedged connection can't hold a tool call open indefinitely; as long as tokens keep flowing, the per-chunk progress keeps the client side alive up to that ceiling.
 
 If the connection stalls (no new tokens for an extended period), you get a partial result instead of a timeout error. The footer shows `TRUNCATED` when this happens, and the quality metadata flags it so Claude knows to treat the output with appropriate caution.
 
@@ -368,8 +456,11 @@ git clone https://github.com/houtini-ai/lm.git
 cd lm
 npm install
 npm run build
+npm run shakedown    # end-to-end self-test + benchmark
 ```
 
+See [DEVELOPER.md](./DEVELOPER.md) for architecture, internals, the reasoning-model pipeline, backend detection, the SQLite performance cache, and instructions for adding new tools or backends.
+
 ## Licence
 
 Apache-2.0