@octocodeai/freellama-mcp-server 0.6.2 → 0.9.0

package/docs/LOCAL_MODELS.md

@@ -0,0 +1,190 @@
+# Local model catalog
+
+Which local model to use for each job on one machine, what was measured, and what is only
+published. The method for choosing a model on other hardware is in
+[Choose local models](MODEL_SELECTION.md). For live answers about your machine, use `models {view:"catalog"}`
+or `freellama catalog`; for the exact calls, see the [cookbooks](cookbooks/README.md).
+
+**Snapshot:** 2026-11-01. Apple M2 Pro, 32 GB unified memory, Ollama 0.40.13, FreeLlama 0.8.0. Other
+work was running throughout, leaving about 4–14 GB available. Each model answered the same questions
+in three rounds, one model at a time (`scripts/local-test/rivals.mjs`). "Best" means best among
+the models tried here, on small question sets, drawn images and generated speech; a difference of one
+or two questions is noise.
+
+Evidence labels used on this page:
+
+- **Measured:** run here through FreeLlama, with known answers.
+- **Published:** the publisher's or Ollama's figures, not reproduced here.
+- **Untested:** no quality result yet.
+
+## How to section local models
+
+Group a model by the job it does, using the capability Ollama reports. FreeLlama shows the type as
+`model_type`.
+
+| Section | Ollama capability | `model_type` | Used for | How to call it |
+|---|---|---|---|---|
+| Text and code | `completion`, plus `tools`, `thinking` | `generative` | Chat, coding, tool-calling agents | `run_task {task:"coding"}` |
+| Vision | `completion` + `vision` | `multimodal` | OCR, screenshots, images | `run_task {task:"vision", files:[path], prompt}` |
+| Audio | `audio` | `multimodal` | Speech to text, translation | `audio {action:"transcribe", file}` |
+| Speech | `speech` | `speech` | Text to speech | `audio {action:"speak", text}` |
+| Embedding | `embedding` | `embedding_only` | Code search, duplicates, image search | `run_task {task:"embedding", files, nearest}` |
+| Decision | `decision` | `decision` | Classify, triage, route, yes/no | `run_task {task:"decision", state, questions}` |
+| Not runnable in Ollama | none | none | Rerankers, tiny classifiers, most dedicated speech models | Needs another runtime; FreeLlama cannot serve them |
+
+## Tested picks for each job
+
+### Use now: installed, and fit alongside normal work
+
+| Job | Model | File | Memory needed | Evidence |
+|---|---|---:|---:|---|
+| Classify, triage, yes/no | `strands-decider:2b-nvfp4` | 1.3 GB | 3.8 GB | Measured: 53/56 in all 3 rounds, 7 ms median. `nimble-v3` scored 51/56 at 1.5 s and needs 11.5 GB |
+| Find code for an issue | `embeddinggemma-2:740m-q8_0` | 1.3 GB | 1.4 GB | Measured: right file first 9/10, top 3 9/10; `nomic-embed-text` 6/10 and 8/10, but indexes in 42 s against 67 s |
+| Find duplicates | `nomic-embed-text:v1.5` | 0.3 GB | 1.4 GB | Measured: 8/12 real issue pairs |
+| Find an image by describing it | `embeddinggemma-2:740m-q8_0` | 1.3 GB | 1.4 GB | Measured: 5/5 screenshots, about 1 s per query |
+| OCR, reading text in images | `hf.co/openbmb/MiniCPM-OCR-1.0-gguf:Q4_K_M` | 1.4 GB | 3.8 GB | Measured: 30/30 in all 3 rounds, 257 ms median; `glm-ocr` also 30/30 at 1,885 ms |
+| Image understanding | `lfm3-vl:2b-q4_K_M` | 1.5 GB | 4.1 GB | Measured: 20/20 drawn images in all 3 rounds, 299 ms; `lfm3-omni` 18/20, `bonsai-2:9b` 17/20 |
+| Speech to text | `lfm3-audio:2b-q4_K_M` | 1.7 GB | 5.5 GB | Measured: 1.7% word error on 12 generated sentences, 276 ms median, French and Spanish translation worked; `lfm3-omni` the same |
+| Text to speech | `phonon:en` | 0.1 GB | not previewed | Measured: 0.8–1.7% word error when transcribed back, 102 ms median; both LFM3 models 0% at 950 ms, but the judge is an LFM3 model |
+| Small coding tasks | `lfm3:1.7b-q4_K_M` | 1.1 GB | 3.3 GB | Measured: 21, 20, 20 of 25, 579 ms median; `bonsai-2:9b` 19, 19, 17 |
+
+### Use when memory is free: installed
+
+| Job | Model | File | Memory needed | Evidence |
+|---|---|---:|---:|---|
+| Coding | `hf.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB:Q4_K_M` | 6.2 GB | 13.5 GB | Measured: 23, 22, 23 of 25, 267 ms median: the best coder tried |
+| Research agent with Octocode | `hf.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB:Q4_K_M` | 6.2 GB | 13.5 GB | Measured: 6/8 local and 5/6 GitHub in all 3 rounds: the best agent tried |
+| Research agent, smaller | `bonsai-2:9b` | 3.2 GB | 10.3 GB | Measured: 4, 5, 5 of 8 local and 3, 5, 5 of 6 GitHub; slow, 82 s median per question |
+
+"Memory needed" is FreeLlama's route preview on 2026-11-01, including its reserve.
+
+### Weak or impractical here
+
+| Model | File | Why |
+|---|---|---|
+| `lfm3-audio:2b-q4_K_M` as a research agent | 1.7 GB | Measured: 3/8 local, 3/6 GitHub in all 3 rounds. Guesses names and repeats calls |
+| `bonsai-2:27b` | 8.3 GB | Installed. FreeLlama asks for 16.3 GB available; this machine never had more than 14.4 GB, so it is not measured here. Published: SWE-bench Verified 60.8%, LiveCodeBench 90.1% |
+| `qwen3.6:27b-q4_K_M` | 17.4 GB | Needs 28.7 GB available, so only on a nearly empty 32 GB machine. Removed from this machine untested |
+
+### Worth adding: not installed, untested here
+
+Pull a tag only with approval for that exact tag and size.
+
+| Job | Tag | File | Why |
+|---|---|---:|---|
+| Small classifier | `tev1:4b-nvfp4` | 3.1 GB | Published: Tev1 4B 73.3% against Jev's 76.0% |
+| Text-only embeddings | `embeddinggemma-2:270m-q8_0-text` | 0.3 GB | Published: about 14% better on code than its predecessor |
+| Small coding fallback | `qwen2.5-coder:7b` | 4.7 GB | Older; estimated about 11 GB needed |
+
+## Models like Jev
+
+Jev is a hosted, closed decision model from TypeSafe AI: it answers typed questions (yes/no, choice,
+score) in one forward pass, with no generated text. Local decision models follow the same
+`/v1/systemone` API, so they can be swapped without code changes.
+
+| Model | JevBench (of 231) | Against Jev directly | File | Fit here |
+|---|---:|---|---:|---|
+| `kev:27b-nvfp4` | 205 | none found | 19 GB | No: 25.56 GiB when loaded |
+| `pplx-decider` (27B) | none found | 85.7% vs 84.5%, Perplexity's own test | large | Almost certainly no |
+| `nimble-v3:9b-nvfp4` | 201 | none found | 6.9 GB | With Chrome and VS Code closed |
+| `nimble-v3:9b-q4_K_M` | 198 | none found | 5.6 GB | With Chrome and VS Code closed |
+| `kev:9b-nvfp4` | 183 | none found | 6.7 GB | 11.16 GiB when loaded |
+| `strands-decider:2b-nvfp4` | 182 | none found | 1.3 GB | Yes; installed |
+| `kev:4b` (BF16) | 175 | none found | 8.4 GB | 12.47 GiB when loaded |
+| `tev1:4b-nvfp4` | none found | Tev1 4B 73.3% vs 76.0%, 13 datasets | 3.1 GB | Probably, when memory is free (estimate) |
+| `kev:0.8b` | 146 | none found | about 1 GB | Yes, but weak |
+
+The original Nimble 9B (not v3) scored 75.7% against Jev 1.13's 76.0% on the same 3,880 human-labeled
+decisions; its publisher ran that test.
+
+**Recommendation:** keep `strands-decider` for everyday decisions, and trial `nimble-v3:9b-q4_K_M`
+for harder ones. Compare them on your own labeled data before switching, since `strands-decider`
+already scores 35/36 on real issues here.
+
+**Limits of this comparison:**
+
+- Kev's figures use JevBench v1.2.16, while Nimble and Strands use a specific commit. Whether those match was not checked.
+- No verified JevBench score for Jev itself was found.
+- Most figures come from the models' own publishers.
+
+Other families:
+
+- `d1`, `d1-omni` and `d3` publish other benchmarks, so they are not ranked here. `d1-omni` matters when decisions need images or audio.
+- `jevos-v4` claims 28–130 ms on a laptop CPU with its own binary and also ships GGUF files. No JevBench score was found, and loading in Ollama was not checked.
+- `openjev` is 27B and licensed CC-BY-NC, which bars commercial use.
+
+## New models on Hugging Face and Ollama
+
+Trending on 2026-10-31 and filtered to what could fit. Quality is the publishers' claim. **Path**:
+Ollama is an official tag, HF GGUF pulls as `hf.co/<repo>:<quant>`, and "not via Ollama" needs another
+runtime.
+
+| Section | Model | Size | Path | Note |
+|---|---|---:|---|---|
+| Text, agents | LFM3-1.7B | about 1 GB | Ollama, HF GGUF | Tools and thinking |
+| Text | Spark-X2.6-4B | 3.5 GB | HF GGUF | Custom architecture; Ollama may not load it |
+| OCR | MiniCPM-OCR-1.0 | 1.4 GB with its vision part | HF GGUF | Official GGUF |
+| OCR | Unlimited-OCR-2 | 2.0 GB | HF GGUF | Custom architecture; Ollama may not load it |
+| OCR | Xiaomi-OCR-0 | 0.87B parameters | HF GGUF | Smallest OCR found |
+| Vision | LFM3-VL-2B | 1.5 GB | Ollama | Image understanding, finding objects |
+| Vision | WeVisOne-VL-4B | 2.7 GB | HF GGUF | Tagged for object detection |
+| Vision | Mage-VL-Cowork | 2.7 GB | HF GGUF | Built for screen and UI work |
+| Audio | LFM3-Omni-2B | 2.0 GB | Ollama, HF GGUF | Speech in and out, plus images |
+| Speech to text | parakeet-tdt-0.6b-v3, granite-speech-5.0-2b, nemotron-3.5-asr-streaming-0.6b | small | Not via Ollama (inference) | Dedicated transcription |
+| Text to speech | Kokoro-82M, Audio8-TTS 0.6b, maya2-native 2.5B, kitten-tts-2 | small | Not via Ollama (inference) | |
+| Embeddings | pplx-embed-v2-context-0.6b | 0.6B parameters | Not via Ollama | |
+| Embeddings | Qwen3-VL-Embedding-2B | 2.1B parameters | HF GGUF | Images and text |
+| Reranker | bge-reranker-v2-m3 | 0.57B parameters | Not via Ollama | FreeLlama has no rerank task |
+| Classifier | vela3-fast, GLiNER2.5-Decide, gliclass-v4-edge | under 0.5B parameters | Not via Ollama | Tiny and fast |
+| Safety | Llama-Prompt-Guard-2-86M | 86M parameters | Not via Ollama | Prompt-injection classifier |
+
+Too large for this machine alongside normal work: Qwen3.8-27B, Nanbeige4.3-33B, flint 40B,
+Clover-35B-A3B, Tiel-Coder-35B-A3B, qwen3-coder:30b (19 GB), qwen3-coder-next (52 GB),
+Nemotron-3.5-Transcribe-30B, Apriel-Voice-27B, drama-3.
+
+## Using Octocode from a local model
+
+[Local tools](octocode/LOCAL.md) and [GitHub tools](octocode/GITHUB.md) are the instructions, each
+within 30 lines. They are also the prompts for `delegate_research {adapter:"octocode"}` and
+`{adapter:"github"}`, which run the installed `octocode` CLI's read-only tools. All 23 examples ran
+through the real CLI.
+
+| Model | Local questions | GitHub questions | Notes |
+|---|---:|---:|---|
+| `lfm3-audio:2b-q4_K_M` | 3, 3, 3 of 8 | 3, 3, 3 of 6 | Guessed names and repeated calls; 3.2 calls per question |
+| `lfm3:1.7b-q4_K_M` | 6, 5, 5 of 8 | 1, 1, 1 of 6 | Good on local lookups, weak on GitHub; 2 calls per question, 20 s median |
+| `bonsai-2:9b` | 4, 5, 5 of 8 | 3, 5, 5 of 6 | 2.2 calls per question; 82 s median per question |
+| Nemotron-3.5-Bolt-E4B | 6, 6, 6 of 8 | 5, 5, 5 of 6 | 2.2 calls per question; 96 s median per question; needs 13.5 GB |
+
+Three rounds each. An answer counts as right only if it contains every expected fact and rests on at
+least one tool call.
+
+## What runs in FreeLlama
+
+- **Managed, with routing, memory checks and receipts:** chat, coding, vision, embeddings, decisions.
+- **Through serve's passthrough, with memory checks:** speech and transcription (`audio`).
+- **Not supported:** a second, non-Ollama runtime; separate images per question in one decision, because Ollama shares images across all questions.
+- **Hugging Face:** GGUF repositories work in discovery and pull through Ollama as `hf.co/<repo>:<quant>`. Multimodal GGUF needs an Ollama-supported architecture.
+
+## Still open
+
+- Measure `bonsai-2:27b` when 16.3 GB is available.
+- Measure audio embeddings through `files`; image embeddings are measured, audio ones are not.
+- Run the candidates under "Worth adding".
+- Test `embeddinggemma-2` on duplicates; only `nomic-embed-text` is measured there.
+
+## Detailed results
+
+Rerun the harnesses in `scripts/local-test/` for current numbers (see
+[Testing](TESTING.md#run-live-model-harnesses)); each writes its report to the git-ignored
+`.octocode/local-test/`.
+
+## Sources
+
+- [Ollama: strands-decider](https://ollama.com/library/strands-decider), [kev](https://ollama.com/library/kev), [nimble-v3](https://ollama.com/library/nimble-v3), [nimble](https://ollama.com/library/nimble), [tev1](https://ollama.com/library/tev1), [pplx-decider](https://ollama.com/library/pplx-decider)
+- [Ollama: d1](https://ollama.com/library/d1), [d1-omni](https://ollama.com/library/d1-omni), [d3](https://ollama.com/library/d3)
+- [Ollama: lfm3](https://ollama.com/library/lfm3), [lfm3-vl](https://ollama.com/library/lfm3-vl), [lfm3-omni](https://ollama.com/library/lfm3-omni), [lfm3-audio](https://ollama.com/library/lfm3-audio), [phonon](https://ollama.com/library/phonon), [bonsai-2](https://ollama.com/library/bonsai-2), [embeddinggemma-2](https://ollama.com/library/embeddinggemma-2)
+- [Ollama: qwen3.6](https://ollama.com/library/qwen3.6), [qwen2.5-coder](https://ollama.com/library/qwen2.5-coder), [qwen3-coder](https://ollama.com/library/qwen3-coder), [qwen3-coder-next](https://ollama.com/library/qwen3-coder-next)
+- [JevBench](https://github.com/fstandhartinger/jevbench), [openjev](https://huggingface.co/openjev/openjev), [autotrust/JEV](https://huggingface.co/autotrust/JEV), [feder-cr/jev](https://huggingface.co/feder-cr/jev)
+- [Nemotron-3.5-Bolt-E4B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB), [MiniCPM-OCR-1.0 GGUF](https://huggingface.co/openbmb/MiniCPM-OCR-1.0-gguf), [Unlimited-OCR-2 GGUF](https://huggingface.co/AtomicChat/Unlimited-OCR-2-GGUF), [Spark-X2.6-4B GGUF](https://huggingface.co/XHToken/Spark-X2.6-4B-GGUF), [WeVisOne-VL-4B GGUF](https://huggingface.co/prithivMLmods/tencent-WeVisOne-VL-4B-GGUF), [Mage-VL-Cowork GGUF](https://huggingface.co/Hansda/Mage-VL-Cowork-GGUF)
+- [Hugging Face trending models](https://huggingface.co/api/models?sort=trendingScore), fetched 2026-10-31

package/docs/MODEL_FORMS_AND_RESULTS.md

@@ -0,0 +1,55 @@
+# Model forms and measured results
+
+Moved from the root README. Dated measurements describe one machine and workload.
+
+## Model forms
+
+Managed forms get routing, admission and a receipt; passthrough forms reach Ollama through `serve`
+with the raw stream cap and memory checks but no routing or receipt.
+
+| Form | Ollama endpoint | Through FreeLlama | Measured here |
+|---|---|---|---|
+| Text, coding, reasoning | `/api/chat` | Managed: `completion`, `coding`, `code_repair`, `long_context`, `think` | `gpt-oss:20b` 4/4 |
+| Tool calling | `/api/chat` with `tools` | Managed: `tools` | `gpt-oss:20b` 1/1 |
+| Structured JSON | `/api/chat` with `format` | Managed: `format` | `gpt-oss:20b` 1/1 |
+| Vision and OCR | `/api/chat` with `images` | Managed: `vision`, `stopOnRepetition` for looping readers | `glm-ocr` 4/4 phrases |
+| Embeddings | `/api/embed` | Managed: `embedding`, batched input, vectors withheld unless asked | `nomic-embed-text:v1.5` retrieval 4/4 |
+| Decisions (classify, route, score) | `/v1/systemone` | Managed: `decision`; the catalog lists the model as `model_type: "decision"` | `strands-decider:2b-nvfp4` 9/9, 3/3, 2/3 |
+| Speech (text to speech) | `/v1/audio/speech` | MCP `audio`, through the passthrough's memory checks; Phonon needs Ollama 0.40.9+ on Apple silicon | `phonon:en` works |
+| Transcription and translation | `/v1/audio/transcriptions`, `/v1/audio/translations` | MCP `audio`, through the passthrough's memory checks; LFM3 audio needs Ollama 0.40.12+ | `lfm3-audio:2b-q4_K_M` transcription 7.5% word error over 5 sentences; translation not measured |
+| Image generation | removed from Ollama in 0.32.6 | not available | — |
+
+Find any of these with `models {view:"library", capabilities:["decision"]}` (or `audio`,
+`embedding`, `vision`...), or a Hugging Face GGUF build with a `query`.
+
+### Local models, measured
+
+One Apple M2 Pro (32 GB), Ollama 0.40.8, FreeLlama 0.6.1, 2026-10-30, while other builds kept the
+one-minute load near 14 on 12 cores. Each check has a known answer; the batteries are small, so read
+the ratings as a first screen for these exact tags, not a benchmark.
+
+| Model | Download | Forms | Result | Speed (warm) | Rating | Use it for |
+|---|---:|---|---|---|---:|---|
+| `gpt-oss:20b` | 13.8 GB | text, tools, thinking, JSON | 7/7: exact output, arithmetic, bat-and-ball, code reading, JSON extraction, tool call, 9/9 ticket routing | 1.4 s per routed ticket; 12 s first call with load | 9/10 | General agent work and tool calls when quality matters |
+| `nomic-embed-text:v1.5` | 274 MB | embedding | Retrieval 4/4 top-1; ticket routing 9/9 from two examples per team | 51 ms per item, batched | 9/10 | Search, deduplication, cheap classification |
+| `strands-decider:2b-nvfp4` | 1.3 GB | decision | Ticket routing 9/9, return-policy yes/no 3/3, damage rubric 2/3 | 0.17 s per question | 8/10 | Fast routing and policy checks with probabilities; rubric scores run low at the top level |
+| `glm-ocr:latest` | 2.2 GB | vision (OCR) | 4/4 phrases read from a dense report strip; 0/7 on text tasks | 11 s per strip at a 400-token cap | 8/10 OCR, 1/10 text | Reading screenshots and scans only |
+
+### What they did for a coding agent
+
+Measured 2026-10-31 on the same M2 Pro (Ollama 0.40.13, FreeLlama 0.8.0) while builds and editors
+kept it swapping; rerun the [live model harnesses](TESTING.md#run-live-model-harnesses) for current tables.
+
+| Agent chore | Model | Result | Cost |
+|---|---|---|---|
+| Find the code a real issue is about (octocode repo, 1,025 Rust files ≈ 1.9 M tokens) | `nomic-embed-text` over one "card" per file (path, module doc, item names) | 7/7 issues had the right area in the top 5 | 27 s to index once, ~50 ms per query; read 5 cards (~4 k tokens) instead of searching |
+| Triage real GitHub issues (bug, feature, security, question) | `strands-decider` | 35/36 | 0.46 s per issue |
+| Route tickets, yes/no policy checks, sentiment | `strands-decider` | 17/17 (severity scores: 1/3, don't use for rubrics) | ~0.1 s |
+| Classify with three examples per class | `nomic-embed-text` centroids | 18/20; both misses were near-ties (margin < 0.02) | 18 ms per item |
+| Find related issues | `nomic-embed-text` | 8/12 found the exact partner; the rest found a related one | ms |
+| Read screenshots, receipts, logs, code, a live status page | `glm-ocr` | 27/27 pages read exactly when the image was cropped to the text | ~2 s per page warm |
+
+What makes the difference: a stable prompt prefix (warm it once) cut prompt evaluation from 544 ms
+to 17–38 ms, 1,251 of 1,266 tokens cached; embeddings on the CPU backend ran as fast as on the GPU
+(200 items: 2.2 s vs 2.5 s) and keep the GPU free for OCR and chat; send OCR a tight crop, not a full
+page with a line of text on it.

package/docs/MODEL_INTROSPECTION.md

@@ -0,0 +1,128 @@
+# Inspecting model precision, placement and activations
+
+Reviewed 1 November 2026 against Ollama's current API documentation and source. The locally installed CLI reports 0.40.13; this session could not reach its server, so no live model or activation was inspected. The commands below are instructions, not recorded observations.
+
+## What “which bit/neuron was enabled” can mean
+
+| Question | Observable through stock Ollama? | Interpretation |
+|---|---|---|
+| Is this model Q4, Q8, FP16 or another format? | Yes: `ollama show`, `/api/show`, and model details in `/api/tags` | Weight-storage quantization; not a neuron activation switch |
+| What type and shape does each weight tensor have? | Some versions/runners return `/api/show` `tensors` | Static names, types and dimensions; no per-prompt activation values |
+| Is a runner using CPU, GPU or mixed placement? | `ollama ps`, `/api/ps`, and engine logs | Residency/placement evidence; not a list of active neurons |
+| How likely were generated tokens? | `logprobs` and `top_logprobs`, when supported | Output-token probabilities; not hidden-layer values |
+| Which hidden units had large values for token T? | No documented standard HTTP activation endpoint | Requires instrumentation of the actual inference engine |
+| Which MoE experts were selected for token T? | No documented standard HTTP routing trace | Requires router/expert instrumentation; static expert counts are insufficient |
+
+Neurons in ordinary transformer computation produce numerical activations. There is no universal enabled/disabled bit for each neuron. For example, SiLU computes `x * sigmoid(x)`, with meaningful negative as well as positive values. A rule such as `abs(activation) > threshold` is an analyst's definition of activity, not proof that the engine skipped every other unit. Mixture-of-experts routing is a different case: a router selects experts per token. [1,2,3,4]
+
+## Inspect precision and placement without generating
+
+Choose an exact tag from the current installed list:
+
+```bash
+ollama list
+MODEL_TAG='EXACT_INSTALLED_TAG'
+ollama show "$MODEL_TAG"
+ollama ps
+```
+
+For a structured, read-only inspection (requires `curl` and `jq`):
+
+```bash
+jq -n --arg model "$MODEL_TAG" '{model:$model, verbose:false}' |
+  curl -fsS --max-time 15 http://127.0.0.1:11434/api/show \
+    -H 'Content-Type: application/json' --data-binary @- |
+  jq '{details, capabilities, model_info,
+       tensors: ((.tensors // []) | map({name,type,shape}))}'
+
+curl -fsS --max-time 5 http://127.0.0.1:11434/api/ps |
+  jq '.models[] | {name,digest,size,size_vram,context_length,expires_at}'
+```
+
+Read `details.quantization_level`, `model_info["general.architecture"]`, and the architecture's block/embedding/feed-forward dimensions. Expert-count fields, where present, describe model structure. The current source defines tensor metadata as `name`, `type` and `shape`; missing tensor metadata is unknown, not an empty network. `verbose:true` can return much larger fields, including tokenizer data, and does not enable activation tracing. [1,2,5]
+
+A label such as Q4_K_M is a quantization recipe, not a guarantee that every tensor uses exactly four bits. Mixed tensor types and block scales contribute storage overhead. Inspect reported tensor types for that artifact; also keep weight precision separate from KV-cache precision and execution arithmetic. [2,6]
+
+On unified-memory hardware, `size_vram` describes GPU allocation within shared physical memory. Do not add it to `size` as though those were independent memory pools. A GPU-resident layer is not the same thing as an active neuron. FreeLlama's configured backend and observed placement are already separated in its receipts; see [CPU/GPU routing](CPU_GPU_ROUTING.md).
+
+## Token probabilities are already accessible
+
+An inference request can ask for:
+
+```json
+{
+  "model": "EXACT_INSTALLED_TAG",
+  "messages": [{"role": "user", "content": "Write one short sentence."}],
+  "stream": false,
+  "logprobs": true,
+  "top_logprobs": 5
+}
+```
+
+This is a generation request and can load a model. It is shown for reference and was not executed. FreeLlama's MCP `run_task` already accepts `logprobs:true` and `topLogprobs:5`; full response data is available through its saved receipt. These values describe the output distribution. Generated thinking text and embedding vectors are also not a trace of every internal neuron. [7]
+
+## Inspect activations in a compatible GGUF model
+
+The closest upstream starting point is llama.cpp's [`llama-eval-callback`](https://github.com/ggml-org/llama.cpp/blob/master/examples/eval-callback/README.md). Its public context parameters include `cb_eval` and `cb_eval_user_data`; the example demonstrates observing graph operations and tensor data during inference. [8,9]
+
+For an already built example and an existing compatible GGUF file:
+
+```bash
+llama-eval-callback \
+  --model /absolute/path/to/existing-model.gguf \
+  --prompt 'Hello' \
+  --seed 42 \
+  -ngl 0
+```
+
+This starts a separate model execution. It does not attach to an existing Ollama HTTP session. The example's displayed values may be abbreviated; a complete analysis needs a callback that captures the selected tensor entries, token positions and layers. Reuse its callback mechanism instead of building a tensor evaluator in FreeLlama.
+
+For a useful trace, record:
+
+- Exact artifact digest, engine revision, quantization, backend, tokenizer and rendered prompt/token IDs.
+- Layer, token position, tensor name and a precisely defined location such as MLP post-activation, attention output or residual stream.
+- A bounded sample or top-k indices by absolute activation, retaining their signed values.
+- For MoE models, the selected expert IDs and corresponding routing weights at each layer/token.
+
+Intermediate tensor names and graph structure depend on architecture and engine version. Fused kernels may not expose the desired intermediate without further instrumentation. Fetching GPU tensors can introduce synchronization and transfers; do not use an instrumented trace's timing as the normal serving benchmark. Keep traces opt-in, with limits on tokens, layers, bytes and retention.
+
+A trace of a separate llama.cpp run is not automatically an exact trace of Ollama. Match model bytes, tokenizer, prompt rendering, options and backend, then compare outputs. Some Ollama models use another runner or representation; a GGUF callback does not inspect those engines. Instrument the engine actually executing the model when exact provenance matters.
+
+## Python alternative for compatible research models
+
+For a model already loaded through Transformers/PyTorch, `output_hidden_states=True` exposes layer outputs. `torch.nn.Module.register_forward_hook` captures a module's output, and a pre-hook can inspect a selected module's input. [10,11]
+
+```python
+import torch
+
+# model and inputs must already exist in this separate research runtime.
+with torch.inference_mode():
+    result = model(**inputs, output_hidden_states=True, use_cache=False)
+values = result.hidden_states[8][0, -1].float()
+indices = values.abs().topk(min(10, values.numel())).indices
+print(list(zip(indices.tolist(), values[indices].tolist())))
+```
+
+This reports large residual-channel values at one layer output for the last input token. It does not enumerate the expanded MLP's individual neurons. To inspect those, hook the architecture-specific MLP activation or the input of its down-projection and explicitly identify that measurement point. MoE expert traces require the router's selection, not merely the residual output.
+
+This example was not run. It assumes a compatible research model already loaded locally; it is not code to pass to the Ollama API. Different weights, quantization, kernels or tokenization can change activations. Activation magnitude also does not prove which unit caused an answer; causal claims need interventions such as ablation or activation patching.
+
+## FreeLlama integration boundary
+
+Keep ordinary routing and resource control on stock Ollama. A future inspection tool can expose quantization, static tensor metadata and residency through existing endpoints. Label activation tracing unsupported unless a specific backend adapter actually provides it.
+
+If deeper tracing is added, use a separate explicitly enabled debug runner or an instrumented Ollama backend, with bounded capture and paged artifacts. Rust should manage jobs, limits, cancellation and trace metadata. It should not infer neuron activity from GPU utilization, memory use, token probabilities or generated reasoning text.
+
+## Primary references
+
+1. [Ollama show-model API](https://docs.ollama.com/api-reference/show-model-details)
+2. [Ollama API types: ShowResponse, ModelDetails and Tensor](https://github.com/ollama/ollama/blob/main/api/types.go)
+3. [PyTorch SiLU](https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.silu.html)
+4. [Transformers expert routing](https://huggingface.co/docs/transformers/main/experts_interface)
+5. [Ollama running-model API](https://docs.ollama.com/api/ps)
+6. [Unsloth quantization recipes in its implementation](https://github.com/unslothai/unsloth/blob/main/unsloth/save.py)
+7. [Ollama chat API: logprobs](https://docs.ollama.com/api/chat)
+8. [llama.cpp evaluation callback example](https://github.com/ggml-org/llama.cpp/blob/master/examples/eval-callback/README.md)
+9. [llama.cpp public context API](https://github.com/ggml-org/llama.cpp/blob/master/include/llama.h)
+10. [Transformers hidden-state outputs](https://huggingface.co/docs/transformers/main_classes/output)
+11. [PyTorch module hooks](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html)

package/docs/MODEL_SELECTION.md

@@ -10,6 +10,8 @@ the sections after it from an Apple M4 Pro with 48 GB unified memory. Re-run
 `npx @octocodeai/freellama doctor`, `models`, `bench-all`, and your quality suite on the target hardware.
 `doctor.machine.memory_bytes` is host RAM; a discrete GPU's usable VRAM must be observed separately.
 
+For the best model per job measured on one machine, with newer candidates, see the [local model catalog](LOCAL_MODELS.md).
+
 For supported features and sourced publisher use-case guidance, inspect
 `models {view:"installed", includeLibrary:true}` through MCP. [Model metadata](#model-metadata) explains digest matching, variant capabilities,
 custom-model fallbacks, and how to read family claims without treating them as quality measurements.
@@ -19,8 +21,12 @@ flowchart TD
     T{"What must the task do?"}
     T -->|"Generate text, code, or inspect images"| L["Start with one qualified large multimodal model"]
     T -->|"Create retrieval vectors"| E["Use a small embedding model"]
+    T -->|"Classify, route, or answer yes/no"| D["Use a small decision model"]
+    T -->|"Transcribe or speak"| A["Use an audio or speech model"]
     L --> Q{"Passes held-out quality checks?"}
     E --> Q
+    D --> Q
+    A --> Q
     Q -->|"No"| N["Reject or evaluate another model"]
     Q -->|"Yes"| M{"Fits with workload concurrency?"}
     M -->|"No"| N
@@ -31,20 +37,25 @@ flowchart TD
 
 Measured on an Apple M2 Pro with 32 GB, Ollama 0.40.8 and FreeLlama 0.6.1 on 2026-10-30, while
 other builds kept the one-minute load near 14 on 12 cores. Every check had a known answer; the
-batteries are small, so treat these as first picks to re-check on your machine, not rankings.
+batteries are small, so treat these as first picks to re-check on your machine, not rankings. The
+calls shown are the current ones. Later and larger measurements, audio and memory needs are in the
+[local model catalog](LOCAL_MODELS.md).
 
 | Form | Model (download) | Measured | Speed (warm) | How an agent calls it |
 |---|---|---|---|---|
 | Text, tools, JSON, reasoning | `gpt-oss:20b` (13.8 GB) | 7/7: exact output, arithmetic, bat-and-ball, code reading, JSON extraction, tool call, 9/9 ticket routing | 1.4 s per routed ticket | `run_task {model:"gpt-oss:20b", prompt, think:"low"}`; add `tools` or `format:"json"` |
-| Embeddings | `nomic-embed-text:v1.5` (274 MB) | Retrieval 4/4 top-1; ticket routing 9/9 from two examples per team | 51 ms per item, batched | `run_task {task:"embedding", input:[...], returnEmbeddings:true}` |
-| Decisions (classify, route, score) | `strands-decider:2b-nvfp4` (1.3 GB) | Ticket routing 9/9, return-policy yes/no 3/3, damage rubric 2/3 | 0.17 s per question | `POST <serve>/v1/systemone {model, state, questions}`: passthrough, no MCP tool yet |
-| OCR | `glm-ocr:latest` (2.2 GB) | 4/4 phrases from a dense report strip; 0/7 on text tasks | 11 s per strip at a 400-token cap | `run_task {task:"vision", model:"glm-ocr", prompt:"Text Recognition:", images:[...], stopOnRepetition:true}` |
-
-`gpt-oss:20b` reasons before answering; the battery ran it with `think:"low"` and a 2,048-token
-output cap, so leave room for the reasoning when you set `num_predict`. `glm-ocr` reads text from images only and keeps generating
-after the visible text ends, so cap its output or set `stopOnRepetition`. The decision model needs
-Apple silicon and answers only through Ollama's decision API; reach it through `serve` so the call
-still passes FreeLlama's memory checks and stream cap (from 0.6.2 a busy CPU no longer refuses it).
+| Embeddings | `nomic-embed-text:v1.5` (274 MB) | Retrieval 4/4 top-1; ticket routing 9/9 from two examples per team | 51 ms per item, batched | `run_task {task:"embedding", files:["src/**/*.rs"] or input:[...], nearest:{query:"..."}}`: ranked labels, not vectors |
+| Decisions (classify, route, score) | `strands-decider:2b-nvfp4` (1.3 GB) | Ticket routing 9/9, return-policy yes/no 3/3, damage rubric 2/3 | 0.17 s per question | `run_task {task:"decision", model, state or files:[path], questions}`; one line per answer |
+| OCR | `glm-ocr:latest` (2.2 GB) | 4/4 phrases from a dense report strip; 0/7 on text tasks | 11 s per strip at a 400-token cap | `run_task {task:"vision", model:"glm-ocr", prompt:"Text Recognition:", files:[path]}` |
+
+`gpt-oss:20b` reasons before answering, and its reasoning counts as output. FreeLlama sends no
+output cap of its own and sizes the context from what the model has needed before, so leave room
+for the reasoning if you set `num_predict` yourself. It needed about 17 GB available, so FreeLlama held
+it when other work left less. `glm-ocr` reads text from images only and keeps generating after the
+visible text ends. `stopOnRepetition` handles that and is on by default without `num_predict`; set it
+explicitly if you also cap the output. The `strands-decider` tags need Apple silicon, and decision
+models answer only through Ollama's decision API, which `run_task {task:"decision"}` reaches with
+FreeLlama's routing and memory checks. A busy CPU does not refuse them; low memory does.
 
 ### Classify text three ways
 
@@ -56,7 +67,8 @@ The same nine support tickets, routed to billing, technical or returns:
 | Decision model | `strands-decider:2b-nvfp4` | 9/9 | 0.5 s (first call includes the load) | You need probabilities, yes/no policy checks or rubric scores |
 | Instruction model with JSON output | `gpt-oss:20b` | 9/9 | 1.4 s | Classes change per request or need explanation |
 
-A decision API request names its questions; each answer comes back under the same name:
+A request names its questions, and each answer comes back under the same name. `run_task` takes the
+same `state` and `questions`; this is the raw form for scripts:
 
 ```bash
 curl -s http://127.0.0.1:11435/v1/systemone -H 'content-type: application/json' -d '{
@@ -67,10 +79,42 @@ curl -s http://127.0.0.1:11435/v1/systemone -H 'content-type: application/json'
 }'
 ```
 
+Ollama answers `noul` (yes/no), `choice` (2–26 options for most models, up to 255 for `kev`) and
+`score` (2–10 levels for most models) questions here, not `box`, which it refuses with 400, up to 64
+per request. Text and schema must fit in 64 KiB (1 MiB for `kev` and `d1`); larger requests get 413,
+and input is never truncated. The request takes no `options`, so `keep_alive` is its only residency
+control. Images go in a top-level `images` array of base64 strings shared by every question, or in
+`run_task {files}`. Only a vision decision model reads them: `strands-decider` refuses them with 400.
+Models with audio input also take WAV clips there, and videos are refused. Ollama drops a
+TypeSafe-style `assets` map without an error, so `serve` moves those images into `images` when every
+image applies to every question and refuses the request otherwise. Find decision models with
+`models {view:"library", capabilities:["decision"]}`.
+
+### Use it as Octocode's classifier
+
+Octocode's `clasify` speaks the same System One format, so FreeLlama can serve it locally. In the
+shell or the Octocode home `.env`:
+
+```bash
+OCTOCODE_CLASSIFICATION_API_HOST=http://127.0.0.1:11435
+OCTOCODE_CLASSIFICATION_MODEL=strands-decider:2b-nvfp4
+OCTOCODE_CLASSIFICATION_API=local   # any nonblank value; the local server ignores it
+```
+
+`clasify` `evaluate` and `sections` work with yes/no, choice and score questions, and image
+questions work with a vision decision model. `operation:"models"` works and lists every installed
+model, not only decision models, so name a decision model in `OCTOCODE_CLASSIFICATION_MODEL`. Box
+questions do not exist in Ollama. Raw requests share one slot by default, so Octocode's
+parallel calls queue behind each other; a busy queue answers 429 with `Retry-After`, which
+Octocode honors. Inline `classification` on search results asks a three-level score question, where
+the 2B model's distributions are flat and few candidates clear the 0.5 selection threshold.
+
 ## Text, code, and vision
 
 `qwen3.8:27b-mlx` was the strongest general model in the local evaluations. It handled coding,
-grounded research, image description, OCR, and summarization. Using one large multimodal model also
+grounded research, image description, OCR, and summarization. These evaluations ran on 48 GB; at
+18 GB it does not fit a 32 GB machine in normal use (FreeLlama asks for 28.7 GB available for the
+similarly sized `qwen3.6:27b-q4_K_M`). Using one large multimodal model also
 avoided contention between separate heavy text and vision runners.
 
 In those evaluations it was strong at grounded lookup, long-context retrieval and strict instruction
@@ -78,9 +122,9 @@ following, needed `think:true` and a generous output budget for multi-step reaso
 third of its unsupervised code-review findings were false or context-limited.
 These examples do not establish that a tag is installed or suitable on your host.
 
-Dedicated OCR models answer best to the prompts their model card documents; send them with base64
-`images` in `run_task`. A model that repeats its answer until the output cap can run with
-`stopOnRepetition:true`. See [Route images](https://github.com/bgauryy/FreeLlama/blob/main/packages/mcp/README.md#route-images)
+Dedicated OCR models answer best to the prompts their model card documents; name the image in
+`run_task {files}` rather than pasting base64 `images`. A model that repeats its answer until its
+output limit stops early with `stopOnRepetition`, on by default without `num_predict`. See [Route images](https://github.com/bgauryy/FreeLlama/blob/main/packages/mcp/README.md#route-images)
 for the early stop and the `output_check` receipt.
 
 `muse-glimmer:30b-mlx` was a credible alternative and answered 29 of 30 questions correctly
@@ -104,6 +148,9 @@ the repository retrieval sample:
 
 Use embeddings for semantic similarity, grouping, deduplication, and classification. For exact code
 keywords, repository search was both faster and more accurate in the measured sample.
+`run_task {nearest}` ranks items inside the server and returns labels, so an agent never reads vectors;
+files are split into line chunks labelled `path:Lstart-Lend`. `embeddinggemma-2` adds image and audio
+items for a multimodal model; see the [local model catalog](LOCAL_MODELS.md) for how it compared.
 
 ## Place helper models on CPU when useful
 
@@ -126,7 +173,7 @@ See the [CLI reference](CLI.md#earn-medium-routing-confidence) for the commands.
 ## Model metadata
 
 The MCP `models` tool reads your Ollama instance and can enrich results with Ollama's public model
-library. Supported features and advertised use cases are metadata: they do not change routing
+library, and its library view also searches Hugging Face GGUF repositories. Supported features and advertised use cases are metadata: they do not change routing
 confidence, and exact-model benchmarks and policy remain the quality evidence. The CLI's `models`
 command reports the core catalog and does not expose these enrichment flags.
 
@@ -136,16 +183,18 @@ command reports the core catalog and does not expose these enrichment flags.
 | `models {view:"installed", includeLibrary:true}` | Inventory enriched with public guidance for verified local tags | Fetches model family and tag pages from ollama.com |
 | `models {view:"detail", model:"gemma3:1b"}` | Ollama details, maximum context, and supported-use explanations | Selected Ollama endpoint only |
 | `models {view:"detail", model:"gemma3:1b", includeLibrary:true}` | Details plus public guidance after a tag and digest check | Selected Ollama endpoint and ollama.com |
-| `models {view:"library", query:"coding"}` | Public family search with descriptions and advertised feature chips | ollama.com and optional local inventory/profile reads |
+| `models {view:"library", query:"coding"}` | Public family search with descriptions and advertised feature chips, plus Hugging Face GGUF repositories | ollama.com, huggingface.co, and optional local inventory/profile reads |
 | `models {view:"library", model:"gemma3"}` | All public tags with size, digest prefix, context, modalities, and cloud status | ollama.com and optional local inventory/profile reads |
+| `models {view:"library", model:"hf.co/<owner>/<repo>"}` | The repository's GGUF tags with size and memory fit | huggingface.co and optional local inventory/profile reads |
 
 
 | Option | Valid views | Default | Meaning |
 |---|---|---|---|
 | `includeLibrary` | `installed`, `detail` | `false` | Requests public metadata; the default makes no website calls |
-| `includeReadme` | Library step 2, or `detail` with `includeLibrary:true` | `false` | Returns README text, capped at 65,536 characters, with its format and truncation flag |
+| `includeReadme` | Library step 2, or `detail` with `includeLibrary:true` | `false` | Returns the README text with its format |
 | `includeVerbose` | `detail` | `false` | Returns Ollama's license and Modelfile fields independently of README enrichment |
-| `limit`, `cursor` | `raw`, library tag listings; `limit` also applies to library search | Tag/raw page size 20; search limit 10 | Pages large lists; the tag cursor rejects a changed tag list |
+| `limit`, `cursor` | `raw`, library tag listings; `limit` also applies to library search | Tag/raw page size 20; search limit 10; maximum 50 | Pages large lists; the tag cursor rejects a changed tag list |
+| `query`, `capabilities`, `order`, `source` | Library search | Popular order, both sources | Filter by capability word (such as `decision`), order by `popular` or `newest`, or limit to `ollama` or `huggingface` |
 | `ollamaEndpoint` | Direct Ollama views and library inventory reads | Configured Ollama endpoint | Chooses the Ollama instance; enriched managed inventory uses each model's assigned backend |
 
 An incompatible option returns an error before any lookup. Installed enrichment omits full READMEs;
@@ -166,6 +215,8 @@ feature with `capability`, `use`, `reason`, and `basis:"supported_feature"`;
 | `vision` | Image input | OCR and visual understanding accuracy |
 | `audio` | Audio workflows advertised by the model | Supported audio task and accuracy |
 | `embedding` | Retrieval and similarity vectors | Retrieval quality on your documents |
+| `decision` | Typed questions through `run_task {task:"decision"}` | Accuracy and calibration on your labeled examples |
+| `speech` | Text to speech through the `audio` tool | Intelligibility and supported voices |
 | `thinking` | A thinking mode | Reasoning accuracy, latency, and token cost |
 | `insert` | Suffix-based text insertion | Completion accuracy in the target code or text |
 | `image` | Image generation | Output quality and supported controls |
@@ -182,9 +233,9 @@ For a verified installed model, `knowledge.library` contains the source page, ta
 | `advertisedFamilyCapabilities` | Family feature chips; these do not extend the local tag's capabilities |
 | `advertisedUseCases` | Recognized coding, tool-calling, image-understanding/OCR, reasoning, embedding, summarization, question-answering, translation, and long-context claims |
 | `advertisedUseCases[].evidence` | A bounded source excerpt, source URL, and fetch timestamp |
-| `usageNotes` | Bounded excerpts about requirements, restrictions, context, memory, languages, or query prefixes |
+| `usageNotes` | Sentences about requirements, restrictions, context, memory, languages, or query prefixes |
 | `readmeAvailable` | Whether a README was extracted |
-| `readme` | Requested text with `format` (`markdown` or `text`) and `truncated` |
+| `readme` | Requested text with `format` (`markdown` or `text`) |
 
 
 Advertised uses have `basis:"advertised"`, `scope:"family"`, and `quality:"unmeasured"`. Extraction
@@ -224,7 +275,8 @@ Each MCP process caches at most 128 families: successes for one hour, failures f
 concurrent requests share one lookup and a restart clears the cache. Installed enrichment runs at
 most four family lookups concurrently. Requests use GET, refuse redirects, cap each page at 2 MiB,
 and time out after the smaller of 10 seconds and `FREELLAMA_MCP_FETCH_TIMEOUT_SECONDS`. They disclose
-the requested public family name to ollama.com. Download size does not establish runtime memory
+the requested public family name to ollama.com, and a library search query or `hf.co` name to
+huggingface.co. Download size does not establish runtime memory
 fit, placement, or quality, and model discovery never authorizes a pull. Sources:
 [model details API](https://docs.ollama.com/api-reference/show-model-details),
 [model-list API](https://docs.ollama.com/api/tags).