@octocodeai/freellama-mcp-server 0.1.0

package/docs/CLI.md

@@ -0,0 +1,219 @@
+# CLI reference
+
+The `freellama` command exposes local diagnostics, deterministic model routing, managed task
+execution, benchmarking, and the Ollama-compatible proxy. The front page covers the shortest path;
+this page is the complete command map.
+
+## Choose a command
+
+```mermaid
+flowchart TD
+    Q{"What do you need?"}
+    Q -->|"Prepare first use"| I["init"]
+    Q -->|"Inspect this machine"| D["doctor or machine"]
+    Q -->|"Inspect or choose a model"| M["models, route, or recommend"]
+    Q -->|"Execute a managed request"| T["session, natural-route, or task"]
+    Q -->|"Run the control plane"| S["serve"]
+    Q -->|"Preserve only Ollama APIs"| P["proxy"]
+    Q -->|"Measure or compare"| B["bench-all, run, or eval"]
+    Q -->|"Create a routing policy"| E["policy-from-eval"]
+    Q -->|"Compare CLI and MCP"| L["tools"]
+```
+
+| Command | Purpose | Needs `freellama serve` |
+|---|---|---|
+| `init` | Inspect prerequisites and print a side-effect-free first-run plan | No |
+| `serve` | Run the control plane and Ollama-compatible proxy | Starts it |
+| `auth-token` | Create a new mode-0600 bearer-token file without printing the secret | No |
+| `models` | List installed models, capabilities, residency, and evidence | Yes |
+| `machine` | Print portable host RAM, CPU, OS, architecture, disk, and the local Ollama endpoint | Yes |
+| `session` | Create an affinity scope for related tasks | Yes |
+| `route` | Choose a model and request profile without executing it | Yes |
+| `recommend` | Return an installed route or a reviewed installation plan | Yes |
+| `natural-route` | Convert natural language to a route intent locally, then route it | Yes |
+| `task` | Route and execute one nonstreaming task | Yes |
+| `proxy` | Run only the Ollama-compatible retry and telemetry sidecar | No |
+| `bench-all` | Measure installed models by capability group | No |
+| `policy-from-eval` | Generate policy from quality-evaluation pass rates | No |
+| `tools` | Print MCP tools and their CLI equivalents | No |
+| `doctor` | Inspect Ollama, hardware, versions, and effective settings | No |
+| `run` | Run a frozen suite against one Ollama build | No |
+| `eval` | Compare the same frozen suite against stock and candidate builds | No |
+
+Run `npx @octocodeai/freellama <command> --help` for every accepted flag and enum value. The executable's help
+is authoritative; this page explains how the commands fit together.
+
+## Start the control plane
+
+```bash
+npx @octocodeai/freellama serve --recommendation-catalog recommendations.example.toml
+```
+
+The default listener is `http://127.0.0.1:11435`, and the default Ollama upstream is
+`http://127.0.0.1:11434`. Use `--listen` and `--upstream` to change them. Nonloopback listeners
+require `--allow-remote` and a token from `--auth-token-file` or
+`FREELLAMA_AUTH_TOKEN_FILE`. Authentication applies to managed and passthrough routes.
+
+`serve` persists bounded adaptive feedback under the platform data directory by default. Override
+the path with `--feedback-file`; use `--ephemeral-feedback` only for disposable runs. See the
+[production runbook](PRODUCTION.md) for token generation, state files, and ingress requirements.
+
+`serve` exposes both managed routes under `/_freellama/v1/*` and byte-preserving Ollama routes under
+`/api/*` and `/v1/*`. Use `proxy` when you need only the latter. See
+[Architecture](ARCHITECTURE.md) for the request flow.
+
+### Assign exact models to CPU
+
+Use `--cpu-upstream` with one or more `--cpu-model` values to send named models to a second Ollama
+process. Other managed models and all raw passthrough traffic remain on the primary process.
+
+```bash
+npx @octocodeai/freellama serve \
+  --upstream http://127.0.0.1:11434 \
+  --cpu-upstream http://127.0.0.1:11436 \
+  --cpu-model nomic-embed-text:latest
+```
+
+FreeLlama rejects a CPU upstream with no assignments, nonloopback upstreams, and endpoints that are
+aliases for the same socket. It also sets `options.num_gpu=0` on CPU-assigned managed requests.
+Read [CPU and GPU model routing](CPU_GPU_ROUTING.md) before deploying this layout.
+
+After startup, require `contracts.placement_observation` and `placement_evidence_gate`, then inspect
+the declared upstreams:
+
+```bash
+curl --silent http://127.0.0.1:11435/_freellama/v1/health |
+  jq '{contracts, backends}'
+```
+
+A missing contract identifies a stale `serve` binary; rebuild and restart it before evaluating
+placement.
+
+### Bound admission
+
+`--max-concurrent-tasks` is the primary/GPU cost budget, not a request count; it defaults to 2.
+`--cpu-max-concurrent-tasks` controls the independent CPU pool and defaults to 1. Embedding costs 1,
+chat costs 2, and vision costs 4 (capped to the selected backend's pool). A saturated GPU pool does
+not consume CPU permits. `--max-queue-wait-seconds` defaults to 120; when no permit becomes
+available, FreeLlama refuses the task with HTTP 503 instead of waiting forever.
+
+These are conservative workload-unit defaults, not detected core or RAM counts. Tune them from
+queue-wait receipts and resident-memory observations on the target host. Ollama still controls
+decoding concurrency with `OLLAMA_NUM_PARALLEL`. It also owns a separate internal queue through
+`OLLAMA_MAX_QUEUE`: managed tasks enter that queue only after FreeLlama admission, while raw proxy
+traffic enters it directly. Start an unmeasured deployment with one loaded model and one parallel
+stream per Ollama process.
+
+`route`, `recommend`, and `task` accept
+`--execution-preference auto|prefer-cpu|prefer-gpu`. This is a fallback-capable hint over models
+already assigned by the operator; it never rewrites raw passthrough and never makes an ineligible
+model eligible. Preview the route and inspect the `execution` receipt to confirm whether the hint
+was satisfied. Add `--min-placement-evidence observed` to refuse cold or physically mismatched
+placement; the default `configured` accepts the operator assignment and observes after execution.
+
+## Inspect and route
+
+Start with the read-only guided receipt, then inspect available models. `init` never pulls a model;
+it stops at exact-tag approval and prints the next prerequisite, serve, managed-task, and MCP steps.
+
+```bash
+npx @octocodeai/freellama init
+npx @octocodeai/freellama doctor
+npx @octocodeai/freellama models
+npx @octocodeai/freellama machine
+```
+
+Preview a deterministic decision without spending generation tokens:
+
+```bash
+npx @octocodeai/freellama route --task coding --objective fastest
+npx @octocodeai/freellama route \
+  --task vision \
+  --objective fastest \
+  --required-capability vision
+```
+
+Task kinds are `completion`, `coding`, `code-repair`, `tools`, `browser`, `vision`, `embedding`, and
+`long-context`. Objectives are `fastest`, `balanced`, and `quality`. `fastest` can use capability
+and local benchmark evidence alone; `balanced` and `quality` require a policy for that task.
+
+`recommend` is side-effect-free. If no installed model qualifies, it can return an installation
+plan from the reviewed recommendation catalog, but it never pulls a model.
+
+`natural-route` asks the configured local intent model to produce structured intent, validates that
+intent, and then invokes the same deterministic router as `route`. It does not let the intent model
+choose the final model directly.
+
+## Execute tasks
+
+The prompt for `task` is positional:
+
+```bash
+npx @octocodeai/freellama task --task completion --objective fastest "Reply with exactly OK."
+npx @octocodeai/freellama task --task coding --min-confidence medium "Explain this patch."
+```
+
+Useful task options include:
+
+- `--model` for an exact installed model.
+- `--session` for affinity across related requests.
+- `--context-tokens` for a minimum context requirement.
+- repeatable `--required-capability` constraints.
+- repeatable `--image` paths for vision tasks.
+- `--input-file` for batched embedding input, one item per line.
+- `--min-confidence low|medium` to refuse insufficiently evidenced routes before generation.
+
+Create a session with `session`, then pass its identifier to related `route` or `task` calls. A
+route preview honors an existing affinity but does not create or change one. Only a successfully
+admitted task execution binds the session to its selected model. Reuse never bypasses capability,
+policy, or memory checks.
+
+The CLI has no keep-alive flag. MCP `run_task` accepts `keepAlive`: `"0"` unloads after the request,
+`"-1"` pins the runner, and omission leaves Ollama's default in place.
+
+## Earn medium routing confidence
+
+Every route starts at low confidence. Medium confidence requires both a task policy and a local
+benchmark record for the selected model.
+
+```mermaid
+flowchart LR
+    Q["Quality evaluation aggregate"] --> P["policy-from-eval"]
+    P --> PF["platform.toml"]
+    T["Local throughput benchmark"] --> BR["benchmark-report.json"]
+    PF --> S["freellama serve"]
+    BR --> S
+    S --> C["Medium confidence for configured, measured routes"]
+```
+
+Generate the policy from correctness data and the benchmark report from local runtime data:
+
+```bash
+npx @octocodeai/freellama policy-from-eval \
+  --aggregate benchmark/local/results/<model>/aggregate.json \
+  --task coding \
+  --min-pass 0.8 \
+  --out platform.toml
+
+npx @octocodeai/freellama bench-all --output benchmark-report.json
+npx @octocodeai/freellama serve --recommendation-catalog recommendations.example.toml
+```
+
+`serve` discovers `platform.toml` and `benchmark-report.json` in its working directory. Explicit
+`--policy-file` and `--benchmark-report` values take precedence.
+
+`policy-from-eval` reads `pass_at_1`, not the throughput produced by `bench-all`. It refuses expired
+aggregates and fewer than three trials unless `--allow-smoke` is explicit, and it skips models that
+are not installed. This prevents speed data from being mislabeled as quality evidence.
+
+## Compare the CLI and MCP surfaces
+
+```bash
+npx @octocodeai/freellama tools
+```
+
+The command prints the maintained parity map. MCP-only operations are `delegate_research` and the
+online `models { view: "library" }` view. CLI-only operations include `serve`, `proxy`, `session`,
+`recommend`, `natural-route`, `bench-all`, `policy-from-eval`, `run`, and `eval`.
+
+For the six MCP tool contracts, read the [MCP package reference](../packages/mcp/README.md).

package/docs/CPU_GPU_ROUTING.md

@@ -0,0 +1,352 @@
+# Run models on CPU and GPU
+
+Use two Ollama processes when a large model should remain on the GPU while smaller helper models run
+on the CPU. FreeLlama assigns exact model tags to the secondary process, pins those managed requests
+with `num_gpu: 0`, and reports the selected backend in catalog and task responses.
+
+This configuration is useful for embeddings, intent interpretation, and other small helper models.
+Benchmark large CPU workloads before adopting them. Apple silicon shares memory bandwidth; systems
+with discrete GPUs have different contention and transfer costs, so neither result transfers
+without measurement.
+
+## Adapt the design to your hardware
+
+The two-process contract is portable; the device-control mechanism and useful workload are
+machine-specific:
+
+| Host | Primary process | CPU process defense in depth | Capacity evidence |
+|---|---|---|---|
+| Apple silicon | Metal auto-detection | Managed `num_gpu: 0`; `OLLAMA_LLM_LIBRARY=cpu` is only a hint | Total unified memory plus `/api/ps` |
+| Linux or Windows with NVIDIA | CUDA auto-detection | `CUDA_VISIBLE_DEVICES=-1` plus managed `num_gpu: 0` | Host RAM, `nvidia-smi`, and `/api/ps` |
+| Linux or Windows with AMD | ROCm or Vulkan auto-detection | `ROCR_VISIBLE_DEVICES=-1` or `GGML_VK_VISIBLE_DEVICES=-1`, plus managed `num_gpu: 0` | Host RAM, vendor tools, and `/api/ps` |
+| CPU-only | CPU | A second process can isolate helpers, but it does not create a second accelerator | Host RAM and measured overlap |
+
+`doctor` reports `machine.memory_bytes` on macOS, Linux, and Windows. It reports
+`machine.unified_memory_bytes` only when unified memory is known; that compatibility field is not a
+VRAM estimate on discrete-GPU hosts. Model-library fit is therefore a host-memory preflight, and
+actual accelerator placement must come from Ollama's resident-runner report.
+
+An agent can choose `auto`, `prefer_cpu`, or `prefer_gpu`, but it cannot invent topology. The
+operator owns the backend endpoints and exact CPU-eligible tags. This split lets the same schema
+work on a laptop, a CPU-only server, or a multi-GPU workstation without silently moving arbitrary
+client requests.
+
+## Understand the boundary
+
+The primary upstream owns the compatibility API and GPU-default managed tasks. The secondary
+upstream owns only exact tags configured with `--cpu-model`:
+
+```mermaid
+flowchart TB
+    CLIENT["Client"] --> F["FreeLlama :11435"]
+    F --> K{"Request path"}
+    K -->|"Raw /api/* or /v1/*"| GP["Primary Ollama :11434"]
+    K -->|"Managed /_freellama/v1/tasks"| R["Route exact model"]
+    R -->|"Default assignment"| GP
+    R -->|"--cpu-model match"| O["Set options.num_gpu = 0"]
+    O --> CP["Secondary Ollama :11436"]
+    GP --> GPU["GPU runner"]
+    CP --> CPU["CPU runner"]
+```
+
+FreeLlama does not inspect or rewrite raw passthrough request bodies. A direct Ollama client that
+needs model-aware CPU placement must use the managed task endpoint instead.
+
+## Start the backends
+
+Start the primary Ollama process with a conservative local-only baseline:
+
+```bash
+OLLAMA_HOST=127.0.0.1:11434 OLLAMA_NO_CLOUD=1 \
+    OLLAMA_MAX_LOADED_MODELS=1 OLLAMA_NUM_PARALLEL=1 ollama serve
+```
+
+Start a second process on another loopback port:
+
+```bash
+OLLAMA_HOST=127.0.0.1:11436 OLLAMA_NO_CLOUD=1 \
+    OLLAMA_MAX_LOADED_MODELS=1 OLLAMA_NUM_PARALLEL=1 \
+    OLLAMA_LLM_LIBRARY=cpu ollama serve
+```
+
+The library override is experimental and varies by Ollama build. On the measured Apple Metal 0.33.2
+server, Ollama accepted `OLLAMA_LLM_LIBRARY=cpu` but still selected Metal until the request included
+`num_gpu: 0`. FreeLlama adds that option to CPU-assigned managed requests.
+
+On other GPU backends, Ollama also documents process-level device controls:
+
+- NVIDIA: `CUDA_VISIBLE_DEVICES=-1`
+- ROCm: `ROCR_VISIBLE_DEVICES=-1`
+- Vulkan: `GGML_VK_VISIBLE_DEVICES=-1`
+
+Use a device control appropriate for the Ollama build in addition to FreeLlama's request option when
+you need defense in depth.
+
+## Assign models
+
+Start FreeLlama with the two upstreams and at least one exact CPU model tag:
+
+```bash
+./target/release/freellama serve \
+    --upstream http://127.0.0.1:11434 \
+    --cpu-upstream http://127.0.0.1:11436 \
+    --cpu-model nomic-embed-text:latest
+```
+
+Repeat `--cpu-model` to assign more tags:
+
+```bash
+./target/release/freellama serve \
+    --upstream http://127.0.0.1:11434 \
+    --cpu-upstream http://127.0.0.1:11436 \
+    --cpu-model nomic-embed-text:latest \
+    --cpu-model qwen2.5:0.5b
+```
+
+Use the exact name returned by Ollama, including `:latest` when present. FreeLlama rejects these
+unsafe or ambiguous configurations:
+
+- A CPU upstream without a CPU model assignment
+- A CPU model assignment without `--cpu-upstream`
+- Empty model names
+- The same socket expressed through aliases such as `localhost:11434` and `127.0.0.1:11434`
+- A non-loopback listener or upstream
+
+Set independent admission budgets when the defaults do not fit the workload:
+
+```bash
+./target/release/freellama serve \
+    --upstream http://127.0.0.1:11434 \
+    --cpu-upstream http://127.0.0.1:11436 \
+    --cpu-model nomic-embed-text:latest \
+    --max-concurrent-tasks 2 \
+    --cpu-max-concurrent-tasks 1
+```
+
+The defaults above admit one ordinary primary-backend chat (cost 2) and one CPU embedding (cost 1)
+at the same time. They are conservative workload units for the common `OLLAMA_NUM_PARALLEL=1`
+layout, not a profile of the development Mac. Increase them only after measuring queue wait,
+resident memory, and Ollama's own parallel setting on the target host.
+
+## Let an agent express placement intent
+
+MCP `run_task` and the managed HTTP route accept `executionPreference`: `auto`, `prefer_cpu`, or
+`prefer_gpu` (HTTP: `execution_preference`). `minPlacementEvidence` accepts `configured` or
+`observed` (HTTP: `min_placement_evidence`). Always preview consequential work:
+
+```json
+{
+  "task": "embedding",
+  "objective": "fastest",
+  "executionPreference": "prefer_cpu",
+  "minPlacementEvidence": "observed",
+  "preview": true
+}
+```
+
+A preview accepts routing constraints only. Add the embedding input when submitting the execution
+request after accepting the route.
+
+The agent does not gain authority to assign models. FreeLlama first applies model installation,
+capability, policy, context, explicit-model, and session-affinity constraints. It only honors the
+preference if an eligible model is already assigned to that backend. Inspect
+`execution.preference_satisfied`, `execution.backend`, `execution.observation`, and
+`execution.reason`. `observed` fails closed for cold models: warm once with `configured`, inspect
+the receipt, then retry with observed evidence.
+
+## Verify placement
+
+Inspect FreeLlama's declared backend contract:
+
+```bash
+curl --silent http://127.0.0.1:11435/_freellama/v1/health | jq '.backends'
+```
+
+The output is similar to the following:
+
+```json
+{
+  "gpu": {
+    "upstream": "http://127.0.0.1:11434"
+  },
+  "cpu": {
+    "upstream": "http://127.0.0.1:11436",
+    "models": ["nomic-embed-text:latest"]
+  }
+}
+```
+
+Run a managed embedding task:
+
+```bash
+curl --silent http://127.0.0.1:11435/_freellama/v1/tasks \
+    --header 'content-type: application/json' \
+    --data '{
+      "task": "embedding",
+      "objective": "fastest",
+      "model": "nomic-embed-text:latest",
+      "input": "placement check",
+      "keep_alive": "-1"
+    }' | jq '{execution, admission, route}'
+```
+
+The `execution` object separates the configured `backend`, requested compatibility `placement`, and
+post-run `observation`. The source is Ollama `/api/ps`; direct inspection remains useful:
+
+```bash
+curl --silent http://127.0.0.1:11434/api/ps | jq '.models[] | {name, size_vram}'
+curl --silent http://127.0.0.1:11436/api/ps | jq '.models[] | {name, size_vram}'
+```
+
+A fully CPU-loaded model reports `size_vram:0`; a fully GPU-loaded model reports positive resident
+VRAM. `observation.status` is `verified`, `mismatch`, or unavailable. Unknown, mixed, and mismatched
+samples are reported but excluded from adaptive feedback.
+
+## Understand concurrency
+
+Each backend has its own weighted admission pool and its own resident/shared and
+transition/exclusive lock. CPU and GPU model transitions therefore do not block each other, and a
+GPU burst cannot spend the CPU helper's permit:
+
+```mermaid
+flowchart TD
+    T["Managed task"] --> B{"Selected backend"}
+    B -->|"GPU"| GA["GPU admission pool<br/>default 2 units"]
+    B -->|"CPU"| CA["CPU admission pool<br/>default 1 unit"]
+    GA --> GL{"Resident?"}
+    CA --> CL{"Resident?"}
+    GL -->|"Yes"| GR["GPU shared permit"]
+    GL -->|"No"| GX["GPU exclusive transition"]
+    CL -->|"Yes"| CR["CPU shared permit"]
+    CL -->|"No"| CX["CPU exclusive transition"]
+    GR --> GO["Primary Ollama"]
+    GX --> GO
+    CR --> CO["Secondary Ollama"]
+    CX --> CO
+```
+
+FreeLlama can overlap requests across the two servers even when each Ollama process uses
+`OLLAMA_NUM_PARALLEL=1`. Raising `OLLAMA_NUM_PARALLEL` affects parallel requests within one server
+and multiplies K/V-cache memory; it is a separate tuning decision. `OLLAMA_MAX_QUEUE` likewise
+bounds each Ollama process after FreeLlama admission. It does not replace either backend's weighted
+FreeLlama budget or queue-wait deadline.
+
+## Understand automatic feedback
+
+With `executionPreference: "auto"`, unpinned `fastest` and `balanced` work can select a backend from
+observed warm task duration. The loop is deliberately bounded:
+
+```mermaid
+flowchart LR
+    X["Successful resident task"] --> O["Record duration by<br/>task + backend"]
+    O --> N{"At least 3 samples<br/>on both backends?"}
+    N -->|"No"| D["Keep deterministic default"]
+    N -->|"Yes"| C{"More than 10% faster<br/>per work unit?"}
+    C -->|"Yes"| P["Prefer measured backend"]
+    C -->|"No"| D
+    P --> R["Return reason + placement receipt"]
+```
+
+Generation compares decode nanoseconds per output token; embeddings compare total nanoseconds per
+input token. Buckets are model-specific and reset when the selected model changes. Only physically
+verified placement contributes; cold transitions and assignment mismatches do not. A difference
+of 10% or less is treated as noise. Feedback never overrides an explicit model, session affinity,
+capability or policy guards, or the `quality` objective. The CLI persists the bounded, versioned,
+prompt-free aggregate snapshot atomically by default; `--ephemeral-feedback` opts into reset-on-
+restart behavior. `GET /_freellama/v1/health` exposes persistence status, sample counts, readiness,
+and per-backend admission capacity so an agent can inspect the loop instead of guessing.
+
+## Interpret the measured Mac result
+
+The local benchmark warmed both models, then ran three matched sequential and parallel trials using
+a 100-line Qwen completion and a 256-item Nomic embedding batch:
+
+| Trial | Sequential | Parallel |
+|---|---:|---:|
+| 1 | 34.586 s | 27.499 s |
+| 2 | 38.988 s | 28.233 s |
+| 3 | 37.997 s | 38.167 s |
+| Median | 37.997 s | 28.233 s |
+
+The median parallel result was 1.346 times faster, a 25.70% wall-time reduction. The third trial
+shows why this feature is workload-dependent: CPU work can slow Metal work through host and unified-
+memory contention.
+
+The placement and compatibility guardrails also passed:
+
+| Guardrail | Observed result |
+|---|---|
+| GPU placement | `qwen3.8:27b-mlx`, `size_vram: 19,175,677,668` |
+| CPU placement | `nomic-embed-text:latest`, `size_vram: 0` |
+| Managed execution | Every recorded request returned HTTP 200 with the correct upstream receipt |
+| Admission | `queue_wait_ms: 0` for every request |
+| GPU output | Identical output length in every sequential and parallel trial |
+| Raw compatibility | `/api/version` through FreeLlama returned primary Ollama 0.33.2 |
+
+An earlier, lighter overlap check paired a small CPU embedding with a cold GPU completion. The CPU
+request completed in 60 ms while the GPU request continued and completed in 7.391 seconds. This is
+the intended sweet spot: a small helper can finish without waiting for unrelated GPU generation.
+
+A fresh post-build smoke check repeated the concurrent path through the persistent services. Both
+requests returned HTTP 200 in 8.303 seconds wall time, both reported zero queue wait, Qwen returned
+`OK` from the primary backend, and Nomic returned one embedding from the CPU backend. That runner
+snapshot reported 18,486,900,456 GPU bytes for Qwen and zero for Nomic. The positive GPU byte count
+is the placement invariant; its absolute value can change with the request profile and runner state.
+
+A production-gate adversarial trial assigned `qwen3.8:27b-mlx` to the CPU endpoint with
+`num_gpu:0`. Ollama still reported all 18,490,183,380 resident bytes in VRAM. FreeLlama returned
+`observation.status:"mismatch"`, `feedback.accepted:false`, kept CPU samples at zero, and refused a
+subsequent `min_placement_evidence:"observed"` preview with HTTP 422. This is why assignment and
+physical processor are distinct fields.
+
+Immediate unload is a managed transaction. For `keep_alive:0`, FreeLlama temporarily retains the
+runner, observes `/api/ps`, records feedback only when placement is verified, requests an explicit
+unload, and verifies that the runner is no longer resident. Inspect both `execution.observation`
+and `execution.lifecycle`; a failed unload is reported without discarding the completed task output.
+
+Prefer the following assignments:
+
+- Small embedding models
+- Lightweight intent models
+- Short extraction or classification helpers with measured CPU behavior
+
+Benchmark these assignments before adopting them:
+
+- Large generative models on CPU
+- Large embedding batches that saturate memory bandwidth
+- Two workloads whose combined resident memory approaches system capacity
+
+The local measurement receipt is written to
+`.octocode/evals/cpu-gpu-concurrency-benchmark.md`. Git intentionally ignores that path, so the
+trial table in "Interpret the measured result" carries the durable result while the local artifact
+retains the full run detail.
+
+## Troubleshoot the secondary backend
+
+If an assigned model does not appear, check the following conditions:
+
+1. Confirm that the exact tag appears in `GET http://127.0.0.1:11436/api/tags`.
+2. Confirm that the secondary server is listening on the configured port.
+3. Run a managed task, then confirm `size_vram: 0` on the secondary `/api/ps` response.
+4. Inspect the secondary Ollama log for a runner command containing `-ngl 0` or equivalent CPU
+   placement evidence.
+5. Confirm that the request used `/_freellama/v1/tasks`, not raw `/api/chat` passthrough.
+
+A configured but unreachable CPU backend causes model discovery to fail closed. GPU-managed tasks
+can also be unavailable until the secondary server returns. Raw passthrough to the primary server
+continues to work because it does not depend on catalog discovery.
+
+For the surrounding runtime settings, see
+[Ollama and FreeLlama optimization](OLLAMA_SYSTEM_OPTIMIZATION.md). For all `serve` flags, see the
+[CLI reference](CLI.md).
+
+## Upstream sources
+
+- [Ollama hardware support](https://docs.ollama.com/gpu) documents GPU visibility controls and
+  Apple Metal support.
+- [Ollama FAQ](https://docs.ollama.com/faq) documents concurrency, residency, `/api/ps` processor
+  reporting, and K/V-cache behavior.
+- [Ollama troubleshooting](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.mdx)
+  documents the experimental `OLLAMA_LLM_LIBRARY` override.
+- [Ollama API types](https://github.com/ollama/ollama/blob/main/api/types.go) defines `num_gpu` as a
+  runner-load option.

package/docs/ECONOMICS.md

@@ -0,0 +1,105 @@
+# Token economics
+
+This reference keeps benchmark detail out of the README's "what, why, and run it" path. The
+per-call rows come from one machine (M4 Pro, 48 GB unified memory); values marked with `~` and the
+session projection are estimates derived from those calls.
+
+## Per-call, measured
+
+| Work | Source tokens | Returned tokens | Preserved |
+|---|---:|---:|---:|
+| 6 grounded code questions | 59,208 | 1,742 | **97.1%** |
+| 4 text embeddings (307 ms) | 17,600 | ~200 | **98.9%** |
+| 1 image, OCR, byte-exact | 1,970 | 37 | **98.1%** |
+| Historical tool-schema surface, per turn | <3.2k | 0 | **100%** |
+
+## Current MCP-envelope audit (2026-09-03)
+
+The earlier table describes a historical workload on an M4 Pro. This audit measures the current
+eight-tool MCP build on the current host. JSON tokenization varies by client and model, so it records
+bytes first and gives `characters / 4` only as a deliberately rough token estimate.
+
+| Current result | Canonical structured chars | Compact text chars | Duplicate chars avoided | Rough duplicate-token estimate |
+|---|---:|---:|---:|---:|
+| `doctor {view:"summary"}` | 3,099 | 35 | 3,064 | ~766 |
+| `doctor {view:"scheduler"}` | 3,933 | 35 | 3,898 | ~975 |
+| `doctor {view:"config"}` | 7,373 | 86 | 7,287 | ~1,822 |
+| `doctor {view:"full"}` | 10,836 | 185 | 10,651 | ~2,663 |
+| `models {view:"raw", limit:1}` | 447 | 43 | 404 | ~101 |
+
+These are **not automatically billed-token savings**. They are the second serialized text copy
+that FreeLlama does not emit. An MCP host that feeds both `structuredContent` and text into a model
+can avoid roughly the listed amount; a host that already keeps only one representation does not.
+The canonical structured payload still exists for deterministic clients that need it.
+
+The current eight-tool schema plus initialization instructions measured 17,935 characters, or
+~4,484 tokens using the same rough conversion. The integration suite enforces a ceiling below 4,500
+estimated tokens. This is tool-schema **rent**, not a saving: clients pay it whenever they expose
+the entire tool list to a model. Read the bundled documentation resource on demand and do not expose
+unneeded tools in a client that supports tool filtering.
+
+Delegated research has a different economics boundary. It keeps source files and full tool
+observations out of the calling agent's active prompt, but it spends local Ollama input/output
+tokens in the delegated agent. The historical research rows below remain useful workload evidence;
+they are not a claim that the current installed model or host can reproduce those savings. Measure
+the same question, source size, model, context policy, and client transcript before using a number
+as a production budget.
+
+## Scaled to a session
+
+20 grounded questions, 200 turns, one embedding index over 322 chunks:
+
+| Component | Tokens preserved |
+|---|---:|
+| Research offload | 191,553 |
+| Embedding vectors withheld | 354,200 |
+| Schema rent avoided, after ~90% prompt-cache discount | 59,740 |
+| **Total** | **≈ 605,000 — about 3 × a 200K window** |
+
+## Measurement and projection limits
+
+Schema rent is nominally **597,400** tokens (200 turns × the former eight-tool schemas). Including
+the nominal figure puts the headline at **1.14M — nearly double**. This report discounts it by ~90%
+because prompt caching makes cached input far cheaper,
+and because the rent only accrues at all if a *separate* small model holds the tool schemas. The
+first two rows are the uncontestable part: that data physically never reaches the orchestrator.
+
+**Do not lead with this number.** Providers, caching strategies, context windows, and pricing all
+move, and a token count moves with them. The durable benefit is **context isolation** — the
+orchestrator never receives whole files, raw vectors, OCR output, intermediate research, or
+repetitive schemas. That improves context *quality* as well as cost, and it does not depend on
+anyone's price list.
+
+## The cost side
+
+| | |
+|---|---|
+| Grounded question latency | **7–62 s** vs ~1 s for a frontier model that already holds the file |
+| Where it goes | **51% prompt re-evaluation**, 42% generation, 7% model load (measured on a 4-turn run) |
+| Scaling | turn-dominated: `seconds ≈ 9.8 × tool_calls`, correlation **0.811** over 60 runs |
+| Break-even | past ~1k tokens of source the token maths wins; below that, read the file directly |
+| Judgment ceiling | ~67% accurate, in the same confident tone as the ~99% case |
+
+**Correction (measured same day): prefix-cache reuse WORKS on this path.** An earlier revision of
+this file claimed the opposite by misreading `prompt_eval_count`, which counts *all* prompt tokens
+whether or not they were served from cache. The durations tell the truth:
+
+| Probe (raw `/api/chat`, 2,462-token prefix) | prompt_eval |
+|---|---:|
+| turn 1, cold | 18,631 ms |
+| turn 2, warm — tiny suffix appended | **281 ms** |
+| a *different* conversation interjected | 22,467 ms (its own cold prefill) |
+| turn 3 of the original, after the interjection | **285 ms — cache survived** |
+
+So the per-turn cost is **prefill of NEW tokens plus generation**, not re-reading old context, and
+the cache even survives interleaved conversations. The `seconds ≈ 9.8 × tool_calls` relationship
+still holds empirically — each turn adds a fresh observation to prefill (~130 tok/s measured) and a
+generation (~40 tok/s) — but the mechanism is new-token cost, not context re-evaluation. Fewer,
+better-aimed commands remain the lever; "the cache is broken" is not the reason.
+
+**One consequence worth acting on: context compaction is cache-hostile.** `fit_to_context` rewrites
+old observations when the window is nearly full, which changes the byte prefix and invalidates the
+cache from that point — the next turn re-prefills everything after the edit. The cheap mitigation is
+headroom: `OLLAMA_KV_CACHE_TYPE=q8_0` roughly halves KV memory (flash attention is already auto-on,
+measured and documented in `doctor`), which buys double the context for the same memory and makes
+compaction — and therefore invalidation — rare.