@octocodeai/freellama-mcp-server 0.1.0 → 0.6.0

package/docs/OLLAMA_SYSTEM_OPTIMIZATION.md

@@ -1,254 +0,0 @@
-# Ollama and FreeLlama optimization boundary
-
-Ollama already owns inference optimization. FreeLlama can improve completed
-work per hour by choosing an eligible model and avoiding unnecessary model
-changes, but it does not make an individual model decode faster.
-
-This page explains the active request flow, the settings Ollama owns, the
-current machine audit, and the optimization work that remains.
-
-The architecture is portable across Ollama's Metal, CUDA, ROCm, Vulkan, and CPU paths. The audit
-later on this page is intentionally Mac-specific. FreeLlama discovers host RAM per operating system
-and keeps device selection in each Ollama process; it does not translate one machine's memory tier,
-core count, or benchmark into another machine's defaults.
-
-## Decision
-
-Use stock Ollama as the inference runtime. Keep FreeLlama as an optional Rust
-control plane for evidence-qualified routing, bounded request profiles, session
-affinity, and policy enforcement.
-
-Do not advertise FreeLlama as an accelerator or an Ollama inference plugin. It does not tune
-kernels, split one model's layers across CPU and GPU, manage macOS memory pressure, or improve that
-model's tokens per second. It can assign different models to isolated CPU and GPU-capable Ollama
-processes, allowing independent workloads to overlap.
-
-## Request flow
-
-FreeLlama has four distinct paths:
-
-| Request | FreeLlama work | Ollama work | Result |
-|---|---|---|---|
-| Native `/api/*` or `/v1/*` | Streams bytes to the primary compatibility upstream | Performs normal inference and scheduling | Exact Ollama response |
-| `POST /_freellama/v1/routes` | Discovers models, filters capabilities, and applies policy | Supplies installed-model and residency data | Model and request profile; no inference |
-| `POST /_freellama/v1/natural-routes` | Normalizes a schema-bound intent, then runs the deterministic router | Runs the small intent model | Model and request profile; selected task does not run |
-| `POST /_freellama/v1/tasks` | Selects a route and backend, applies managed-task admission, and constructs one bounded request | Runs one nonstreaming chat or embedding request on the assigned process | Route receipt, placement, admission mode, prompt-free metrics, and Ollama response |
-
-The natural-language path is intentionally two-stage. The small model cannot
-name a final model. It returns only a task, objective, context hint, and tool or
-vision requirements. Deterministic guards correct explicit requirements before
-the ordinary router selects a model.
-
-Calling `natural-routes` does not run the selected task. A client must use the
-returned profile or submit a structured `tasks` request. There is no atomic
-natural-language route-and-run endpoint in the current release.
-
-Chat and embedding tasks forward route options. A captured-upstream contract
-test verifies embedding option parity.
-
-## What Ollama already optimizes
-
-Across supported backends, Ollama provides the following behavior without FreeLlama:
-
-- The runtime selects supported acceleration such as Metal, CUDA, ROCm, or Vulkan from the host and
-  its process-level device visibility.
-- The scheduler fits model, context, and compute allocations to available
-  device memory.
-- The runner selects CPU threads and GPU execution; applications do not need to
-  divide one inference manually between the CPU and GPU.
-- Ollama manages request queues plus model loading and eviction. It supports
-  same-model parallel requests when memory permits.
-- Ollama reports token counts plus load, prompt-processing, and generation
-  durations in API responses.
-- Ollama supports request-specific context length and model residency through
-  `num_ctx` and `keep_alive`.
-
-Ollama's documented concurrency controls are global server settings. Increasing
-`OLLAMA_NUM_PARALLEL` multiplies context memory by the parallel-request count.
-Increasing `OLLAMA_MAX_LOADED_MODELS` does not create memory; models still need
-to fit. Use `1` for both values on an unmeasured deployment, then increase one variable at a time
-only when a representative concurrent benchmark improves completed tasks per hour without memory
-pressure.
-
-Keep three signals distinct when operating FreeLlama: `context_window_fit` only compares requested
-context with advertised model context; FreeLlama's K/V preflight is a conservative metadata lower
-bound; and a completed runner is the only placement observation. None is live free-memory or
-throughput proof. `doctor {view:"scheduler"}` reports configuration and a point-in-time admission
-snapshot, not a fairness or parallel-decoding benchmark.
-
-FreeLlama's admission queue and Ollama's internal queue solve different problems. FreeLlama limits
-weighted managed work per backend and applies `--max-queue-wait-seconds`; `OLLAMA_MAX_QUEUE` bounds
-requests that an Ollama process has accepted but cannot schedule. Raw compatibility traffic skips
-FreeLlama admission and reaches the primary Ollama queue directly.
-
-Ollama request context and research-adapter context are also separate budgets. Managed tasks set
-request-specific `num_ctx` from the selected task profile. The coding adapters fit their tool
-conversation inside `FREELLAMA_AGENT_NUM_CTX`: by default, they preserve the system prompt and
-original question, compact the oldest observations first, retain recent evidence, and fail before
-the Ollama call if pinned content cannot fit. Raising the Ollama server's global context does not
-disable adapter compaction or increase its configured budget. See [Agent adapters](../AGENTS.md)
-for the complete pagination and compaction contract.
-
-## Current Mac audit
-
-This audit captures the following snapshot from September 1, 2026. Treat it as machine
-evidence, not a portable default.
-
-| Item | Observed value | Interpretation |
-|---|---|---|
-| Hardware | Apple M4 Pro, 14 CPU cores, 48 GB unified memory | CPU and GPU share the memory system |
-| macOS | 15.7.2 | Native Ollama supports this Apple-silicon system |
-| Ollama server | 0.33.2 | Active application server on `127.0.0.1:11434`; re-read with `doctor` rather than trusting this row |
-| First CLI on `PATH` | `/usr/local/bin/ollama`, 0.33.2 | Matches the active server |
-| Metal device budget | 36 GiB reported by Ollama | This is the scheduler's observed device budget, not all 48 GB |
-| Default context | 32,768 tokens | Ollama selected its 24–48 GiB device-memory tier |
-| Parallel requests | 1 | Appropriate baseline for interactive and large-model tests |
-| Keep-alive | 5 minutes | Ollama default |
-| Loaded-model limit | `OLLAMA_MAX_LOADED_MODELS=2` | Explicit service configuration for the measured topology |
-| K/V-cache type | `OLLAMA_KV_CACHE_TYPE=q8_0` | Explicit memory-saving configuration; requalify quality per important model |
-| Flash Attention | `auto` | Ollama enables it automatically on supported backends; force `1` only after a measured compatibility check. Required for quantized K/V cache. |
-| GGUF CPU threads | 10 worker threads | Ollama selected the 10 performance cores rather than all 14 cores |
-| MLX | GPU runner observed | MLX models use their own runner path, not the GGUF tuning path |
-| Power mode | Automatic on battery and AC power | High Power Mode remains an unmeasured experiment |
-
-The server log also showed Metal fusion, concurrent Metal execution, graph
-optimization, and automatic parameter fitting. FreeLlama must not claim these
-as its own optimizations.
-
-### Verify CLI and server alignment
-
-The selected CLI is `/usr/local/bin/ollama`, and both the CLI and active server report 0.33.2.
-Verify this alignment after changing Ollama installations or service definitions:
-
-```bash
-command -v ollama
-ollama --version
-curl --silent http://127.0.0.1:11434/api/version
-```
-
-The client and server versions must match before you test a CLI feature. A mismatch does not prove
-an inference slowdown, but it can produce incompatible commands, flags, or diagnostics.
-
-## What FreeLlama adds today
-
-The active Rust implementation adds:
-
-- capability filtering before speed ranking;
-- explicit `fastest`, `balanced`, and `quality` evidence contracts;
-- fail-closed quality routing when the policy has no qualified model;
-- task-specific context, output, thinking, and tool-validation profiles;
-- session affinity and preference for an eligible resident model;
-- separate transition locks for primary and CPU backends, with shared execution permits for
-  resident tasks and an exclusive permit for a cold transition;
-- exact-model placement on an optional second CPU Ollama process, including `num_gpu=0` for its
-  managed requests;
-- guarded agent placement preferences plus a normalized, three-warm-sample, 10%-advantage feedback
-  loop for `fastest` and `balanced` work;
-- independent 3:2:1 priority-fair weighted GPU and CPU admission pools, so a GPU burst cannot
-  starve a CPU helper or indefinitely starve background managed work;
-- prompt-free load, prompt-processing, and output-generation metrics derived
-  from Ollama's response fields;
-- a local natural-language intent interpreter that cannot choose a model;
-- an Ollama-compatible loopback endpoint and an escape hatch to port 11434.
-
-These features can reduce reloads or avoid an unsuitable model. They do not
-change the selected model's prompt-processing or decode rate.
-
-The measured dual-backend workload improved median completion time from 37.997 to 28.233 seconds
-(1.346 times, or 25.70%). Qwen reported 19,175,677,668 GPU-resident bytes, Nomic reported zero,
-and all requests carried correct backend receipts with zero FreeLlama queue wait. One of three
-parallel trials was slower than sequential, so this remains a workload-level optimization rather
-than an inference-speed claim.
-
-## Unsupported optimizations
-
-FreeLlama does not provide:
-
-- a live minimum-free-VRAM guard. FreeLlama does calculate a metadata-backed F16 K/V lower bound
-  and refuses a known impossible CPU/unified-memory floor, but Ollama remains the authority for
-  current free memory, runner graph allocation, and final loading;
-- transition coordination for native passthrough requests or processes that
-  bypass the managed `tasks` endpoint;
-- live thermal, power-mode, or swap-aware routing;
-- online token-rate, error-rate, or cold-load-duration scoring;
-- per-engine tuning for GGUF compared with MLX;
-- dynamic `keep_alive` or concurrency limits by workload;
-- an atomic natural-language route-and-run operation;
-- held-out quality policies for completion, coding, tools, browser, vision, or
-  long-context tasks in the example policy;
-- quality-aware or thermal-aware CPU/GPU load balancing. Automatic placement uses persisted,
-  prompt-free warm-latency aggregates only within exact operator-owned assignments.
-
-The RFC describes several of these capabilities as planned phases. Do not treat
-the RFC as evidence that they have shipped.
-
-## Optimization decisions
-
-Use this table before changing Ollama or FreeLlama:
-
-| Candidate change | Default decision | Reason |
-|---|---|---|
-| Expose Ollama beyond loopback | Do not add | Keep `OLLAMA_HOST` on loopback and put authenticated TLS ingress in front of FreeLlama instead |
-| Permit Ollama cloud features in a local-only deployment | Do not add | Set `OLLAMA_NO_CLOUD=1` when the deployment promises local-only operation |
-| Force layers of one model between CPU and GPU | Do not add | Ollama already fits and offloads the model; partial CPU fallback usually reduces decode throughput |
-| Assign different models to isolated CPU and GPU-capable processes | Supported, benchmark per workload | Process isolation is reliable, but CPU speed and shared-memory pressure still determine whether overlap helps |
-| Add macOS kernel or `sysctl` tuning | Do not add | Ollama does not document a supported, product-specific kernel setting. Unified-memory behavior belongs to macOS and Metal |
-| Run Ollama in Docker on macOS | Do not use for performance | Docker Desktop does not provide Ollama GPU acceleration on macOS |
-| Increase context globally | Do not add | Context memory grows with length; set the smallest sufficient `num_ctx` per request |
-| Set parallel requests above 1 | Benchmark first | It can improve concurrent throughput but multiplies context memory and can hurt interactive latency |
-| Keep more than one model loaded per process | Benchmark first | Model weights, runtime buffers, and every active context must fit together; start with `OLLAMA_MAX_LOADED_MODELS=1` |
-| Leave Ollama's queue without a deployment limit | Do not add | Set a measured `OLLAMA_MAX_QUEUE`; it remains separate from FreeLlama's admission deadline |
-| Pin multiple 18–21 GB models | Do not add by default | The observed 36 GiB Metal budget cannot safely establish that two heavy models plus contexts fit |
-| Force Flash Attention | Benchmark per engine and model | The inspected GGUF path already enabled it automatically; MLX has a different runner |
-| Use `q8_0` K/V cache | **8/10; qualify, then prefer for parallel/long-context work** | Ollama documents roughly half the KV memory of `f16` with very small precision loss; it is the practical companion to higher `OLLAMA_NUM_PARALLEL`, but it remains a process-wide quality tradeoff |
-| Use `q4_0` K/V cache | Reject as a default | It saves more memory with a larger possible quality loss |
-| Enable High Power Mode | Sustained-load experiment only | Apple documents higher sustained performance. Local token-rate and energy effects are not measured |
-| Keep the small intent model resident | Keep bounded | It reduces routing latency but consumes memory and can affect a large-model transition |
-
-## Measurement contract
-
-Measure optimization with Ollama's response fields:
-
-- prompt tokens per second = `prompt_eval_count / prompt_eval_duration` in
-  seconds;
-- output tokens per second = `eval_count / eval_duration` in seconds;
-- cold-load cost = `load_duration`;
-- end-to-end latency = `total_duration` plus measured FreeLlama overhead;
-- workload throughput = successful, quality-qualified tasks per hour.
-
-Compare one variable at a time against stock Ollama. Include cold and warm runs,
-fixed prompts, fixed output budgets, exact-output guardrails, quality guardrails, resident
-memory, swap, and failure counts. A higher token rate is not an improvement if
-quality falls, requests fail, or model transitions dominate the workload.
-
-Use `cargo run -- bench-all` (`packages/rust-core/src/model_bench.rs`) for model selection evidence.
-Use [CPU and GPU model routing](CPU_GPU_ROUTING.md) for the separate-process benchmark and
-verification procedure.
-
-- Local-model router RFC — lived in the gitignored `.octocode/` workspace and is no longer in the tree; the shipped design it argued for is [ARCHITECTURE.md](ARCHITECTURE.md)
-
-## Recommended next work
-
-1. Keep the CLI executable aligned with the active Ollama server; `doctor` reports the selected
-   path, both versions, and effective memory-related settings.
-2. Back up the versioned prompt-free feedback snapshot with the policy and deployed binary.
-   Verified warm work-unit latency and queue wait survive restart; raw prompts are never stored.
-3. Validate the metadata-backed K/V lower bound against a frozen alternating-model workload before
-   widening it beyond CPU/unified-memory impossible-floor refusal. Keep Ollama as final authority.
-4. Extend transition coordination only if managed streaming or another parsed
-   execution path needs it. Native passthrough remains Ollama-owned.
-5. Evaluate request-specific context and residency profiles before testing global K/V-cache or
-   concurrency changes. Keep Flash Attention automatic on supported backends or set it explicitly
-   to `1` before selecting a quantized K/V cache.
-6. Promote a hardware-and-engine profile only after `benchmark/hardware/run_validation.py` and its
-   held-out quality workload pass on a real self-hosted runner.
-
-## Sources
-
-- [Ollama FAQ and server settings](https://docs.ollama.com/faq)
-- [Ollama context length](https://docs.ollama.com/context-length)
-- [Ollama hardware and Metal support](https://docs.ollama.com/gpu)
-- [Ollama API usage metrics](https://docs.ollama.com/api/usage)
-- [Ollama embedding API](https://docs.ollama.com/api/embed)
-- [Ollama macOS support](https://docs.ollama.com/macos)
-- [Apple power modes](https://support.apple.com/en-lamr/101613)

package/docs/PRODUCT_POSITIONING.md

@@ -1,291 +0,0 @@
-# Product positioning
-
-FreeLlama is a **governed local-model delegation layer for developers who run AI agents on
-Ollama**. It helps an agent decide whether local inference is appropriate and select an eligible
-installed model. It executes within machine and policy limits and returns evidence about what happened.
-
-The shortest accurate description is:
-
-> Ollama runs local models. FreeLlama helps agents use them deliberately.
-
-This positioning is intentionally narrower than “AI gateway” and more useful than “Ollama proxy.”
-FreeLlama preserves Ollama's API, but its distinctive work is the decision and evidence layer around
-inference: model qualification, fail-closed routing, admission, CPU/GPU placement, grounded
-research, and execution receipts.
-
-## Product category decision
-
-| Question | Decision |
-|---|---|
-| Product category | Local-model delegation control plane |
-| Primary audience | Developers operating coding or research agents with Ollama |
-| Core job | Move suitable work to local models without silently lowering quality or losing control of the machine |
-| Primary alternative | Call Ollama directly and implement routing, verification, and resource coordination in every agent |
-| Product promise | Make local delegation explicit, bounded, observable, and measurable |
-| Not the product | A chat UI, an Ollama fork, a hosted inference service, or a generic multi-provider gateway |
-
-“Trustworthy local-model delegation” is the clearest value phrase. “Localhost control plane” is an
-accurate architectural phrase, but it supports the explanation rather than leads it.
-
-## How the product fits
-
-```mermaid
-flowchart LR
-    U["Developer"] --> A["AI agent or application"]
-    A -->|"MCP, CLI, Rust, or control API"| F["FreeLlama"]
-    F --> D{"Delegate?\nWhich model?\nWhich backend?"}
-    D -->|"qualified GPU task"| G["Primary Ollama\nGPU-capable runner"]
-    D -->|"explicit CPU assignment"| C["Secondary Ollama\nCPU runner"]
-    D -->|"guard fails"| X["Refuse before inference"]
-    G --> E["Answer + execution receipt"]
-    C --> E
-    X --> E
-    E --> A
-```
-
-[Ollama](https://docs.ollama.com/api/introduction) owns model execution and its API. FreeLlama
-owns the policy around managed use of that runtime. The [Model Context Protocol
-(MCP)](https://modelcontextprotocol.io/specification/2025-06-18/architecture) makes those controls
-available to compatible agent hosts as seven bounded tools. Native Ollama requests can still pass
-through unchanged to the primary upstream.
-
-See [FreeLlama architecture](ARCHITECTURE.md) for the complete responsibility and request-flow
-contracts.
-
-## Audience hierarchy
-
-### Primary: developers operating AI agents
-
-The strongest user is a developer who already uses a capable cloud or local agent and wants to
-delegate bounded work to Ollama. Examples include grounded repository research, embeddings, OCR,
-bulk transforms, and helper-model tasks.
-
-Their job is not merely “run a model.” It is:
-
-1. Determine whether a local model is suitable for the task.
-2. Avoid sending work to an incapable, unmeasured, or oversized model.
-3. Keep large files, vectors, images, and intermediate tool output out of the orchestrator's
-   context.
-4. Coordinate concurrent work without uncontrolled model loads or memory pressure.
-5. Receive citations, verdicts, placement, queue, and timing signals that the agent can inspect.
-
-FreeLlama fits this user because its MCP server, CLI, agent skill, and control API expose exactly
-those decisions. The core job-to-be-done is:
-
-> When my agent encounters expensive but bounded work, decide whether a local model can handle it,
-> run it under explicit limits, and give the agent enough evidence to trust or reject the result.
-
-### Secondary: Ollama power users and local-AI operators
-
-An operator with several installed models can use FreeLlama to diagnose environment drift, inspect
-residency, preview routes, coordinate admission, assign selected helpers to a CPU backend, and turn
-benchmark results into routing policy.
-
-This audience can use the CLI without an MCP host. The relevant value is machine-aware operation,
-not agent integration alone. FreeLlama discovers the current host and observes Ollama state instead
-of embedding assumptions from the development Mac. See [model selection](MODEL_SELECTION.md) and
-[CPU/GPU routing](CPU_GPU_ROUTING.md).
-
-### Secondary: developers embedding local-model policy
-
-Library and tool authors can use the Rust core, Node native boundary, or HTTP control API instead
-of rebuilding discovery, routing, admission, and evaluation. This is an integration audience, not
-the clearest first-run story.
-
-### Adjacent: Ollama developers and contributors
-
-FreeLlama can help Ollama contributors as a downstream compatibility and workload testbed. Its
-diagnostics expose version drift, native passthrough tests protect API compatibility, and its
-dual-process experiments document observed runner placement.
-
-Ollama developers are not the primary audience. FreeLlama does not replace Ollama's scheduler,
-runner, model storage, accelerator backends, or token generation. It is a sidecar that exercises
-and governs Ollama from the client side. Describe the relationship as complementary:
-
-> Ollama makes local inference possible; FreeLlama makes agent delegation to it governable.
-
-### Not a target: mainstream users
-
-FreeLlama is not a mainstream local-AI product. It has no end-user chat interface. Meaningful use
-assumes familiarity with Ollama and its model tags. Users also need a terminal or MCP host and an
-understanding of policy or benchmark concepts.
-[Open WebUI](https://github.com/open-webui/open-webui) already addresses approachable self-hosted
-AI interfaces.
-
-A separate desktop or web interface can sit on top of FreeLlama as a new product layer. Keep that
-layer from blurring the current audience or forcing consumer concerns into the control plane.
-
-## The gap FreeLlama fills
-
-Calling Ollama directly is the correct path when the caller already knows the exact model
-and request. FreeLlama becomes useful when the caller has to make and defend a delegation decision.
-
-| Direct Ollama question | FreeLlama question |
-|---|---|
-| Which model tag receives this request? | Which installed model qualifies for this task and confidence threshold? |
-| Did the request return HTTP 200? | Was it admitted, where did it execute, and what did it cost? |
-| Can the model call tools? | Did bounded research read evidence and produce verifiable citations? |
-| Can Ollama load this model? | Does it fit the discovered machine and current resident workload? |
-| Can I set request options? | Can this managed model use the explicitly assigned CPU backend without changing raw traffic? |
-
-This distinction also separates FreeLlama from a broad gateway such as
-[LiteLLM](https://github.com/BerriAI/litellm). LiteLLM unifies many providers and adds centralized
-authentication, budgets, spend tracking, and load balancing. FreeLlama is Ollama-specific,
-machine-aware, and centered on agent delegation evidence. Its bearer boundary supports a trusted
-operator or team; it does not attempt LiteLLM's tenant, spend, or provider-management scope.
-
-[RouteLLM](https://github.com/lm-sys/RouteLLM) is the closer routing-specific comparison. It trains
-and evaluates routers that select between stronger and weaker model endpoints. FreeLlama does not
-implement a learned prompt router. It applies deterministic local eligibility and operator policy,
-then adds host fit, admission, physical CPU/GPU placement evidence, and grounded-research receipts.
-
-## The product loop
-
-The flexible part of FreeLlama is not an unconstrained agent choosing hardware. It is a guarded
-feedback loop with explicit operator and policy boundaries.
-
-```mermaid
-flowchart TD
-    T["Agent describes task, objective, and constraints"] --> P["Preview route"]
-    P --> Q{"Capability, policy, confidence,\nand capacity qualify?"}
-    Q -->|"no"| R["Refuse with reasons"]
-    Q -->|"yes"| A["Admit on selected backend"]
-    A --> O["Ollama executes"]
-    O --> V["Return result, placement, queue,\nand timing evidence"]
-    V --> F["Record normalized warm feedback"]
-    F -.->|"auto; fastest or balanced only\nafter enough samples"| P
-```
-
-An agent can request `auto`, `prefer_cpu`, or `prefer_gpu`, but the request remains a preference.
-Explicit model choice, session affinity, capability requirements, policy, confidence, and operator
-CPU assignments remain authoritative. Runtime feedback only influences eligible `auto` routes for
-speed-oriented objectives after comparable warm samples exist; it never overrides a quality
-objective. This design lets the system adapt to another machine without turning resource placement
-into an opaque autonomous decision.
-
-## Message hierarchy
-
-Use one message at each level:
-
-| Context | Message |
-|---|---|
-| Category | Local-model delegation control plane |
-| One line | FreeLlama helps AI agents use Ollama deliberately, with routing, resource limits, and evidence. |
-| Problem | Local inference is direct to call but hard to delegate to safely across different models and machines. |
-| Promise | Move suitable work local without silently lowering quality or destabilizing the runtime. |
-| Mechanism | Qualify, preview, admit, execute, and return a receipt. |
-| Boundary | Ollama runtime underneath; FreeLlama policy around it. |
-
-Good headline directions are outcome-first:
-
-- **Use local models without lowering your standards.**
-- **Give your AI agent a governed path to Ollama.**
-- **Delegate locally. Keep the evidence.**
-
-Avoid “AI orchestration platform.” It is broad enough to hide the product. Avoid leading with
-“proxy,” because raw passthrough is a compatibility feature rather than the principal value.
-
-## Build credible attention
-
-Create interest through short, reproducible contrasts rather than market-size or
-speed claims.
-
-### Demonstration 1: refuse before wasting work
-
-Show an agent previewing a task, rejecting an unqualified model, and returning the reason without
-an inference call. The story is not “the router is smart”; it is “a weak local model cannot quietly
-produce a confident answer.”
-
-### Demonstration 2: preserve frontier context
-
-Give `delegate_research` a narrow repository question. Show the local adapter reading allowlisted
-files and returning an answer, citations, and a verification verdict instead of returning all source
-material and intermediate tool output to the orchestrator.
-
-One measured six-question run reduced 59,208 source tokens to 1,742 returned tokens, or 97.1%
-context offload. That is a single measured workload, not a universal savings rate. The durable
-claim is context isolation; see [token economics](ECONOMICS.md).
-
-### Demonstration 3: use the machine as a system
-
-Run a large GPU completion and a small CPU helper concurrently, then show both placement receipts.
-On the measured 48 GB Apple M4 Pro, three warmed trials reduced median wall time from 37.997 to
-28.233 seconds, a 1.346-times speedup. One of the three parallel trials was slower than sequential,
-so the claim is “these workloads can overlap,” not “CPU/GPU mode is always faster.” See the full
-[CPU/GPU measurement](CPU_GPU_ROUTING.md#interpret-the-measured-mac-result).
-
-These demonstrations map directly to three memorable benefits: **trust, context, and resources**.
-Turn them into a short terminal recording, copyable README recipes, and versioned benchmark
-receipts. Each demo needs the command, machine profile, model tags, policy, result, and
-failure conditions.
-
-## Adoption path
-
-The first-use sequence needs to reveal value before asking for sophisticated configuration:
-
-```mermaid
-flowchart LR
-    I["Install beside Ollama"] --> D["doctor\nsee machine and drift"]
-    D --> P["preview\nsee a route and reasons"]
-    P --> T["run one managed task\nsee the receipt"]
-    T --> G["delegate research\nsee citations and verdict"]
-    G --> B["benchmark\nmeasure local models"]
-    B --> Y["policy\nallow proven routes"]
-    Y --> R["resource tuning\nCPU/GPU and admission"]
-```
-
-The product must not require a new user to understand every tool, KV-cache setting, or routing
-objective before the first managed task. Advanced controls become relevant after the first receipt
-makes the control loop visible.
-
-## Claim and naming guardrails
-
-| Do not say | Say instead |
-|---|---|
-| Free local inference | Local inference uses hardware, power, memory, and time; FreeLlama can preserve paid frontier context. |
-| Automatically picks the optimal model | Routes among eligible installed models using explicit policy and available evidence. |
-| Makes Ollama faster | Can improve workload efficiency; it does not increase one model's decode speed. |
-| Runs CPU and GPU in parallel on every machine | Can overlap explicitly assigned workloads when the host and Ollama backends support them; measure the result. |
-| Works the same on any hardware | Uses platform-specific host discovery on macOS, Linux, and Windows. Portable defaults do not remove the need to observe accelerator behavior. |
-| Replaces Ollama | Runs beside Ollama and preserves its native API. |
-| For everyone | Built first for developers and local-AI operators. |
-| Llama models only | Works with qualifying models exposed by Ollama; “Llama” is the product name, not a model-family restriction. |
-
-Use the word “Free” to refer to reducing reliance on metered frontier inference, not zero-cost
-computing. Readers can misread “Llama” as Meta Llama-only, so put the Ollama relationship and model
-agnosticism within Ollama near the first description.
-
-## Product decision
-
-Narrow the story before broadening the product:
-
-1. Lead with developers operating AI agents, not all developers.
-2. Make trustworthy delegation the category-defining job.
-3. Use MCP as the primary agent integration and the CLI as the operator surface.
-4. Treat Ollama power users and embedders as secondary audiences.
-5. Treat Ollama contributors as an adjacent ecosystem audience, not the main customer.
-6. Do not pursue a mainstream chat interface until the developer onboarding loop is clear and
-   repeatable.
-
-This decision matches the code that exists today. The Rust core, CLI, MCP server, proxy, agent
-skill, benchmark harnesses, and policy system all support a developer control plane. Positioning it
-as a consumer app promises a product surface the repository does not provide.
-
-## Related documentation
-
-- [Architecture](ARCHITECTURE.md): ownership boundaries and request flows
-- [MCP server](../packages/mcp/README.md): six agent-facing tools and their contracts
-- [CLI](CLI.md): operator commands and control-plane workflows
-- [Model selection](MODEL_SELECTION.md): evidence, qualification, and recommendation
-- [CPU/GPU routing](CPU_GPU_ROUTING.md): portable configuration, feedback, and measured limits
-- [Testing](TESTING.md): verification tiers and live-runtime boundaries
-- [Token economics](ECONOMICS.md): measured context isolation and its limits
-
-## Sources
-
-- [Ollama API introduction](https://docs.ollama.com/api/introduction)
-- [Ollama tool calling](https://docs.ollama.com/capabilities/tool-calling)
-- [Model Context Protocol architecture](https://modelcontextprotocol.io/specification/2025-06-18/architecture)
-- [Open WebUI](https://github.com/open-webui/open-webui)
-- [LiteLLM](https://github.com/BerriAI/litellm)
-- [RouteLLM](https://github.com/lm-sys/RouteLLM)