@octocodeai/freellama-mcp-server 0.5.0 → 0.6.1

package/docs/dev/OLLAMA_SYSTEM_OPTIMIZATION.md

@@ -1,255 +0,0 @@
-# Ollama and FreeLlama optimization boundary
-
-FreeLlama manages model selection and resources for tasks offloaded by agents. Ollama owns
-inference optimization. FreeLlama can improve completed work per hour by avoiding unsuitable
-routes and unnecessary model transitions; measure that effect for the target workload.
-
-This page explains the active request flow, the settings Ollama owns, the
-current machine audit, and the optimization work that remains.
-
-The architecture is portable across Ollama's Metal, CUDA, ROCm, Vulkan, and CPU paths. The audit
-later on this page is intentionally Mac-specific. FreeLlama discovers host RAM per operating system
-and keeps device selection in each Ollama process; it does not translate one machine's memory tier,
-core count, or benchmark into another machine's defaults.
-
-## Decision
-
-Use stock Ollama as the inference runtime. Keep FreeLlama as an optional Rust
-control plane for evidence-qualified routing, bounded request profiles, session
-affinity, and policy enforcement.
-
-Do not advertise FreeLlama as an accelerator or an Ollama inference plugin. It does not tune
-kernels or split one model's layers across CPU and GPU. It responds to observed host pressure by
-holding managed work; the operating system manages physical memory. FreeLlama can assign different
-models to isolated CPU and GPU-capable Ollama
-processes, allowing independent workloads to overlap.
-
-## Request flow
-
-FreeLlama has four distinct paths:
-
-| Request | FreeLlama work | Ollama work | Result |
-|---|---|---|---|
-| Native `/api/*` or `/v1/*` | Streams bytes to the primary compatibility upstream | Performs normal inference and scheduling | Exact Ollama response |
-| `POST /_freellama/v1/routes` | Discovers models, filters capabilities, and applies policy | Supplies installed-model and residency data | Model and request profile; no inference |
-| `POST /_freellama/v1/natural-routes` | Normalizes a schema-bound intent, then runs the deterministic router | Runs the small intent model | Model and request profile; selected task does not run |
-| `POST /_freellama/v1/tasks` | Selects a route and backend, applies managed-task admission, and constructs one bounded request | Runs one nonstreaming chat or embedding request on the assigned process | Route receipt, placement, admission mode, prompt-free metrics, and Ollama response |
-
-The natural-language path is intentionally two-stage. The small model cannot
-name a final model. It returns only a task, objective, context hint, and tool or
-vision requirements. Deterministic guards correct explicit requirements before
-the ordinary router selects a model.
-
-Calling `natural-routes` does not run the selected task. A client must use the
-returned profile or submit a structured `tasks` request. There is no atomic
-natural-language route-and-run endpoint in the current release.
-
-Chat and embedding tasks forward route options. A captured-upstream contract
-test verifies embedding option parity.
-
-## What Ollama already optimizes
-
-Across supported backends, Ollama provides the following behavior without FreeLlama:
-
-- The runtime selects supported acceleration such as Metal, CUDA, ROCm, or Vulkan from the host and
-  its process-level device visibility.
-- The scheduler fits model, context, and compute allocations to available
-  device memory.
-- The runner selects CPU threads and GPU execution; applications do not need to
-  divide one inference manually between the CPU and GPU.
-- Ollama manages request queues plus model loading and eviction. It supports
-  same-model parallel requests when memory permits.
-- Ollama reports token counts plus load, prompt-processing, and generation
-  durations in API responses.
-- Ollama supports request-specific context length and model residency through
-  `num_ctx` and `keep_alive`.
-
-Ollama's documented concurrency controls are global server settings. Increasing
-`OLLAMA_NUM_PARALLEL` multiplies context memory by the parallel-request count.
-Increasing `OLLAMA_MAX_LOADED_MODELS` does not create memory; models still need
-to fit. Use `1` for both values on an unmeasured deployment, then increase one variable at a time
-only when a representative concurrent benchmark improves completed tasks per hour without memory
-pressure.
-
-Keep three signals distinct when operating FreeLlama: `context_window_fit` only compares requested
-context with advertised model context; FreeLlama's K/V preflight is a conservative metadata lower
-bound; and a completed runner is the only placement observation. None is live free-memory or
-throughput proof. `doctor {view:"scheduler"}` reports configuration and a point-in-time admission
-snapshot, not a fairness or parallel-decoding benchmark.
-
-FreeLlama's admission queue and Ollama's internal queue solve different problems. FreeLlama limits
-weighted managed work per backend and applies `--max-queue-wait-seconds`; `OLLAMA_MAX_QUEUE` bounds
-requests that an Ollama process has accepted but cannot schedule. Raw compatibility traffic skips
-FreeLlama admission and reaches the primary Ollama queue directly.
-
-Ollama request context and research-adapter context are also separate budgets. Managed tasks set
-request-specific `num_ctx` from the selected task profile. The coding adapters fit their tool
-conversation inside `FREELLAMA_AGENT_NUM_CTX`: by default, they preserve the system prompt and
-original question, compact the oldest observations first, retain recent evidence, and fail before
-the Ollama call if pinned content cannot fit. Raising the Ollama server's global context does not
-disable adapter compaction or increase its configured budget. See [adapter contracts](../../benchmark/local/docs/07-adapter-contracts.md)
-for the complete pagination and compaction contract.
-
-## Mac audit (2026-09-01)
-
-This audit captures the following snapshot from September 1, 2026. Treat it as machine
-evidence, not a portable default.
-
-| Item | Observed value | Interpretation |
-|---|---|---|
-| Hardware | Apple M4 Pro, 14 CPU cores, 48 GB unified memory | CPU and GPU share the memory system |
-| macOS | 15.7.2 | Native Ollama supports this Apple-silicon system |
-| Ollama server | 0.33.2 | Active application server on `127.0.0.1:11434`; re-read with `doctor` rather than trusting this row |
-| First CLI on `PATH` | `/usr/local/bin/ollama`, 0.33.2 | Matches the active server |
-| Metal device budget | 36 GiB reported by Ollama | This is the scheduler's observed device budget, not all 48 GB |
-| Default context | 32,768 tokens | Ollama selected its 24–48 GiB device-memory tier |
-| Parallel requests | 1 | Appropriate baseline for interactive and large-model tests |
-| Keep-alive | 5 minutes | Ollama default |
-| Loaded-model limit | `OLLAMA_MAX_LOADED_MODELS=2` | Explicit service configuration for the measured topology |
-| K/V-cache type | `OLLAMA_KV_CACHE_TYPE=q8_0` | Explicit memory-saving configuration; requalify quality per important model |
-| Flash Attention | `auto` | Ollama enables it automatically on supported backends; force `1` only after a measured compatibility check. Required for quantized K/V cache. |
-| GGUF CPU threads | 10 worker threads | Ollama selected the 10 performance cores rather than all 14 cores |
-| MLX | GPU runner observed | MLX models use their own runner path, not the GGUF tuning path |
-| Power mode | Automatic on battery and AC power | High Power Mode remains an unmeasured experiment |
-
-The server log also showed Metal fusion, concurrent Metal execution, graph
-optimization, and automatic parameter fitting. FreeLlama must not claim these
-as its own optimizations.
-
-### Verify CLI and server alignment
-
-The selected CLI is `/usr/local/bin/ollama`, and both the CLI and active server report 0.33.2.
-Verify this alignment after changing Ollama installations or service definitions:
-
-```bash
-command -v ollama
-ollama --version
-curl --silent http://127.0.0.1:11434/api/version
-```
-
-The client and server versions must match before you test a CLI feature. A mismatch does not prove
-an inference slowdown, but it can produce incompatible commands, flags, or diagnostics.
-
-## What FreeLlama adds today
-
-The active Rust implementation adds:
-
-- capability filtering before speed ranking;
-- explicit `fastest`, `balanced`, and `quality` evidence contracts;
-- fail-closed quality routing when the policy has no qualified model;
-- task-specific context, output, thinking, and tool-validation profiles;
-- session affinity and preference for an eligible resident model;
-- separate transition locks for primary and CPU backends, with shared execution permits for
-  resident tasks and an exclusive permit for a cold transition;
-- exact-model placement on an optional second CPU Ollama process, including `num_gpu=0` for its
-  managed requests;
-- guarded agent placement preferences plus a normalized, three-warm-sample, 10%-advantage feedback
-  loop for `fastest` and `balanced` work;
-- independent 3:2:1 priority-fair weighted GPU and CPU admission pools, so a GPU burst cannot
-  starve a CPU helper or indefinitely starve background managed work;
-- prompt-free load, prompt-processing, and output-generation metrics derived
-  from Ollama's response fields;
-- a local natural-language intent interpreter that cannot choose a model;
-- an Ollama-compatible loopback endpoint and an escape hatch to port 11434.
-
-These features can reduce reloads or avoid an unsuitable model. They do not
-change the selected model's prompt-processing or decode rate.
-
-The measured dual-backend workload improved median completion time from 37.997 to 28.233 seconds
-(1.346 times, or 25.70%). Qwen reported 19,175,677,668 GPU-resident bytes, Nomic reported zero,
-and all requests carried correct backend receipts with zero FreeLlama queue wait. One of three
-parallel trials was slower than sequential, so this remains a workload-level optimization rather
-than an inference-speed claim.
-
-## Unsupported optimizations
-
-FreeLlama does not provide:
-
-- exact runner-allocation prediction. Managed admission uses observed available RAM, free VRAM
-  when available, and a metadata-backed F16 K/V estimate; unknown layouts remain unknown. Ollama
-  remains authoritative for runner allocation and final loading;
-- coordination for processes that bypass FreeLlama. Mutating passthrough requests in `serve`
-  share the primary transition lock;
-- direct control over thermal state, power mode, or swapping. Observed pressure can hold admission;
-- model-quality scoring inferred from operational latency or errors;
-- per-engine tuning for GGUF compared with MLX;
-- unbounded automatic residency or concurrency. Runtime limits remain operator-owned;
-- an atomic natural-language route-and-run operation;
-- held-out quality policies for completion, coding, tools, browser, vision, or
-  long-context tasks in the example policy;
-- quality-aware or thermal-aware CPU/GPU load balancing. Automatic placement uses persisted,
-  prompt-free warm-latency aggregates only within exact operator-owned assignments.
-
-The RFC describes several of these capabilities as planned phases. Do not treat
-the RFC as evidence that they have shipped.
-
-## Optimization decisions
-
-Use this table before changing Ollama or FreeLlama:
-
-| Candidate change | Default decision | Reason |
-|---|---|---|
-| Expose Ollama beyond loopback | Do not add | Keep `OLLAMA_HOST` on loopback and put authenticated TLS ingress in front of FreeLlama instead |
-| Permit Ollama cloud features in a local-only deployment | Do not add | Set `OLLAMA_NO_CLOUD=1` when the deployment promises local-only operation |
-| Force layers of one model between CPU and GPU | Do not add | Ollama already fits and offloads the model; partial CPU fallback usually reduces decode throughput |
-| Assign different models to isolated CPU and GPU-capable processes | Supported, benchmark per workload | Process isolation is reliable, but CPU speed and shared-memory pressure still determine whether overlap helps |
-| Add macOS kernel or `sysctl` tuning | Do not add | Ollama does not document a supported, product-specific kernel setting. Unified-memory behavior belongs to macOS and Metal |
-| Run Ollama in Docker on macOS | Do not use for performance | Docker Desktop does not provide Ollama GPU acceleration on macOS |
-| Increase context globally | Do not add | Context memory grows with length; set the smallest sufficient `num_ctx` per request |
-| Set parallel requests above 1 | Benchmark first | It can improve concurrent throughput but multiplies context memory and can hurt interactive latency |
-| Keep more than one model loaded per process | Benchmark first | Model weights, runtime buffers, and every active context must fit together; start with `OLLAMA_MAX_LOADED_MODELS=1` |
-| Leave Ollama's queue without a deployment limit | Do not add | Set a measured `OLLAMA_MAX_QUEUE`; it remains separate from FreeLlama's admission deadline |
-| Pin multiple 18–21 GB models | Do not add by default | The observed 36 GiB Metal budget cannot safely establish that two heavy models plus contexts fit |
-| Force Flash Attention | Benchmark per engine and model | The inspected GGUF path already enabled it automatically; MLX has a different runner |
-| Use `q8_0` K/V cache | Qualify quality and memory effects per workload | KV quantization changes memory use and precision for the whole process; compare representative task quality and resident memory before deployment |
-| Use `q4_0` K/V cache | Reject as a default | It saves more memory with a larger possible quality loss |
-| Enable High Power Mode | Sustained-load experiment only | Apple documents higher sustained performance. Local token-rate and energy effects are not measured |
-| Keep the small intent model resident | Keep bounded | It reduces routing latency but consumes memory and can affect a large-model transition |
-
-## Measurement contract
-
-Measure optimization with Ollama's response fields:
-
-- prompt tokens per second = `prompt_eval_count / prompt_eval_duration` in
-  seconds;
-- output tokens per second = `eval_count / eval_duration` in seconds;
-- cold-load cost = `load_duration`;
-- end-to-end latency = `total_duration` plus measured FreeLlama overhead;
-- workload throughput = successful, quality-qualified tasks per hour.
-
-Compare one variable at a time against stock Ollama. Include cold and warm runs,
-fixed prompts, fixed output budgets, exact-output guardrails, quality guardrails, resident
-memory, swap, and failure counts. A higher token rate is not an improvement if
-quality falls, requests fail, or model transitions dominate the workload.
-
-Use `cargo run -- bench-all` (`packages/rust-core/src/model_bench.rs`) for model selection evidence.
-Use [CPU and GPU model routing](../CPU_GPU_ROUTING.md) for the separate-process benchmark and
-verification procedure.
-
-- Local-model router RFC — lived in the gitignored `.octocode/` workspace and is no longer in the tree; the shipped design it argued for is [ARCHITECTURE.md](../ARCHITECTURE.md)
-
-## Recommended next work
-
-1. Keep the CLI executable aligned with the active Ollama server; `doctor` reports the selected
-   path, both versions, and effective memory-related settings.
-2. Back up the versioned prompt-free feedback snapshot with the policy and deployed binary.
-   Verified warm work-unit latency and queue wait survive restart; raw prompts are never stored.
-3. Validate the metadata-backed K/V lower bound against a frozen alternating-model workload before
-   widening it beyond CPU/unified-memory impossible-floor refusal. Keep Ollama as final authority.
-4. Extend transition coordination only if managed streaming or another parsed
-   execution path needs it. Native passthrough remains Ollama-owned.
-5. Evaluate request-specific context and residency profiles before testing global K/V-cache or
-   concurrency changes. Keep Flash Attention automatic on supported backends or set it explicitly
-   to `1` before selecting a quantized K/V cache.
-6. Promote a hardware-and-engine profile only after `benchmark/hardware/run_validation.py` and its
-   held-out quality workload pass on a real self-hosted runner.
-
-## Sources
-
-- [Ollama FAQ and server settings](https://docs.ollama.com/faq)
-- [Ollama context length](https://docs.ollama.com/context-length)
-- [Ollama hardware and Metal support](https://docs.ollama.com/gpu)
-- [Ollama API usage metrics](https://docs.ollama.com/api/usage)
-- [Ollama embedding API](https://docs.ollama.com/api/embed)
-- [Ollama macOS support](https://docs.ollama.com/macos)
-- [Apple power modes](https://support.apple.com/en-lamr/101613)