@octocodeai/freellama-mcp-server 0.6.2 → 0.9.0

package/docs/CPU_GPU_ROUTING.md

@@ -27,8 +27,8 @@ machine-specific:
 | CPU-only | CPU | A second process can isolate helpers, but it does not create a second accelerator | Host RAM and measured overlap |
 
 `doctor` reports `machine.memory_bytes` on macOS, Linux, and Windows. It reports
-`machine.unified_memory_bytes` only when unified memory is known; that compatibility field is not a
-VRAM estimate on discrete-GPU hosts. `machine.gpu` names the accelerator when the OS reports it
+`machine.memory_kind` as `unified` only when host and accelerator share one memory pool; host RAM
+is not a VRAM estimate on discrete-GPU hosts. `machine.gpu` names the accelerator when the OS reports it
 (Apple model and core count, or NVIDIA name and memory on Linux), and `performance_cpus` /
 `efficiency_cpus` split hybrid CPUs on macOS. These are identity facts, not placement proof. Model-library fit is therefore a host-memory preflight, and
 actual accelerator placement must come from Ollama's resident-runner report.
@@ -68,7 +68,18 @@ OLLAMA_HOST=127.0.0.1:11434 OLLAMA_NO_CLOUD=1 \
     OLLAMA_MAX_LOADED_MODELS=1 OLLAMA_NUM_PARALLEL=1 ollama serve
 ```
 
-Start a second process on another loopback port:
+Or let `serve` start and own the CPU process: `--spawn-cpu-backend` runs `ollama serve` on
+`127.0.0.1:11436` (`--cpu-backend-port`, `--ollama-binary`) with `OLLAMA_NO_CLOUD=1`,
+`OLLAMA_LLM_LIBRARY=cpu`, `OLLAMA_NUM_PARALLEL=1` and every GPU runtime hidden
+(`CUDA_VISIBLE_DEVICES`, `ROCR_VISIBLE_DEVICES`, `HIP_VISIBLE_DEVICES`, `GGML_VK_VISIBLE_DEVICES` set
+to `-1`), waits until it answers, logs it to `cpu-ollama.log` in the FreeLlama home, and stops it
+when `serve` exits. A port that already answers is refused rather than adopted:
+
+```bash
+npx @octocodeai/freellama serve --spawn-cpu-backend --cpu-model nomic-embed-text:v1.5
+```
+
+To run the second process yourself, start it on another loopback port:
 
 ```bash
 OLLAMA_HOST=127.0.0.1:11436 OLLAMA_NO_CLOUD=1 \
@@ -205,7 +216,7 @@ curl --silent http://127.0.0.1:11435/_freellama/v1/tasks \
     }' | jq '{execution, admission, route}'
 ```
 
-The `execution` object separates the configured `backend`, requested compatibility `placement`, and
+The `execution` object separates the configured `backend`, the `requested_processor`, and
 post-run `observation`. The source is Ollama `/api/ps`; direct inspection remains useful:
 
 ```bash
@@ -221,9 +232,12 @@ samples are reported but excluded from adaptive feedback.
 
 ## Understand concurrency
 
-Each backend has its own weighted admission pool and its own resident/shared and
-transition/exclusive lock. CPU and GPU model transitions therefore do not block each other, and a
-GPU burst cannot spend the CPU helper's permit:
+Each backend has its own weighted admission pool and its own load lock (see
+[Load lock and use lease](#load-lock-and-use-lease)). CPU and GPU model loads therefore do not block
+each other, and a GPU burst cannot spend the CPU helper's permit:
+
+Within a priority class, each pool prefers a waiter whose model is already running on its backend
+(or was admitted there last), within bounded skips; see [the admission design](ARCHITECTURE.md#managed-task-execution).
 
 Both pools also share local host-pressure sampling and forecast-memory reservations. Independent
 queues do not imply independent RAM on a unified-memory machine. A request can wait for available
@@ -238,10 +252,10 @@ flowchart TD
     B -->|"CPU"| CA["CPU admission pool<br/>default 1 unit"]
     GA --> GL{"Resident?"}
     CA --> CL{"Resident?"}
-    GL -->|"Yes"| GR["GPU shared permit"]
-    GL -->|"No"| GX["GPU exclusive transition"]
-    CL -->|"Yes"| CR["CPU shared permit"]
-    CL -->|"No"| CX["CPU exclusive transition"]
+    GL -->|"Yes"| GR["Use lease only"]
+    GL -->|"No"| GX["GPU load lock until the runner is observed"]
+    CL -->|"Yes"| CR["Use lease only"]
+    CL -->|"No"| CX["CPU load lock until the runner is observed"]
     GR --> GO["Primary Ollama"]
     GX --> GO
     CR --> CO["Secondary Ollama"]
@@ -260,16 +274,92 @@ the entire pool's ceiling. Default vision weight four can serialize two vision r
 four-unit budget even when Ollama has two slots. A lower weight requires a qualified workload;
 it does not add engine slots or relax memory checks. See
 [monitoring controls](MONITORING.md#adaptive-concurrency).
-Raw generation requests also require exclusive access to the primary backend's managed transition
-lock and receive 503 if it stays busy past the raw queue wait. Metadata GET/HEAD, `/api/show`,
-model-store calls (pull, push, copy, delete), and empty `keep_alive:0` unloads remain available.
-This prevents raw inference from overlapping a managed generation on that backend.
+Raw generation requests follow the same rules on the primary backend. A raw request for a runner
+Ollama already holds at a matching digest and context takes no lock. A raw load takes the load lock
+and receives 503 (`proxy busy: another model load is active`) if it stays busy past the raw queue
+wait. Metadata GET/HEAD, `/api/show`, model-store calls (pull, push, copy, delete), and empty
+`keep_alive:0` unloads remain available. Raw and managed work share the lock, the use leases and the
+memory forecast, so a raw load and a managed load on one backend never overlap.
+
+### Load lock and use lease
+
+Two different things protect a model. Its use lease lasts through generation; the backend load
+lock ends as soon as a compatible runner is observed, including before a nonstream response returns
+its headers.
+
+| | Held by | Held for | Blocks |
+|---|---|---|---|
+| **Use lease** (per model, per backend) | every managed task and raw generation for that model | the whole request, until the response ends or the caller disconnects | eviction and unload of that model. Tasks arriving for a model being unloaded wait for the unload |
+| **Load lock** (per backend) | a request that loads or reloads a runner, and a managed explicit unload | only until `/api/ps` shows the runner (or the request ends or is cancelled); managed unloads retain it through bounded confirmation | other loads and managed explicit unloads on the same backend |
+
+The rules that follow:
+
+- A request whose runner is already resident at the expected digest and context (`resident_shared`)
+  takes no load lock. It waits for no other model's load, and its lease keeps its runner from being
+  evicted while it decodes.
+- A request for a model that is not resident (`nonresident_transition_exclusive`) takes the load
+  lock, reads `/api/ps` again, and shares the runner if a same-model load just finished. Otherwise
+  it holds the lock until the runner appears, then releases it and keeps decoding under its lease.
+  Same-model arrivals queue behind that load and then share the loaded runner instead of loading
+  it again. A cancelled loader releases the lock at once.
+- A load that needs memory held by a busy model waits. Eviction only ever takes models with no
+  lease, and each time any model's use ends the waiting load plans its eviction again, so it
+  proceeds as soon as that model is idle and never earlier.
+- Managed explicit unloads take the model's lease first and then the load lock. Raw explicit
+  unloads and automatic idle eviction use the per-model lifecycle claim without the backend load
+  lock, so they can free an idle runner while another model is preparing a load. They still wait
+  for that model's users and retain ownership through bounded unload observation.
+- Observing runner readiness wakes resource waiters to re-size their demand. A same-model follower
+  waiting for a cold-load forecast can then share the runner while its first request still decodes.
+
+A cold task reads `/api/ps` six times in all: discovery, the receipt's capacity preview, the
+dispatch-time forecast, the observation before the lock, the observation after winning it, and the
+placement observation after the run. A resident task skips the post-lock read.
+
+### Degraded runner identity
+
+Reusing a runner without the load lock requires an observed identity. `/api/ps` must report the
+runner's digest and it must equal the catalog digest. The reported `context_length` must equal the
+requested `num_ctx`; when `/api/ps` omits it, the runner is shared only if the request sends no
+`num_ctx` (Ollama's default applies, so nothing explicit can contradict it). A missing digest is never
+compatible: the runner could be any build of the tag, so the task takes the load lock instead.
+
+The reason is in the task receipt at `admission.reason` (copied from `execution.runner_reuse.reason`):
+`resident_identity_unobserved`, `resident_digest_changed`, `resident_context_mismatch`,
+`resident_context_unobserved`, `unverified_load_options`, `resident_placement_mismatch`,
+`resident_context_unobserved_default_context_assumed` (shared, with the context assumed), or `null`
+for a plain match or when no runner is resident. `execution.runner_reuse.status` is `compatible`, `incompatible_or_unverified`,
+`absent` or `unavailable`.
+
+### GPU placement rule
+
+One rule decides whether a resident runner satisfies a request's placement, for managed and raw work
+alike (`PlacementRule`). Placement is judged from the same `/api/ps` entry's `size_vram`:
+
+| Rule | When | A runner satisfies it when |
+|---|---|---|
+| `any` | no placement evidence required (the default, and all raw passthrough) | always |
+| `cpu_only` | the request sets `num_gpu:0` | `size_vram` is `0` |
+| `gpu_resident` | `minPlacementEvidence: "observed"` on a GPU-assigned task | `size_vram` is above `0` |
+
+`execution.runner_reuse.placement_rule` reports the rule applied. A runner that fails it is
+`resident_placement_mismatch` and the task takes the load lock and reloads.
+
+### Raw generation whose allocation cannot be forecast
+
+A raw generation is admitted only with a memory forecast. When the model's metadata cannot be read
+in time, or the request body names a model the proxy cannot resolve (a multipart body, say) and
+could still load one, nothing could size or serialize that load. It is refused with HTTP 503,
+`Retry-After: 1` and `code: "model_forecast_unavailable"` instead of being admitted with a zero-byte
+reservation. A well-formed JSON body that names no usable model, or a model that is not installed,
+reaches Ollama's own validation and gets its 4xx.
 
 ## Understand automatic feedback
 
 With `executionPreference: "auto"`, unpinned `fastest` and `balanced` work can select a backend from
-comparable observed warm task duration. Learned hints require two eligible backends, explicit
-context, and a known execution payload-control projection. A decision-only preview cannot supply
+comparable observed warm execution profiles. Learned hints require two eligible backends, a verified
+effective context and known execution controls. Automatic context preparation is shared with execution,
+including Ollama's known default where suitable. A decision-only preview cannot supply
 that payload projection and uses capacity and policy signals. The loop is bounded:
 
 ```mermaid
@@ -283,19 +373,31 @@ flowchart LR
     P --> R["Return reason + placement receipt"]
 ```
 
-Generation compares decode nanoseconds per output token; embeddings compare total nanoseconds per
-input token. Each task/backend has one current profile window, reset when identity changes.
-Identity includes installed/resident digest, rate metric kind, explicit observed context, effective controls,
+When both candidates have qualified phase history and idle compatible runners, routing compares
+estimated setup + prefill + decode time. It requires more than a 10% predicted advantage and takes
+no speculative prefix-cache credit. Cached-token counters are required to learn generation prefill
+rates; older Ollama responses without those counters retain the existing normalized-rate fallback.
+Busy queues and cold-load costs remain unknown rather than becoming zero.
+
+The fallback compares generation decode nanoseconds per output token or embedding nanoseconds per
+input token. Each task/backend retains up to 32 exact profiles with 32 observations each; switching
+models does not erase another model's history. Identity includes installed/resident digest, rate
+metric kind, observed effective context, effective controls,
 endpoint-attributed process settings, and serial admission class. Unknown or changed evidence,
 cold transitions, and overlapping managed work supply no speed sample. A difference
 of 10% or less is treated as noise. Feedback never overrides an explicit model, session affinity,
-capability or policy guards, or the `quality` objective. The CLI persists the bounded, versioned,
-prompt-free snapshot atomically by default; `--ephemeral-feedback` opts into reset-on-restart
-behavior. Legacy aggregate fields survive loading but do not qualify learned hints. Health's
+capability or policy guards, or the `quality` objective. A measured advantage also never forces a
+cold load: when the faster backend's model is not loaded but the other backend's eligible model is,
+the loaded one runs (`resident_backend_avoids_cold_load`); warm the faster model to switch. The CLI
+keeps the bounded, versioned, prompt-free snapshot in the shared database by default; `--ephemeral`
+opts into reset-on-restart behavior. Aggregate duration counters are diagnostic only and do not
+qualify learned hints; a snapshot of another schema version is refused. Health's
 `decision_ready` describes the stored profile window; the current request must independently
 match it before routing can use its speed. `GET /_freellama/v1/health` exposes profile identity,
-profile sample counts, persistence status, and per-backend admission capacity. Physically verified
-warm completions still contribute legacy duration diagnostics even when their interval cannot
+profile sample counts, persistence status, and per-backend admission capacity. Task receipts include
+`execution.completion_planning` with candidate estimates and fallback explanations. These are snapshots,
+not capacity reservations or measurements of saved time. Physically verified
+warm completions still contribute aggregate duration diagnostics even when their interval cannot
 qualify a speed-learning sample.
 
 Configured processor intent, verified placement, and speed-learning eligibility remain separate

package/docs/DEFERRED_JOBS.md

@@ -0,0 +1,79 @@
+# Deferred jobs and recovery
+
+FreeLlama's deferred jobs keep one accepted task identity through polling, cancellation and result retrieval. A missing result or an uncertain outcome never authorizes automatic inference replay. Ollama still owns execution and physical cancellation; these records coordinate the FreeLlama worker.
+
+## Enable durable receipts
+
+Persistence is disabled by default. Set `FREELLAMA_DURABLE_JOBS=true` (or `1`) before starting the server with its file-backed SQLite store. Rust embedders can use `PlatformConfig::with_store(store).with_durable_jobs(true)`. Enabling it without a file-backed store fails server construction. `false` and `0` disable it; invalid environment values fail configuration.
+
+The journal opens an independent connection to that database, using SQLite WAL with `synchronous=FULL`. The ordinary usage/learning connection keeps its existing synchronization setting. Acceptance is committed before a new worker is spawned. This uses SQLite's durability guarantees subject to the filesystem and hardware honoring synchronization; it is not a distributed execution lease.
+
+The journal persists receipts, bounded results/errors, and keyed hashes of request identity and input (see [request hashes](#request-hashes)). It does not save executable requests for replay. Results and upstream errors may themselves contain sensitive user content. Database access and retention therefore need the same care as task output storage.
+
+## Submit, inspect and cancel
+
+`POST /_freellama/v1/tasks` with `defer:true` returns `deferred:true`, a `job` containing its `id`, and `retention` limits. Use `GET /_freellama/v1/jobs/{id}` for its result and `GET /_freellama/v1/jobs` for a metadata-only list. The list strips result/error payloads in SQLite before loading recovered records into the control plane.
+
+An embedding task may carry `nearest` (`{query?, top?, min_score?, labels?}`, with `return_embeddings` to keep the vectors). The core embeds the items (the query first, in slices of 32 pinned to one verified runner identity), ranks them by cosine similarity and stores `nearest: {mode, matches|pairs}` in place of vectors, so a deferred job's result is ranked and small. The whole input is one job and must fit the 16 MiB job input limit; without a query at most 4,000 items are compared pairwise. A malformed `nearest` is refused at submission. Through MCP, `run_task {defer:true, tasks:[...]}` submits every item as its own job and returns `{deferred:true, jobs:[{id, job}]}`; the HTTP `/tasks/batch` endpoint still refuses deferred items.
+
+`POST /_freellama/v1/jobs/{id}/cancel` requests cancellation and waits for the attached worker to settle and release local permits. It also waits for pending terminal persistence. A result already committed by the execution path retains its completed status. Acknowledged local cancellation is not proof that Ollama physically stopped accepted work.
+
+`DELETE /_freellama/v1/jobs/{id}` first settles locally owned work, then hides the record and removes its retained payload. With persistence enabled, its idempotency tombstone remains until expiry. An unattached worker with an unknown outcome cannot be removed early. A failed terminal database commit also prevents removal of its still-unfinished durable record.
+
+MCP task cancellation and managed research use addressable jobs regardless of whether the caller requested progress notifications. The MCP wrapper retains accepted job IDs after polling, cancellation or result errors; it does not replay them synchronously. Managed research and MCP task calls reject definitive pre-execution job refusals with `cancellable_task_unavailable`, `execution_outcome:"not_started"` and `retry_safe:true`. They do not switch to unaddressable synchronous execution.
+
+`cancellation_state:"unknown"` means the acknowledgement is unknown; a request may have reached the server even if its response was lost. `cancellation_requested:true` records the local attempt, not confirmed delivery or physical stop. Native MCP polling currently has no abort signal: cancellation dispatch starts immediately, but an in-flight job GET can delay the caller's error until that GET returns or its control timeout expires (30 seconds by default, configurable with `FREELLAMA_CONTROL_TIMEOUT_SECONDS`). The five-second cancellation-report cap does not interrupt that pending native poll.
+
+## Idempotency
+
+With persistence enabled, the HTTP task request may include `idempotency_key`, containing 1–128 UTF-8 bytes. It requires `defer:true`; synchronous requests and servers without durable jobs reject the field. The current MCP tool schema does not add an idempotency-key argument.
+
+Keys are hashed (HMAC-SHA256, see [request hashes](#request-hashes)) with authenticated caller identity and scoped to a server topology: listen address, primary endpoint, CPU endpoint and configured CPU model list. Keep that configuration and caller identity stable when recovering jobs. Task input is serialized and hashed after removing the key, together with the set of explicitly supplied fields. Retries must preserve both values and field presence: omission can inherit scope defaults while an explicit default overrides them. Reusing a key with changed input returns `task_job_idempotency_conflict`.
+
+A repeated key returns the original job, including a retained terminal result, before checking local registry capacity. Claiming the key and reserving its durable record is one transaction. Concurrent requests cannot both claim that key. A new durable claim rejected by local capacity is recorded as failed with `execution_outcome:"not_started"` and `retry_safe:true`; repeating the key returns that original refusal instead of starting work.
+
+After restart, only recorded terminal outcomes are recoverable as settled results. An unfinished record becomes a read-time `failed` receipt with reason `execution_outcome_unknown`, error code `task_job_outcome_unknown`, its original `job_id`, and `retry_safe:false`. It may represent an interrupted worker, a worker still attached to another server, or a submission interrupted around acceptance. No worker is resumed or recreated. If the original worker later commits a terminal record, subsequent retrieval can observe it.
+
+Idempotency applies only while the record is retained. After expiry, a key can start new work. Changing topology, caller identity or database also changes the deduplication boundary. Use one coordinating FreeLlama authority per topology; durable receipts do not coordinate model resources across servers or provide exactly-once execution.
+
+## Request hashes
+
+The request identity (caller identity plus idempotency key) and the serialized task input are stored only as HMAC-SHA256 digests, domain-separated for the two purposes. The key is a 32-byte per-installation secret in a file beside the database, `<database file name>.job-key`, never in the `task_jobs` table. Someone who can read only the database cannot confirm a guessed key or prompt by hashing it. Someone who can read the secret file can, so give it the permissions of the database itself and back both up together.
+
+The file is created on first durable use with mode `0600` on Unix. The complete key is written under a private temporary name and hard-linked into place, so concurrent starts never replace one another's key. A file with looser permissions is tightened to `0600` when read. A file that is not exactly 32 bytes fails server construction instead of being replaced, because replacing it would silently end deduplication for every retained key.
+
+Idempotency survives restart as long as the same secret file sits beside the same database. Deleting or replacing the file, or moving the database without it, changes every digest: retained keys stop matching and a retry starts new work. There is no recognition of rows written under earlier unkeyed hashes; those rows do not match and expire normally. The server-topology namespace remains a plain SHA-256 of non-secret configuration. It scopes keys and is not a secret.
+
+## Persistence states and retention
+
+Execution status and storage status are separate. Each local receipt exposes `persistence.state` and `persistence.stage`:
+
+| State | Meaning |
+|---|---|
+| `disabled` | Only process-local retention is enabled |
+| `confirmed`, stage `acceptance` | The original queued record committed; this does not confirm an eventual result |
+| `pending`, stage `terminal` | The local worker settled, but its terminal database transaction has not finished |
+| `confirmed`, stage `terminal` | The terminal transaction committed; normal payload eviction can still apply |
+| `failed`, stage `terminal` | The local result/error is available, but its commit failed; `persistence_error` gives the failure |
+
+Consumers requiring restart recovery must check terminal persistence confirmation, not only `status:"completed"`. A local result can be returned while persistence is pending. The database retains the original acceptance if terminal persistence fails, so restart recovery reports an unknown outcome rather than inventing completion.
+
+| Limit | Contract |
+|---|---|
+| Local handles | 64 total; settled handles may be evicted to admit new work |
+| Durable records | 1,024 across the entire database, including other topology namespaces and removed tombstones |
+| Durable receipt budget | 32 MiB of serialized receipts across that database |
+| Durable metadata | 16 KiB per accepted/terminal receipt excluding terminal result/error payloads |
+| Individual result or error | 2 MiB of encoded JSON |
+| Job input | 16 MiB of encoded JSON per job and 64 MiB across queued and running jobs (`task_jobs_input_budget`, retry after 5 s); executable input is not retained in the journal |
+| Terminal retention | 600 seconds after terminal commit; local retention starts when the worker settles |
+| Unfinished retention | Accepted timeout plus 600 seconds from claim |
+| Database workers | At most four blocking journal operations per process; canceled awaiters do not release a running operation's permit early |
+
+When the receipt budget is exceeded, the journal removes older terminal payloads while preserving job identity and its unexpired key. Such a receipt has `status:"failed"`, reason `retained_payload_unavailable`, `original_status`, and error `task_job_result_evicted` with `retry_safe:false` and `result_recoverable:false`. This describes unavailable retained output; `original_status` preserves whether execution completed, failed or was cancelled. Explicit removal uses `task_job_result_removed` on a repeated key. Neither operation starts a replacement inference. A still-retained local handle may continue to provide its in-memory result even if the database payload has been evicted.
+
+Once all durable slots are occupied by unexpired keys, new claims fail; keys are not discarded to admit unrelated work. Expired rows are pruned on a later claim. These are logical serialized-data limits, not exact process-RSS or SQLite-file-size bounds. WAL, indexes, allocator overhead and reusable database pages add storage overhead; expiry and removal are not secure erasure.
+
+## Validation
+
+Focused tests exercise keyed hash stability across restart, per-installation secrets, secret-file permissions and refusal of a malformed secret, concurrent same-key claims, file-backed restart recovery, unknown outcomes, FULL-sync isolation, deduplication at local capacity, terminal result delivery, refused-work accounting, bounded payload eviction, hidden tombstones, slim listing, persistence failures and cancellation during database work. They use no live models. See [testing](TESTING.md) for the full project checks.

package/docs/FEATURES.md

@@ -0,0 +1,303 @@
+# FreeLlama features and how it works
+
+Moved from the root README. For install and a first task, start with the [README](../README.md).
+
+![A brown cartoon llama stands on a pastel cloud background.](../assets/logo.jpg)
+
+**Run your agent's bulk work on local models without overloading your machine.**
+
+FreeLlama sits between an AI agent and [Ollama](https://ollama.com). For each task it picks an
+installed model that fits, admits the task against live memory, CPU and heat signals, runs it on the
+GPU or a CPU-only Ollama, and returns the result with a receipt of where and how it ran. Any Ollama
+model works.
+
+```mermaid
+flowchart LR
+    Agent["Agent or CLI"] --> FL["FreeLlama: pick model, check memory, queue"]
+    FL -->|admitted| Ollama["Ollama: GPU or CPU"]
+    FL -->|held| Wait["Bounded wait or refusal with reason"]
+    Ollama --> Receipt["Result + routing, timing, placement receipt"]
+```
+
+## Why FreeLlama
+
+Coding agents spend paid tokens and context on work a local model can do: embedding files, reading
+screenshots and scans, rewriting or summarizing many files. Calling Ollama directly, an agent cannot
+tell:
+
+- **Which model:** tag names do not show capabilities, context window or memory footprint.
+- **Whether there is room:** a new load can push the machine into swap, several agents can pile onto
+  one GPU, and a busy or hot machine slows everything.
+- **Whether it worked:** a model can repeat itself until the token cap, and nothing says whether the
+  task ran on the GPU or fell back to the CPU.
+
+FreeLlama routes from installed metadata and observed behavior, admits against host signals, stops
+runaway repetition, and returns a checkable receipt. The agent still decides what to offload and
+verifies the answer. The name refers to less metered inference, not only Meta's Llama models.
+
+## How it works
+
+### What to offload
+
+Offload bounded work whose result your agent can check; keep judgment in the agent.
+
+```mermaid
+flowchart TD
+    T{"Task for a local model?"}
+    T -->|"Embed, OCR/vision, bulk rewrite,<br/>summarize many files"| Fit["Offload"]
+    T -->|"Question about workspace files"| R["delegate_research"]
+    T -->|"Small lookup, high-stakes decision,<br/>final verification"| Keep["Keep in your agent"]
+    Fit --> N{"How many items?"}
+    N -->|"one"| RT["run_task"]
+    N -->|"many, independent"| B["run_task {tasks}"]
+    N -->|"long-running"| D["run_task {defer:true}<br/>then task_jobs"]
+```
+
+### The smart flow
+
+Every managed task passes the same gates, each deciding from installed metadata, live host signals
+and observed behavior, never from a model's name.
+
+```mermaid
+flowchart LR
+    T(["Task"]) --> R{"Route"}
+    R -->|"capabilities · context · policy<br/>· observed loops"| A{"Admit"}
+    A -->|"fair queue + host signals"| B{"Backend"}
+    B -->|"default"| GPU["Ollama on GPU"]
+    B -->|"operator-assigned helper"| CPU["Ollama on CPU"]
+    GPU --> X["Execute<br/>optional stop on repetition"]
+    CPU --> X
+    X --> Rc(["Result + receipt"])
+    A -->|"queue full or wait budget spent"| No(["Refused with reason<br/>and Retry-After"])
+```
+
+An explicit `model` always wins over routing.
+
+## Features
+
+Each feature is reachable from MCP, the CLI and the HTTP control API under `/_freellama/v1` unless a
+cell says otherwise.
+
+| Feature | What you get | MCP tool | CLI | HTTP |
+|---|---|---|---|---|
+| [Model selection](#model-selection) | The right installed model for the task, with a confidence level | `run_task {preview:true}`, `models` | `route`, `models` | `POST /routes`, `GET /models` |
+| [Resource admission](#resource-admission) | Tasks run, wait or are refused with a reason | every `run_task` receipt, `doctor {view:"status"}` | `status`, `ready` | `GET /health` (liveness), `GET /ready` (can serve now), `GET /status` |
+| [CPU and GPU backends](#cpu-and-gpu-backends) | Helper models on CPU next to GPU work, with proof of where each ran | `run_task {executionPreference}` | `serve --cpu-upstream --cpu-model` | `POST /tasks` |
+| Decisions | Label, triage or yes/no-check items with a decision model: one forward pass, one line per answer | `run_task {task:"decision", state, questions}` | `task --task decision --questions` | `POST /tasks` |
+| Ranked search and duplicates | Embed files or texts and get the closest items to a query, or the most similar pairs, as labels with scores instead of vectors; files are chunked by lines and labelled `path:Lstart-Lend` | `run_task {task:"embedding", files, nearest}` | `task --task embedding --input-file F --nearest QUERY` | — |
+| Speech | Transcribe or translate a WAV, or speak text to a WAV file, on a local audio model | `audio` | — | Ollama `/v1/audio/*` through serve |
+| Inputs by path | The server reads the files an agent names, so logs, screenshots and source files are never pasted into its output | `run_task {files}` | `task --input-file`, `--image` | — |
+| [Loop control](#loop-control) | A repeating answer stops early with every token kept | `run_task {stopOnRepetition}` | `task --stop-on-repetition` | `stop_on_repetition` |
+| [Machine awareness](#machine-awareness) | GPU, cores, memory, GPU load and heat read from the OS | `doctor` | `machine`, `doctor` | `GET /machine` |
+| [Batches, jobs and history](#batches-jobs-and-history) | Fan-out, submit now and collect later, multi-turn chat without resending history | `run_task {tasks}` / `{defer}`, `task_jobs`, `scope`, `session` | `task --defer`, `jobs`, `scope`, `session` | `/task-batches`, `/jobs`, `/scopes`, `/sessions` |
+| Warm-up and unload | Load a model ahead of use through the same memory checks; unload a resident one once its tasks finish (never loads first) | `ollama_manage {action:"warm"}`, `ollama_manage {action:"unload"}` | `warm`, `unload` | `POST /warm`, `POST /unload` |
+| Model catalog and cookbooks | Tested picks per job with their evidence, what is installed and fits now, and a recipe for each | `models {view:"catalog"}`, `freellama://docs/cookbooks/<job>` | `catalog [--job JOB]`, [docs/cookbooks](cookbooks/README.md) | — |
+| Find and install models | ollama.com and Hugging Face GGUF search, every tag with its size and memory fit, pulls only on approval | `models {view:"library"}`, `ollama_pull`, `ollama_delete` | MCP only | MCP only |
+| Job mode and prompts | Name a job and FreeLlama runs its first catalog pick that is installed and fits, with its documented prompt, options and embedding prefixes; your fields win. MCP prompts give each job's call, and `saveAs` hands an answer to the next task as a file | `run_task {job, saveAs}`, MCP prompts, [cookbook](cookbooks/jobs.md) | MCP only | MCP only |
+| Chunked sweeps | A local model checks a code tree, log or docs site chunk by chunk in the background, saves every result, re-checks flags with a stronger model and, against a baseline, reports only what changed | `sweep`, [cookbook](cookbooks/sweep.md) | MCP only | MCP only |
+| [Grounded research](#grounded-research) | Read-only repository answers with file and line citations | `delegate_research` | MCP only | MCP only |
+| [Small agent context](FEATURES.md#small-agent-context) | Compact results and paged details instead of raw dumps | every tool, `read_response` | JSON on stdout | JSON |
+| [Observe and evaluate](#observe-and-evaluate) | Live status, usage, metrics, benchmarks, routing policy from evaluations | `doctor {view:"usage"}` | `status`, `usage`, `config`, `bench-all`, `policy from-eval` | `/status`, `/usage`, `/metrics`, `/config`, `/_freellama/ui` |
+| Ollama passthrough | Any Ollama `/api/*` or `/v1/*` call, byte-preserving, with retries and a stream cap | — | `serve`, `proxy` | Ollama's own paths |
+
+### Model selection
+
+Routing filters installed models by task, capabilities (`tools`, `vision`, `embedding`, ...), context
+window, routing policy and memory fit, and reports a confidence level. `minConfidence:"medium"`
+refuses weakly evidenced routes, `objective:"quality"` requires a policy-backed model, and `model`
+pins one. A bare name resolves as in Ollama: `name:latest`, or the only installed tag of that family.
+
+Routing never ranks by a model's name or architecture and adds no model-specific request options. A
+model that loops on a task kind (at least two loops in at most twice as many runs) ranks last for that
+kind; `models` lists the evidence as `observed_loops`. See [model selection](MODEL_SELECTION.md).
+
+### Resource admission
+
+A task queues fairly, waits for budget units on its backend and for the host signals that apply to
+it, then runs or is refused with `Retry-After`.
+
+- **Budget.** Each Ollama decoding slot (`OLLAMA_NUM_PARALLEL`) is worth two units: a chat costs 2, a
+  vision task 4 (capped to the budget), an embedding 1 per four inputs. The GPU queue holds 16 tasks,
+  the CPU queue 8. A task waits at most 120 s, or `maxWaitSeconds`.
+- **Fairness.** Priority classes get 3, 2 and 1 turns per round, FIFO within a class. After six
+  bypasses the oldest blocked task reserves capacity, so nothing starves. In a live run, 4
+  interactive tasks submitted last finished before the normal and background tasks queued ahead.
+- **Fast refusals.** A full queue answers 429 at once, an expired wait 503. Both carry `Retry-After`
+  from the tasks ahead, how many of that cost run at once, and recent service time.
+
+Each host signal holds only the work it affects:
+
+| Signal | Holds |
+|---|---|
+| OS memory pressure (macOS `kern.memorystatus_vm_pressure_level`, Linux PSI) at warning | new model loads; at critical, everything |
+| Low available RAM | judged per load: a new load runs if it leaves its own reserve, else waits; a loaded model keeps serving |
+| Active swapping | new loads of 1 GiB or more, or of unknown size; a loaded model keeps serving, raw passthrough included |
+
+A load that waits for memory is told who holds it: the 503, the progress message and the route
+preview name the largest programs (`node: tsserver.js ×3 3.8 GB, rust-analyzer ×2 2.3 GB`).
+| CPU load | CPU-bound work only, never a GPU load or GPU inference |
+| Thermal throttling (macOS) | CPU-bound work and new loads |
+
+A new load must leave a reserve free: 15% of RAM, at least 1 GiB, capped at the load's own size. If
+it does not fit, idle models are unloaded first; if it still does not fit, it waits and is refused at
+its deadline, on every OS. On macOS the available figure already includes reclaimable pages
+(inactive, speculative, purgeable), so loading past it means compression and swap.
+`serve --low-memory advisory` (or `FREELLAMA_LOW_MEMORY=advisory`) restores admit-with-warning.
+While any model is loaded, `serve` also samples the host itself: under low memory, OS pressure or
+swapping it unloads the cheapest idle models (audit trigger `pressure_sweep`), never pinned,
+`keep_alive: -1` or busy ones. Because of the reserve, a load can wait while the OS still shows free
+memory ([admission controls](CLI.md#bound-admission)). Total deadlines (`timeoutSeconds`) and
+per-backend circuit breakers complete admission; see [monitoring and tuning](MONITORING.md).
+
+### Model memory and switching
+
+A loaded model answers at once, a cold one first pays its load time, and loading past what the
+machine holds swaps and stalls everything. FreeLlama decides what stays loaded:
+
+```mermaid
+flowchart TD
+    Req(["Request names a model<br/>managed task or raw Ollama call"]) --> Res{"Loaded at a<br/>compatible context?"}
+    Res -->|yes| Serve(["Serve now<br/>memory holds never apply"])
+    Res -->|no| Size["Size the load<br/>Ollama's measured footprint, else file + KV + graph"]
+    Size --> Room{"Room needed?"}
+    Room -->|"GPU VRAM short"| Plan
+    Room -->|"over the model memory budget"| Plan
+    Room -->|"host memory low or swapping"| Plan
+    Room -->|"fits"| Load
+    Plan["Pick the cheapest idle set to unload<br/>reload time x (1 + recent uses) x weight<br/>never pinned, busy or the target"] --> Unload["Unload (keep_alive 0)<br/>wait until Ollama frees it"]
+    Unload --> Load["Load and serve"]
+    Load --> Audit["Audit: load, unload, cause, switch time<br/>flag a model loaded 3+ times in 10 min"]
+```
+
+- **Model memory budget.** Loaded models on one backend may hold at most `model_memory_budget`
+  together: 75% of RAM by default, `"24GiB"`, or `"off"`. Ollama checks only what is free at that
+  moment, so without a budget loads keep landing until macOS swaps. It applies where models share
+  host memory (Apple silicon, or a CPU backend).
+- **What to unload.** The cheapest idle set that frees enough: measured reload time, scaled by recent
+  use and an optional per-model weight. Pinned models, models with queued or running work (raw
+  passthrough included) and the model being loaded are never chosen. Warms and preloads teach load
+  time but do not count as uses, so they cannot keep themselves resident.
+- **No ping-pong between backends.** A measured speed advantage never forces a cold load while the
+  other backend has an eligible model loaded.
+- **Audit.** `status` shows the budget in use, loads, unloads and evictions by cause, switch time per
+  model, and models swapped back and forth. Every MCP result carries `timing` (wall time, plus queue,
+  load, prompt and output time for a task).
+- **Learned per model.** Each model revision's output needs (reasoning included), load time, memory
+  footprint and loops are measured from its runs and saved. FreeLlama sends no output cap of its own;
+  a model that ran out of room gets twice as much next time.
+- **Preload.** `preload.startup` loads listed models at startup; with `messages` (such as a long
+  system prompt) it also evaluates them into the runner's prompt cache. Predictive preload counts
+  model-to-model switches and, once a successor is clear, loads it in the background, only when it
+  fits without unloading anything and the host is calm. Agents warm a prefix with
+  `ollama_manage {action:"warm", model, systemPrompt}`.
+- **Shared state.** Measured sizes, load times, placement feedback, usage and the audit live in one
+  `SQLite` file, `~/.octocode/freellama/freellama.sqlite`, shared by every FreeLlama process on the
+  machine.
+
+Measured switch times on an Apple M2 Pro (32 GB) through `serve`, model files in the OS file cache:
+
+| Model | Loaded size | Load (switch) | Request when loaded |
+|---|---:|---:|---:|
+| `gpt-oss:20b` | 11.9 GiB | 5.1–9.7 s | 0.9 s (short answer) |
+| `glm-ocr:latest` | 2.7 GiB | 1.5–2.0 s | 0.45 s (one image strip) |
+| `strands-decider:2b-nvfp4` | 1.4 GiB | first call 1.2–1.4 s, load included | 3–60 ms (one question) |
+| `nomic-embed-text:v1.5` | 0.3 GiB | 0.27–0.5 s | 70 ms (one input) |
+
+Unloading takes milliseconds on Apple silicon. With a 14 GiB budget and all four in rotation, loading
+glm-ocr unloaded gpt-oss for the budget, loading gpt-oss unloaded the helpers (for the budget, then
+for low host memory), and the audit flagged all four as thrashing: raise the budget, pin the big
+model, or give helpers their own backend.
+
+### Keys, budgets and other engines
+
+- **Client keys.** `clients.toml` in the home (or `--client-keys`) names one bearer token per agent
+  or tool, each with an optional `daily_tokens` budget and `admin` right. Usage counts output plus
+  prompt tokens not reused from a cache, per UTC day, for managed tasks and raw Ollama or OpenAI calls
+  (`task: "raw"`, from the counts each response reports). A spent budget refuses generation with 429
+  and `Retry-After` until the next UTC day; reads still work. Only admin keys may pull, push, copy,
+  create or delete models or reload the runtime file. The operator token is unmetered.
+- **Other engines.** `upstreams.toml` (or `--upstreams`) adds OpenAI-compatible engines. A `/v1/...`
+  request for `lmstudio/qwen3-8b` goes to upstream `lmstudio` as `qwen3-8b`; a bare name goes to
+  Ollama when Ollama has it, else to the engine that lists it. `GET /v1/models` lists all. A local
+  engine waits out host holds as a loaded model would. The caller's FreeLlama token never leaves;
+  each engine gets its own key, sent only over HTTPS or to this machine.
+
+```toml
+# ~/.octocode/freellama/clients.toml
+[[client]]
+name = "octocode"
+token_file = "keys/octocode.token"   # freellama auth-token --out keys/octocode.token
+daily_tokens = 2000000
+
+# ~/.octocode/freellama/upstreams.toml
+[[upstream]]
+name = "lmstudio"
+url = "http://127.0.0.1:1234/v1"
+```
+
+### CPU and GPU backends
+
+An optional second, CPU-only Ollama runs exact helper models, such as an embedder, so they never
+compete with GPU work: `serve --spawn-cpu-backend --cpu-model TAG` starts it on loopback, waits for it
+and stops it with `serve`, or point `--cpu-upstream` at one you run. Each backend has its own budget, queue and model-transition lock. The operator
+assigns models; agents can only state an `executionPreference`. Ask for
+`minPlacementEvidence:"observed"` when placement matters, and trust the receipt's observed placement
+over the configuration. See [CPU/GPU routing](CPU_GPU_ROUTING.md).
+
+### Loop control
+
+Some models answer correctly, then repeat the answer until their output limit (a `num_predict` cap or
+the end of their context). With the repetition stop, FreeLlama streams the answer, checks it every 16
+tokens, and stops once the text has repeated twice over at least 96 characters; closing the
+connection cancels generation in Ollama. It is on by default for free text with no output cap (JSON
+and tool calls are left whole); `stopOnRepetition:false` (CLI `--stop-on-repetition false`) turns it
+off, `true` turns it on for a capped task. Nothing is trimmed: `response` holds the full text and
+`output_check` says where the first copy ends. A loop that ran to the limit without the stop is
+flagged the same way, and routing records it either way.
+
+### Machine awareness
+
+`freellama machine`, `freellama status` and MCP `doctor` report what the OS says, without elevated
+privileges:
+
+- **Machine:** chip, logical CPUs, Apple silicon's performance/efficiency split, total memory and
+  whether it is unified, and the GPU (Apple model and cores; NVIDIA name and memory via `nvidia-smi`).
+- **Live:** available memory, OS memory pressure, CPU load, GPU busy percent (Apple, NVIDIA) and Apple
+  GPU-driver memory. macOS heat is the thermal pressure level (`nominal` to `sleeping`, throttled from
+  `heavy`) plus the `pmset` CPU speed limit on Intel Macs; Linux and Windows report it as unknown.
+- **Placement:** fully GPU, fully CPU or split, as Ollama reports each loaded model.
+
+### Batches, jobs and history
+
+| Need | Use |
+|---|---|
+| Many independent items | `run_task {tasks, maxParallelism}` with up to 64 `{id, independent:true, task}` items |
+| Submit now, collect later | `run_task {defer:true}` returns `job.id`, then `task_jobs {action:"get", jobId}` |
+| Multi-turn chat without resending history | `scope {action:"create"}`, then `run_task {scopeId, scopeRevision, prompt}` |
+| Keep related calls on one model | `session {action:"create"}`, then `run_task {sessionId}` |
+| Hot model before latency-sensitive work | `ollama_manage {action:"warm", model, contextTokens}` |
+
+Scopes, sessions and deferred jobs live in memory and are lost when the service restarts. See
+[scopes and warming](SCOPES_AND_WARMING.md).
+
+### Grounded research
+
+`delegate_research {question, workspacePath}` gives a local model bounded, read-only search and read
+tools over an allowed workspace and returns an answer, file and line citations, and an independently
+computed verdict. Discard `escalate` results and verify the citations on the rest.
+
+### Small agent context
+
+`run_task` and `delegate_research` return a compact view unless `receipt:"full"`; every tool saves its
+whole result to a `response_file`, which `read_response` pages by section. Embedding vectors are
+omitted unless `returnEmbeddings:true`. The optional `telemetry.externalEquivalent` estimates avoided
+API cost from a rate card you configure. See [token economics](../packages/mcp/README.md#token-economics).
+
+### Observe and evaluate
+
+- **Live status:** queues, memory holds, loaded models, circuit breakers and usage per day and model,
+  from `doctor {view:"status"}`, `freellama status` or `/_freellama/ui`.
+- **Evaluation:** `bench-all` benchmarks installed models; `policy from-eval` writes a routing policy
+  from quality-evaluation results. See [model selection](MODEL_SELECTION.md).
+- **Ollama compatible:** `/api/*` and `/v1/*` reach Ollama byte-for-byte, streaming included;
+  FreeLlama never rewrites raw bodies. See [Ollama sidecar](ARCHITECTURE.md#ollama-sidecar).