@octocodeai/freellama-mcp-server 0.7.0 → 0.10.0

package/docs/ARCHITECTURE.md

@@ -45,10 +45,18 @@ flowchart LR
     F -->|"raw /api and /v1"| G["Primary Ollama"]
     F -->|"managed primary task"| G
     F -->|"managed assigned task"| U["Optional CPU Ollama"]
+    F -->|"/v1 for an engine's model"| O["Optional OpenAI-compatible engines<br/>LM Studio, llama.cpp, vLLM, cloud"]
     G --> E["Metal, MLX, CUDA, ROCm, Vulkan, or CPU runner"]
     U --> E2["CPU runner"]
 ```
 
+Bearer checks run first: the operator token, or a client key whose budget and rights are checked
+and whose name tags the request (`proxy::Caller`). Raw responses stream through unchanged while
+`proxy/usage.rs` reads the token counts and `load_duration` they report; the platform's raw hooks
+turn them into usage-ledger rows, audit events and load-time samples. `proxy/upstreams.rs` routes
+OpenAI-compatible requests by model to other engines. `platform/preload.rs` runs startup loads and
+learns model successors (`next:<model>` facts) for predicted loads.
+
 FreeLlama does not make an individual model decode faster. It can improve workload completion by
 avoiding unsuitable routes, reducing avoidable transitions, refusing unsupported work early, and
 overlapping independent CPU and GPU model requests when the machine benefits from that layout.
@@ -159,7 +167,8 @@ The control API uses snake_case JSON. A managed task body is the routing fields
 installed tag of that family; an unknown or ambiguous name is refused with the installed candidates.
 `messages` (Ollama message objects, images included) replaces `prompt`; `input` serves embeddings;
 `context_tokens` owns `num_ctx` and placement owns `num_gpu`, so neither is accepted inside
-`request_options.options`. `stop_on_repetition: true` streams the chat from Ollama and stops as soon
+`request_options.options`. `stop_on_repetition` (on by default when no `num_predict` caps the output,
+`false` to turn it off) streams the chat from Ollama and stops as soon
 as the output repeats its unit twice after the first copy over 96+ characters: the connection
 closes, which cancels generation upstream, and everything generated is kept with
 `done_reason: "repetition"`. An Ollama "token repeat limit" abort is kept the same way
@@ -179,9 +188,9 @@ A batch nests whole task bodies:
 
 The machine profile is host-derived rather than model-name-derived. It reports total physical
 memory on macOS (`sysctl`), Linux (`/proc`), and Windows (CIM through PowerShell), along with CPU, OS,
-architecture, and disk. `memory_bytes` is portable host RAM. `unified_memory_bytes` is populated
-only when host and accelerator memory are known to share a pool; it must not be treated as discrete
-GPU VRAM. Ollama's `/api/ps` remains the placement authority.
+architecture, and disk. `memory_bytes` is portable host RAM. `memory_kind` is `unified` only
+when host and accelerator memory are known to share a pool; host RAM must not be treated as
+discrete GPU VRAM. Ollama's `/api/ps` remains the placement authority.
 
 All other paths use the compatibility proxy. It buffers request bodies up to 64 MB so a retry can
 resend identical bytes, then streams the upstream response. It removes hop-by-hop headers, adds
@@ -314,13 +323,13 @@ flowchart TD
     K --> A{"Acquire selected backend's<br/>3:2:1 fair weighted budget"}
     A -->|"No"| E503["429 queue full or<br/>503 wait timeout"]
     A -->|"Yes"| B{"Execution backend"}
-    B -->|"Primary"| GL{"GPU model resident<br/>under transition lock?"}
+    B -->|"Primary"| GL{"GPU runner resident<br/>at expected identity?"}
     B -->|"CPU assignment"| CO["Add num_gpu: 0"]
-    CO --> CL{"CPU model resident<br/>under transition lock?"}
+    CO --> CL{"CPU runner resident<br/>at expected identity?"}
     GL -->|"Yes"| GR["GPU shared lock"]
-    GL -->|"No"| GX["GPU exclusive transition lock"]
+    GL -->|"No"| GX["GPU load lock until runner observed"]
     CL -->|"Yes"| CR["CPU shared lock"]
-    CL -->|"No"| CX["CPU exclusive transition lock"]
+    CL -->|"No"| CX["CPU load lock until runner observed"]
     GR --> OQ["Send to selected Ollama queue"]
     GX --> OQ
     CR --> OQ
@@ -357,6 +366,15 @@ After six capacity bypasses, the scheduler reserves released capacity for the ol
 so a stream of small requests cannot indefinitely starve a larger request. Cancellation releases
 that reservation. Admission, resource waiting, and transition locking share one queue deadline.
 
+Within the class the round robin picks, admission prefers model affinity: the oldest fitting waiter
+whose selected model has a task running on that backend, or was the backend's most recently
+admitted model, goes ahead of older waiters for other models, which would otherwise make Ollama
+load or swap a runner. Affinity never crosses priority classes, never admits a waiter that does not
+fit, and never overrides the capacity reservation. It is bounded by aging: a waiter passed over four
+times, or queued for two seconds, restores FIFO order for its class. Affinity uses only admission's
+own record of admitted work, never an Ollama call. Each pool's receipt counts
+`affinity_promotions`; health reports the bounds under `admission.priority_fairness.model_affinity`.
+
 A task waiting for host resources returns its execution slot after two seconds and continues
 waiting within the same deadline. It remains counted against the backend's bounded queue, so
 memory pressure cannot create an invisible backlog. Health reports slot and resource waiters
@@ -365,9 +383,11 @@ new arrivals and a smaller reloaded queue limit cannot displace accepted work. E
 prevent admission even when an execution slot is free. Cancellation removes either waiter, and releasing a memory reservation wakes resource
 waiters immediately; host telemetry still obeys the sampling and recovery policy below.
 
-FreeLlama acquires the admission slot before a transition lock to avoid deadlock. A model marked
-resident during discovery is checked again while the shared transition lock is held; stale or
-unavailable residency falls back to the exclusive transition path. Session affinity is bound only
+FreeLlama acquires the admission slot before the backend load lock to avoid deadlock. A model marked
+resident during discovery is observed again before dispatch; a stale or unavailable listing means a
+load, which takes the load lock and observes once more after winning it (a same-model load that
+finished meanwhile is then shared).
+See [Load lock and use lease](CPU_GPU_ROUTING.md#load-lock-and-use-lease). Session affinity is bound only
 after successful upstream execution, so refused and failed tasks cannot change later routing.
 An explicit `done:false` chat response or a reported upstream error is a failure, even with HTTP 200.
 It cannot train runtime feedback or bind affinity. Completed failed exchanges retain the raw response
@@ -385,7 +405,10 @@ The K/V preflight derives single-sequence F16 K+V bytes-per-token for any archit
 metadata declares the block count and head counts. Explicit key/value dimensions take precedence
 over embedding length divided by heads; sliding-window, shared-KV, recurrent, cross-attention, and
 unknown layouts report unknown. F16 is advisory because quantized KV can
-use less memory. The coarse model-file budget refuses files over 80% of local CPU/unified RAM only
+use less memory: reservations scale it by each backend's `OLLAMA_KV_CACHE_TYPE` unless
+`OLLAMA_FLASH_ATTENTION` forces flash attention off (unset is Ollama's `auto`), and keep Ollama's
+F16 fit-check figure beside the quantized one to flag a load Ollama may evict or offload for
+([KV cache and flash attention](TUNING.md#kv-cache-and-flash-attention)). The coarse model-file budget refuses files over 80% of local CPU/unified RAM only
 for loopback backends. Forecasts use each backend's independently attributed parallelism and KV
 cache settings where process inspection is available, with explicit fallback sources otherwise.
 Process observations refresh on requests after five seconds; measurements made under different
@@ -394,24 +417,29 @@ authority. Configured settings alone do not prove physical processor placement.
 
 The shared [resource governor](https://github.com/bgauryy/FreeLlama/blob/main/packages/rust-core/src/platform/resources/mod.rs) samples local
 available-memory estimates, load per logical CPU, OS memory pressure, active swap-out growth, and
-thermal throttling when reported. OS memory pressure holds every local demand; active swap-out
-growth holds new loads; the CPU-load hold applies only to CPU-assigned work (saturated cores slow
+thermal throttling when reported. OS memory pressure holds new loads, and resident runners only
+at the critical level; active swap-out growth holds new loads (a raw passthrough request counts as
+resident when its `model` is in `/api/ps` at a compatible context); the CPU-load hold applies only to CPU-assigned work (saturated cores slow
 nothing on the GPU, so raw passthrough, which cannot know residency, is not refused for it), and to
 all work on a host whose sampler sees no accelerator; the
 thermal hold applies to CPU-assigned work and to cold loads (a new heat source) but not to a bounded
 request on a runner already resident on the GPU (`ResourceDemand::is_held_by`; the assessment's
 `held_by` lists the holds that applied). Low available RAM (below the larger of 15% or 1 GiB, clearing after two
-healthy samples above the larger of 20% or 2 GiB) depends on the platform (`ResourcePolicy::low_memory`).
+healthy samples above the larger of 20% or 2 GiB) holds nothing host-wide: each load is judged by the capacity
+check below, which follows `ResourcePolicy::low_memory`. Active swapping holds new loads of 1 GiB or more, or of
+unknown size; a smaller load of known size still has to fit its reserve. A held or refused load names the largest
+memory users (`resources::memory_holders`, one memory-sorted `ps` sample cached for 10 s).
 A new load must leave that reserve free, but never more than its own size above the 1 GiB floor.
-On macOS (`advisory`) the estimate undercounts what the OS can reclaim, so a load that does not cover
-it first unloads idle runners and then dispatches with a `memory_advisory`. On Linux and other
-platforms (`refuse`) a new load that does not fit first unloads idle runners and then waits
+By default (`refuse`, every platform) a new load that does not fit first unloads idle runners and then waits
 with `held_insufficient_capacity` until a permit release or a fresh sample frees capacity, or refuses
 with 503 at its deadline: Linux `MemAvailable` already includes reclaimable cache and is the hard cap
 in a container, so over-admitting invites the OOM killer. Already-resident runners add no memory and
 are admitted in both modes, also during an active-swapping hold, so a large load never locks out
 reuse of its own runner; swapping still holds new loads. Historical swap occupancy alone is not pressure. Samples are cached for two
 seconds; unavailable signals stay unknown. Remote backends bypass this host gate.
+On macOS the available figure adds bounded reclaimable pages (inactive, speculative, purgeable), so a
+load past it pushes the host into compression and swap. `--low-memory advisory` (`FREELLAMA_LOW_MEMORY`)
+admits such a load with a `memory_advisory` instead, after unloading idle runners.
 
 Each pressure signal recovers independently after two fresh samples that meet its recovery
 threshold. A recovered CPU-load signal therefore clears even while RAM remains low.
@@ -424,14 +452,15 @@ platforms have no host collector, so they default to `best_effort` and admit wit
 Unsupported Linux collector signals remain explicitly unavailable rather than being fabricated.
 The pure `ResourceSnapshot::assess_capacity` calculation is shared by admission and previews.
 Previews use the same footprint lookup, subtract existing reservations, and expose both queue and
-resource readiness. Insufficient model headroom reports `runnable_low_memory_estimate` and admits
-no siblings on macOS; elsewhere it reports `held_insufficient_capacity`. Preview capacity is advisory and conservative, never a reservation.
+resource readiness. Insufficient model headroom reports `held_insufficient_capacity`, or
+`runnable_low_memory_estimate` under `--low-memory advisory`. Preview capacity is advisory and conservative, never a reservation.
 
-Managed CPU/unified-memory tasks reserve estimated additional model-file and known F16 KV bytes
+Managed CPU/unified-memory tasks reserve estimated additional model-file and known KV bytes
 across both backends. Verified runner footprints replace that estimate only for the same backend,
 immutable model digest, and equal-or-larger observed context. Exact resident digest/context matches
-avoid double-counting existing allocations. The reservation is rechecked after transition locking
-without waiting while holding the backend lock; an increased requirement that no longer fits the
+avoid double-counting existing allocations. The reservation is rechecked after winning the load lock
+(forecast from the residency listing that decided the transition) without waiting while holding
+that lock; an increased requirement that no longer fits the
 estimate dispatches with a `memory_advisory`, and a pressure hold refuses with 503. Permits release on completion or cancellation. On discrete-GPU hosts, observed free
 VRAM discounts the estimated host-memory requirement by the bytes the GPU can hold. Missing VRAM
 telemetry reserves the full estimated footprint in host RAM. The governor cannot constrain external
@@ -442,24 +471,41 @@ that covers the UTF-8 prompt/schema estimate, output reserve, and 512-token temp
 by the selected model's advertised limit. Larger requests require explicit context or compaction.
 Multimodal and embedding requests retain their profiles. The execution receipt exposes
 `context_sizing`; previews contain routing fields only and therefore retain task defaults.
-Managed chat sends `truncate:false` and `shift:false` so supporting Ollama backends refuse overflow
-instead of silently losing earlier instructions. Managed embeddings send `truncate:false` too.
-Only the dedicated CPU backend supplies a thread default: half the logical CPU count on other
-platforms, while macOS leaves Ollama to select its performance cores. `FREELLAMA_CPU_NUM_THREAD`
-overrides that default; explicit request options take precedence. The primary backend leaves thread
-selection to Ollama to avoid changing the options of a shared runner. `runtime_options` reports
-forwarded values; runner support is not assumed.
-
-Resident tasks share a backend lock. Nonresident tasks take that backend's exclusive lock so a cold
-load cannot race another managed task on the same server. CPU and GPU backends have separate locks
-and admission pools, so they can progress independently.
-
-In `serve`, raw generation requests also take the primary backend's exclusive transition lock,
-returning 503 when it stays busy past the raw queue wait. The response body holds this guard through
-EOF or disconnect. Metadata GET/HEAD, `/api/show`, model-store calls (pull, push, copy, delete, and
+Managed chat sends `truncate:false`, so supporting Ollama backends refuse an oversized prompt
+instead of silently losing earlier instructions, and by default `shift:false`, so a generation that
+reaches the context limit stops instead of discarding its earliest tokens. Ollama reloads a runner
+whose launch-time context-shift setting differs from a request's, so the runtime setting
+`context_shift = "ollama_default"` omits `shift` to share runners with raw clients that leave it at
+Ollama's default; `truncate:false` stays in both modes. Managed embeddings send `truncate:false` too.
+Thread selection defaults to Ollama on every platform. `FREELLAMA_CPU_NUM_THREAD` explicitly
+overrides the dedicated CPU backend; explicit request options take precedence. FreeLlama does not
+guess physical cores from half the logical CPU count. `runtime_options` reports forwarded values;
+runner support is not assumed. Overrides of load controls absent from `/api/ps` take the conservative
+transition path rather than claiming verified runner compatibility.
+
+A compatible resident runner needs no backend lock: nonempty model digest and effective context must
+match the observed runner (a request that sends no `num_ctx` also accepts a runner whose context
+`/api/ps` omits). Name-only residency is insufficient, and `admission.reason` names the degraded
+identity. Every task holds a per-model use lease for its whole request, which keeps its runner from
+being evicted or unloaded. Forecast release invalidates the
+host sample generation so an older available-memory observation cannot spend released capacity.
+A task that must load or reload a runner takes that backend's load lock until the runner is
+observed in `/api/ps`, so loads and unloads on one server are serialized and same-model arrivals
+share the loaded runner; decoding on another model does not wait for it. CPU and GPU backends have
+separate locks and admission pools, so they can progress independently.
+
+In `serve`, raw generation requests follow the same rules on the primary backend: a resident match
+takes no lock, and a load takes the load lock and returns 503 when it stays busy past the raw queue
+wait. The response body holds the model's use lease through EOF or disconnect. A generation whose
+allocation cannot be forecast is refused with 503 `model_forecast_unavailable`. Metadata GET/HEAD, `/api/show`, model-store calls (pull, push, copy, delete, and
 blob uploads), and empty `keep_alive:0` unload requests take no lock and bypass pressure. Raw
 inference waits at most two seconds for host pressure to clear, then returns 503 with `Retry-After`. Raw request-body collection has the configured request deadline too; an incomplete upload
-returns HTTP 408 and releases its guards without contacting Ollama. Admission receipts report
+returns HTTP 408 and releases its guards without contacting Ollama. Before any body is read, raw
+and control-API requests reserve its size (`Content-Length`, or the request limit when none is
+declared) from one process-wide buffered-body budget, `FREELLAMA_MAX_BUFFERED_BYTES` (256 MiB by
+default), and return it when the buffered body is released. A request that finds the budget full
+waits up to the raw queue wait (control API: the managed queue wait), then gets 503
+`request_body_budget_exhausted` with `Retry-After`; empty bodies reserve nothing. Admission receipts report
 `queue_wait_ms`, `resource_wait_ms`, `transition_wait_ms`, and
 their sum as `total_wait_ms`. Health and route receipts expose sampled pressure and reservations.
 
@@ -496,14 +542,24 @@ flowchart TD
 
 The hint never changes the operator's CPU assignment. If the preferred backend has no eligible
 model, routing falls back and returns `preference_satisfied: false`. Feedback is task-specific,
-bounded, versioned, and prompt-free. The CLI persists it with atomic file replacement by default;
-restart-reset behavior is available only through the explicit `--ephemeral-feedback` mode. It
+bounded, versioned, and prompt-free. The CLI keeps it in the shared database by default;
+restart-reset behavior is available only through the explicit `--ephemeral` mode. It
 records normalized latency only for already-resident, physically verified tasks so prompt size and
 cold-load cost do not poison the comparison. It requires more than a 10% advantage; capacity can
 break a tie before the sample threshold. Quality routing, explicit model selection, and session
 affinity do not follow the latency loop.
 
-The same selection function supplies preview receipts and managed task execution. This keeps advertised placement and actual request routing aligned.
+Payload context fit is checked before automatic model ranking. Objective/quality gates apply to the
+complete eligible catalog before it is split by backend. Qualified idle resident candidates can use
+setup + prefill + decode estimates instead of a decode-rate-only comparison; cold and busy candidates
+retain the conservative fallback. Exact-profile history is bounded to 32 profiles × 32 observations
+per task/backend, with idempotent process snapshot merging.
+
+The same selection function supplies preview receipts and managed task execution. Preview cannot
+estimate an execution payload it does not accept; its placement is a non-reserving snapshot.
+Execution receipts expose `completion_planning`, including estimated costs and whether the comparison
+influenced selection. Ollama's cached-token counts inform prefill-rate learning; FreeLlama does not
+export or manage KV tensors.
 
 Raw proxy traffic always uses the primary upstream. For setup and runtime verification, see
 [Run models on CPU and GPU](CPU_GPU_ROUTING.md).
@@ -600,18 +656,18 @@ FreeLlama keeps these values in memory:
 - Independent GPU and CPU priority-fair weighted admission pools
 - Per-task normalized warm latency and queue feedback for each backend, restored from an optional
   versioned atomic snapshot
-- Separate CPU and GPU transition locks
+- Per-model use leases and separate CPU and GPU backend load locks
 
 Sessions are bounded (default 1024) and expire after idle time (default one hour); they store only
 model affinity, never prompts or Ollama KV. Restarting `freellama serve` clears sessions and the catalog cache. Persisted placement feedback is
-reloaded; `--ephemeral-feedback` intentionally resets it. Ollama owns model residency, so loaded
+reloaded; `--ephemeral` intentionally resets it. Ollama owns model residency, so loaded
 models survive a FreeLlama restart. Scope histories and their revisions do not survive restart.
 They store messages independently from affinity and append only a successful generation at the
 expected revision. Warming uses ordinary admission and fit checks; omitted residency controls use
 a finite measured policy. See [Scope history and model warming](SCOPES_AND_WARMING.md) for the
 public contract and configured limits.
 
-In `serve`, raw generation traffic shares the primary transition lock and receives
+In `serve`, raw generation traffic shares the primary load lock and use leases and receives
 host-pressure gating, while managed tasks also reserve forecast memory. Direct Ollama callers bypass
 these controls. FreeLlama cannot preempt inference or constrain other processes. A configured but
 unreachable CPU backend makes managed catalog discovery fail closed; raw primary passthrough remains

package/docs/BUILD_AND_NOTES.md

@@ -0,0 +1,31 @@
+# Build from source and operating notes
+
+Moved from the root README.
+
+## Good to know
+
+- Local inference still costs memory, power and time; offloading shrinks the calling agent's
+  context, not the total work.
+- Searches and recommendations never download; a pull needs approval for one exact tag and size.
+- Managed tasks return one result; raw Ollama passthrough keeps streaming.
+- The MCP server saves every tool result except `read_response`'s, prompts included, to
+  `~/.octocode/freellama/responses/` (owner-only, oldest dropped past 1 GiB). `FREELLAMA_RESPONSE_STORE_BYTES=0` keeps nothing on disk.
+- Without CA certificates, as in slim containers, FreeLlama still serves plain-HTTP Ollama. Install
+  `ca-certificates` for HTTPS upstreams.
+- The service binds to loopback. For remote access, see [Production](PRODUCTION.md).
+- Port 11435 is also Ollaya's default ([Ollaya API](https://github.com/ollaya-dev/ollaya/blob/main/docs/api.md)).
+  To run both, use `serve --listen` on another port and set the MCP server's
+  `FREELLAMA_SERVE_ENDPOINT` to match.
+
+## Build from source
+
+Requires Rust 1.89+ (`rust-toolchain.toml` pins the compiler) and Node.js 22.12+ with Corepack.
+
+```bash
+corepack enable && yarn install && yarn build
+./target/release/freellama serve
+node packages/mcp/dist/index.js        # MCP server from this checkout
+```
+
+npm releases can lag this checkout. [Testing](TESTING.md) describes the checks; the release gate
+is `yarn verify:production`.

package/docs/CLI.md

@@ -26,20 +26,23 @@ flowchart TD
 | `serve` | Run the control plane and Ollama-compatible proxy | Starts it |
 | `auth-token` | Create a new mode-0600 bearer-token file without printing the secret | No |
 | `models` | List installed models, capabilities, residency, and evidence | Yes |
-| `machine` | Print portable host RAM, CPU, OS, architecture, disk, and the local Ollama endpoint | Yes |
+| `catalog` | Tested model picks per job, with evidence, what is installed, and what fits now (`--job`); without `serve`, `installed` and `fits_now` stay null | No |
+| `machine` | Print the machine execution profile (CPU, memory, GPU, backends) visible to the platform | Yes |
 | `status` | Live queues (also as a page at `http://127.0.0.1:11435/_freellama/ui`), current and adaptive limits, circuit breakers, loaded models, host memory, Ollama's effective settings, today's usage | Yes |
 | `usage` | Task and token totals per day and per model (`--days`, default 7) | Yes |
 | `config` | Effective runtime settings and the source of each; `--reload` re-reads the runtime file | Yes |
 | `session` | Create model affinity for related tasks | Yes |
 | `scope` | Create, inspect, fork, or delete bounded message history | Yes |
-| `warm` | Warm an installed model through managed admission | Yes |
+| `ready` | Exit 0 once `serve` can run managed work (Ollama answers and its catalog reads); otherwise print what it waits for and exit non-zero | Yes |
+| `warm` | Load an installed model (chat or embedding) through managed admission | Yes |
+| `unload` | Unload a resident model once its running tasks finish; a model that is not loaded is reported (`status: not_resident`), never loaded first, and still exits 0 (idempotent, like `rm -f`) | Yes |
 | `route` | Choose a model and request profile without executing it | Yes |
 | `task` | Route and execute one nonstreaming task | Yes |
 | `jobs` | List, inspect, cancel, or remove deferred tasks and warming operations | Yes |
 | `proxy` | Run only the Ollama-compatible retry and telemetry sidecar | No |
 | `bench-all` | Measure installed models by capability group | No |
 | `policy from-eval` | Write a task policy (`platform.toml`) from a quality-evaluation aggregate and the installed tags | No |
-| `doctor` | Inspect Ollama, hardware, versions, and effective settings | No |
+| `doctor` | Verify that an Ollama server exposes the required diagnostic APIs and report the hardware, versions, and settings it found | No |
 
 Run `npx @octocodeai/freellama <command> --help` for every accepted flag and enum value. The executable's help
 is authoritative; this page explains how the commands fit together.
@@ -58,10 +61,18 @@ file must be a regular file, not a symlink, and hold one token of at least 32 by
 whitespace is trimmed. On Unix it must also be owner-only (mode 0600). Windows file permissions
 (ACLs) are not checked, so protect the file with your user profile's permissions there.
 
-`serve` persists bounded adaptive feedback under the platform data directory by default. Override
-the path with `--feedback-file`; use `--ephemeral-feedback` only for disposable runs. See the
+`serve` keeps what it learns (runner sizes, placement feedback, usage and the residency audit) in
+`~/.octocode/freellama/freellama.sqlite`, shared with every other FreeLlama process on the machine.
+`--home` (or `FREELLAMA_HOME`) moves the home; use `--ephemeral` only for disposable runs. See the
 [production runbook](PRODUCTION.md) for token generation, state files, and ingress requirements.
 
+`--client-keys` (or `FREELLAMA_CLIENT_KEYS`, default `clients.toml` in the home) adds named client
+keys with daily token budgets, and `--upstreams` (or `FREELLAMA_UPSTREAMS`, default
+`upstreams.toml` in the home) adds OpenAI-compatible engines that `/v1/*` requests route to by
+model; see [client keys and other engines](PRODUCTION.md#client-keys). The runtime file's
+`[preload]` table loads models at startup and predicts the next one; see
+[Preload](SCOPES_AND_WARMING.md#preload).
+
 `serve` exposes both managed routes under `/_freellama/v1/*` and byte-preserving Ollama routes under
 `/api/*` and `/v1/*`. Use `proxy` when you need only the latter. See
 [Architecture](ARCHITECTURE.md) for the request flow.
@@ -78,8 +89,17 @@ npx @octocodeai/freellama serve \
   --cpu-model nomic-embed-text:latest
 ```
 
-FreeLlama rejects a CPU upstream with no assignments, `--cpu-model` without `--cpu-upstream`, and a
-CPU upstream that is the same Ollama endpoint as `--upstream`, including loopback aliases on one port.
+Or let `serve` start and own the CPU process (stopped when `serve` exits; a port that already answers
+is refused):
+
+```bash
+npx @octocodeai/freellama serve --spawn-cpu-backend --cpu-model nomic-embed-text:latest \
+  [--cpu-backend-port 11436] [--ollama-binary /path/to/ollama]
+```
+
+FreeLlama rejects a CPU upstream with no assignments, `--cpu-model` without `--cpu-upstream` or
+`--spawn-cpu-backend`, both of those together, and a CPU upstream that is the same Ollama endpoint as
+`--upstream`, including loopback aliases on one port.
 It also sets `options.num_gpu=0` on CPU-assigned managed requests.
 Read [CPU and GPU model routing](CPU_GPU_ROUTING.md) before deploying this layout.
 
@@ -129,7 +149,7 @@ over the cap waits up to `--raw-queue-wait-seconds` (default 10) and is then ref
 All of these, plus pinned models, idle eviction, adaptive concurrency, the circuit breaker, and
 context sizing, can also live in a runtime file (`--runtime-config`) that `serve` re-reads when it
 changes. See [Monitoring and live tuning](MONITORING.md). The standalone
-`proxy` command keeps its cap opt-in for compatibility.
+`proxy` command leaves its cap off unless set.
 
 The local host-pressure gate reports low available RAM below 15% and clears it above 20% after two
 healthy samples; minimum reserves are 1 GiB and 2 GiB, respectively. Tune the percentages with
@@ -137,18 +157,21 @@ healthy samples; minimum reserves are 1 GiB and 2 GiB, respectively. Tune the pe
 lower than the resume value, and neither can exceed 100. Health reports the policy, sampled signals,
 unknown measurements, and reserved memory. These flags do not change Ollama or OS settings.
 
-On macOS, low available RAM is advisory and never refuses a task. Effective available RAM is free
-memory plus a bounded reclaimable estimate, so it can be far lower than what the OS can actually
-reclaim. When it does not cover already-reserved bytes, the task's estimated bytes, and the reserve,
-FreeLlama first unloads idle unpinned runners, then dispatches anyway: the receipt reports
+When effective available RAM does not cover already-reserved bytes, the task's estimated bytes and the
+reserve, FreeLlama first unloads idle unpinned runners, then holds the new load with
+`held_insufficient_capacity` until capacity returns, refusing with 503 at its deadline. This is the
+default on every platform: Linux and Windows available RAM already counts reclaimable cache (and is
+the hard cap in a container), and on macOS effective available RAM is free memory plus a bounded
+reclaimable estimate. A load already resident is never held by it. `--low-memory advisory` (or
+`FREELLAMA_LOW_MEMORY=advisory`) instead dispatches such a load: the receipt reports
 `status: "admitted_low_memory_estimate"` and `assessment.memory_advisory: "below_reserve"` with
-`required_with_reserve_bytes` and `effective_available_bytes`. Ollama and the OS decide whether the
-load fits. On Linux, Windows, and other platforms, where available RAM already counts reclaimable
-cache (and is the hard cap in a container), the same shortfall unloads idle runners and then holds the
-new load with `held_insufficient_capacity` until capacity returns, refusing with 503 at its deadline.
-OS memory pressure holds all work where the OS reports it (macOS, and Linux when the kernel
-exposes `/proc/pressure/memory`; Windows has no host collector yet). Active swapping and low RAM hold new loads only:
-an already-resident runner adds no memory and stays admitted. The CPU-load hold (one-minute load at
+`required_with_reserve_bytes` and `effective_available_bytes`, and Ollama and the OS decide whether
+it fits.
+OS memory pressure holds new loads where the OS reports it (macOS, and Linux when the kernel
+exposes `/proc/pressure/memory`; Windows has no host collector yet), and holds already-resident
+runners too only at the critical level: a warning is what a large load itself raises. Active
+swapping and low RAM hold new loads only: an already-resident runner adds no memory and stays
+admitted, for raw passthrough requests too when the request's `model` is in Ollama's `/api/ps`. The CPU-load hold (one-minute load at
 or above 1.5 per logical CPU) applies only to CPU-assigned work, which decodes on those cores; GPU
 work, loaded or not, proceeds, so another program's build never blocks a GPU model. On a host where
 the sampler sees no accelerator (no GPU device or GPU memory), every model runs on the CPU, so the
@@ -189,7 +212,8 @@ Start with the read-only guided receipt, then inspect available models. `init` n
 it stops at exact-tag approval and prints the next prerequisite, serve, managed-task, and MCP steps.
 Its `status` is `blocked` when Ollama is unavailable, `setup_required` when the managed service or
 installed models are missing, and `ready` when all three are present. Separate `readiness` flags
-identify the missing part. Readiness does not qualify a model for a particular task; use a route
+identify the missing part. `init` prints the receipt in every case but exits non-zero only when the
+status is `blocked`; `setup_required` and `ready` exit 0. Readiness does not qualify a model for a particular task; use a route
 preview to inspect eligibility and rejection reasons.
 
 ```bash
@@ -199,6 +223,14 @@ npx @octocodeai/freellama models
 npx @octocodeai/freellama machine
 ```
 
+`serve` answers `GET /_freellama/v1/health` as soon as it listens (there is no bare `/health` route). Wait on `ready` instead before sending work from a
+script; it returns 503 with `waiting_for: "ollama" | "cpu_backend" | "catalog"` until it can serve:
+
+```bash
+until npx @octocodeai/freellama ready >/dev/null 2>&1; do sleep 1; done
+npx @octocodeai/freellama unload --model nomic-embed-text:v1.5
+```
+
 Preview a deterministic decision without spending generation tokens:
 
 ```bash
@@ -209,13 +241,24 @@ npx @octocodeai/freellama route \
   --required-capability vision
 ```
 
-Task kinds are `completion`, `coding`, `code-repair`, `tools`, `browser`, `vision`, `embedding`, and
-`long-context`. Objectives are `fastest`, `balanced`, and `quality`. `fastest` can use capability
+Task kinds are `completion`, `coding`, `code-repair`, `tools`, `browser`, `vision`, `embedding`,
+`long-context`, and `decision` (hyphens and underscores are interchangeable). Objectives are `fastest`, `balanced`, and `quality`. `fastest` can use capability
 and local benchmark evidence alone. Default `balanced` prefers eligible task-policy candidates;
 without them it falls back to capability/context-compatible models and reports low confidence
 and missing quality evidence. `quality` requires a policy unless you supply an explicit model.
 An explicit minimum-confidence gate still refuses insufficient evidence in every objective.
 
+`route` ranks by task eligibility and quality evidence, not by free memory, so under memory pressure
+it can pick a large model whose receipt says `dispatch_readiness: held_host_pressure`. That is
+deliberate: the route is a deterministic preview from the catalog, and host memory is checked later
+at admission. When admission refuses for memory, the error states `needs X GB incl. Y GB reserve,
+Z GB free`, names the largest memory users, and lists installed models of the same task capability
+that fit the free memory (or says none does). Pin one with `--model`.
+
+Failures exit non-zero (1 for runtime errors, 2 for usage errors): `init` when `blocked`, and every
+command that cannot reach `serve` or is refused by it. A malformed JSON flag names itself, for
+example `--questions is not valid JSON: <parser error>`.
+
 ## Execute tasks
 
 The prompt for `task` is positional:
@@ -225,6 +268,10 @@ npx @octocodeai/freellama task --task completion --objective fastest "Reply with
 npx @octocodeai/freellama task --task coding --min-confidence medium "Explain this patch."
 ```
 
+`freellama catalog [--job JOB]` prints the shipped model picks for each job, with their evidence, what
+is installed and what fits now. Without a running `serve`, `installed` and `fits_now` stay null. Recipes
+are in the [cookbooks](cookbooks/README.md).
+
 Useful task options include:
 
 - `--model` for an exact installed model.
@@ -234,7 +281,9 @@ Useful task options include:
 - `--context-tokens` for the complete input and output context window (`num_ctx`).
 - repeatable `--required-capability` constraints.
 - repeatable `--image` paths for vision tasks.
-- `--input-file` reads the input from a file: one item per line for `--task embedding` (batched in one request), otherwise the whole file is the prompt.
+- `--input-file` reads the input from a file: one item per line for `--task embedding` (batched in one request), the state for `--task decision` (JSON stays JSON), otherwise the whole file is the prompt.
+- `--nearest QUERY [--top N]` for `--task embedding --input-file FILE`: ranks the file's lines against the query and prints the closest (line number, score, text) instead of vectors; use the model's own prefixes, such as `search_query: ` for nomic.
+- `--questions JSON|@FILE` for `--task decision`: named `noul`, `choice` or `score` questions answered by a decision model through `/v1/systemone`, for example `freellama task --task decision --questions '{"kind":{"type":"choice","instructions":"Which kind?","criteria":{"bug":"Broken","feature":"New"}}}' "Checkout returns 500 since 9am."`.
 - `--min-confidence low|medium` to refuse insufficiently evidenced routes before generation.
 - `--priority interactive|normal|background` for fair admission scheduling.
 - `--max-wait-seconds` for a shorter admission/resource wait budget.
@@ -251,8 +300,8 @@ through MCP `task_jobs`.
 Cancellation and removal wait for local execution permits to be released; they do not unload a shared model
 or confirm that the physical runner has stopped.
 
-Jobs stay in server memory: at most 64 receipts, 1 MiB per input, and 2 MiB per retained result or
-error. Finished receipts expire after 10 minutes and can be evicted earlier to make room for new
+Jobs stay in server memory: at most 64 receipts, 16 MiB per input (64 MiB across queued and running
+jobs), and 2 MiB per retained result or error. Finished receipts expire after 10 minutes and can be evicted earlier to make room for new
 work. A restart discards all jobs. Admission queue limits still apply, and deferred work can fail
 after acceptance. See [monitoring](MONITORING.md) for states and waiting reasons.
 
@@ -297,8 +346,8 @@ npx @octocodeai/freellama serve
 ```
 
 `serve` resolves each file in this order: the explicit flag, its environment variable, the working
-directory, then the per-user config directory (`$XDG_CONFIG_HOME/freellama/` or
-`~/.config/freellama/`; `%APPDATA%\FreeLlama\` on Windows).
+directory, then the FreeLlama home (`~/.octocode/freellama`, or `--home` / `FREELLAMA_HOME`). The home also holds
+`runtime.toml`, read when neither `--runtime-config` nor `FREELLAMA_RUNTIME_CONFIG` is set.
 
 | File | Flag | Environment variable | Discovered name |
 |---|---|---|---|

package/docs/COMPARISON.md

@@ -0,0 +1,40 @@
+# How FreeLlama compares
+
+Moved from the root README.
+
+## How it compares
+
+FreeLlama is not an inference engine: Ollama loads and runs the model, FreeLlama decides whether,
+where and how each task runs. Other projects' behavior is taken from their docs and source, checked
+on 2026-11-01. Moving-branch features may differ from installed releases.
+
+| Decision | Other local stacks | FreeLlama |
+|---|---|---|
+| Can this load fit right now? | Ollama already estimates memory, reuses runners, queues work and evicts models ([scheduler](https://github.com/ollama/ollama/blob/main/server/sched.go), [FAQ](https://docs.ollama.com/faq)). llama.cpp has model loading and eviction ([server docs](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)); llama-swap documents declared capacity pools and cost-based eviction ([config](https://github.com/mostlygeek/llama-swap/blob/main/docs/config.example.yaml)); LocalAI has memory-management policies with backend-dependent estimates ([docs](https://localai.io/docs/advanced/vram-management/)). | Checks each load against live available memory plus a reserve sized to the load, unloading idle runners first. A load that still does not fit waits, then is refused with a reason and a `Retry-After` from measured service time; resident models keep serving. While models are loaded, a background sweep unloads idle ones when the host runs short. |
+| Is the machine busy or hot? | Ollama and the llama.cpp router do not gate on host CPU load or heat; llama-swap exports swap and GPU temperature as metrics without acting on them ([metrics](https://github.com/mostlygeek/llama-swap/blob/main/internal/perf/prometheus.go)). | Holds CPU-bound work (all work on a host without a GPU) while the 1-minute load average is at least 1.5 per CPU, and CPU-bound work and new loads under Mac thermal pressure; the receipt names the signal. Another program's CPU load never blocks GPU work. |
+| What loads before it is asked for? | llama-swap, the llama.cpp router (`load-on-startup`) and LocalAI (`--load-to-memory`) load a fixed list at startup ([llama-swap](https://github.com/mostlygeek/llama-swap/blob/main/docs/config.example.yaml), [llama.cpp](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md), [LocalAI](https://github.com/mudler/LocalAI/blob/master/docs/content/reference/cli-reference.md)). These examples describe startup configuration, not an exhaustive feature comparison. | A startup list, optionally with a prompt prefix evaluated into the runner's prompt cache, plus the model that usually comes next, loaded only when it fits without unloading anything. |
+| Who may spend what? | LiteLLM enforces per-key budgets from its database ([users](https://docs.litellm.ai/docs/proxy/users)); llama-swap and the llama.cpp router accept a static key list. | Named client keys with per-key usage, a daily token budget that excludes prompt-cache reuse, and admin rights for model files, on managed and raw traffic alike. |
+| Other engines? | LiteLLM, Olla and LocalAI route across engines and fail over between them ([LocalAI failover](https://github.com/mudler/LocalAI/blob/master/docs/content/features/model-failover.md)). | OpenAI-compatible engines (LM Studio, llama.cpp, vLLM, a cloud API) behind the same `/v1` endpoint, routed by model name. |
+| Who goes first? | LiteLLM returns 429 once a deployment's parallel limit is reached, with an optional priority queue ([routing](https://docs.litellm.ai/docs/routing), [scheduler](https://docs.litellm.ai/docs/scheduler)). | Weighted fair admission per backend (interactive 3 : normal 2 : background 1) with starvation prevention and bounded waits; separate GPU and CPU pools. |
+| Which model? | Ollama runs the tag it is given. LiteLLM picks among deployments by load, latency or cost; RouteLLM trains a strong/weak router ([RouteLLM](https://github.com/lm-sys/RouteLLM)). | Picks an installed model by capabilities, context window, memory estimate and residency; a model that keeps looping on a task kind ranks last for it. |
+| Did it loop? | Ollama aborts some loops with a "token repeat limit" error. | Streams, stops once the output repeats and keeps the text (`stopOnRepetition`); `output_check` on every task. |
+| Loading or waiting? | Ollama streams generation and reports final timing; outer admission and host-resource phases belong to the coordinator. | MCP progress notifications (when the client sends a `progressToken`): `queued`, `waiting for memory: need 9.1 GB, 6.4 GB available`, `loading MODEL (~N s)`, `running`; the answer line adds `loaded N s, evicted X`. `GET /ready` answers 503 naming what is missing (`ollama`, `cpu_backend`, `catalog`). |
+| What reaches the agent's context? | Ollama returns its native response; presentation and filtering depend on the client. | A compact result (answer, verdict, model, tokens) plus a `response_file` holding thinking, vectors, research observations and receipts, read a section or page at a time with `read_response`. |
+| Where did it run? | Ollama exposes running-model memory/placement in `/api/ps` and generation timing in responses. | A receipt with the chosen model and why, queue and resource waits, and the placement Ollama reported in `/api/ps`, so a silent CPU fallback is visible. |
+
+Limits: FreeLlama does not change Ollama's decode speed, does not fail over between engines, and runs
+CPU work only on a second, CPU-only Ollama process (`serve --spawn-cpu-backend` starts and stops one for you).
+
+Measured on one Apple M2 Pro (32 GB) unless noted; single-machine measurements of specific
+workloads, not a general speed claim. The gains come from not loading, waiting or generating what the
+task does not need:
+
+| Workload | Without | With FreeLlama |
+|---|---|---|
+| OCR of 12 report strips, `glm-ocr`, 1,500-token cap | 12,948 output tokens, 126 s | `stopOnRepetition`: 6,452 tokens, 69 to 103 s, every label still read |
+| Resident GPU tasks during a compiler build | 64 to 120 s waits (host-wide CPU-load hold, 0.5.0) | No wait: the CPU-load hold applies only to CPU-assigned work |
+| `Retry-After` for a queued burst | 76 s quoted for a 32 s wait (0.5.0) | 32 s |
+| A GPU model and a CPU model side by side (M4 Pro, 48 GB) | 38.0 s median, one Ollama process | 28.2 s median with a second CPU-only Ollama process (1.35x; one of three runs was slower) |
+| Cost of a managed task over raw Ollama, resident embedding, n=20 (0.8.0, 2026-10-31) | raw passthrough: 2.7–2.9 ms median | 4.0 ms median, 5–7 ms p90, with routing, admission and a receipt (56 ms before the per-task process probe was cached) |
+| CPU placement through a second CPU-only Ollama (0.8.0, 2026-10-31) | — | `nomic-embed-text:v1.5` reported `processor: "cpu"`, `size_vram: 0`, verified from `/api/ps`; 265 ms load |
+| `gpt-oss:20b` cold load while the host had 12.7 GB available (0.8.0, 2026-10-31) | 0.7.0 admitted it with a memory warning on macOS; a run like this froze the machine | Held (`held_insufficient_capacity`: needs 18.0 GB with the reserve), then refused at the caller's deadline with `Retry-After`; the model never loaded |