@octocodeai/freellama-mcp-server 0.1.0 → 0.6.0

package/docs/MODEL_SELECTION.md

@@ -1,10 +1,18 @@
 # Choose local models
 
-Choose models from local evidence, not model-card popularity. The measurements below come from one
-Apple M4 Pro machine with 48 GB unified memory and are not compiled into routing. Re-run
+To offload a task, select an installed model with the required capabilities and measured quality,
+then check its memory fit under the intended concurrency. FreeLlama applies model eligibility
+and admission; the calling agent still verifies the output.
+
+The measurements in this guide come from one Apple M4 Pro machine with 48 GB unified memory and
+are not compiled into routing. Re-run
 `npx @octocodeai/freellama doctor`, `models`, `bench-all`, and your quality suite on the target hardware.
 `doctor.machine.memory_bytes` is host RAM; a discrete GPU's usable VRAM must be observed separately.
 
+For supported features and sourced publisher use-case guidance, inspect
+`models {view:"installed", includeLibrary:true}` through MCP. [Model metadata](#model-metadata) explains digest matching, variant capabilities,
+custom-model fallbacks, and how to read family claims without treating them as quality measurements.
+
 ```mermaid
 flowchart TD
     T{"What must the task do?"}
@@ -24,6 +32,15 @@ flowchart TD
 grounded research, image description, OCR, and summarization. Using one large multimodal model also
 avoided contention between separate heavy text and vision runners.
 
+Read the [retained model profile](https://github.com/bgauryy/FreeLlama/blob/main/skills/freellama/references/model-profile-qwen3.8-27b-mlx.md)
+and [delegation measurements](https://github.com/bgauryy/FreeLlama/blob/main/skills/freellama/references/task-delegation.md) for scope and limits.
+These examples do not establish that a tag is installed or suitable on your host.
+
+Dedicated OCR models answer best to the prompts their model card documents; send them with base64
+`images` in `run_task`. A model that repeats its answer until the output cap can run with
+`stopOnRepetition:true`. See [Route images](https://github.com/bgauryy/FreeLlama/blob/main/packages/mcp/README.md#route-images)
+for the early stop and the `output_check` receipt.
+
 `muse-glimmer:30b-mlx` was a credible alternative and answered 29 of 30 questions correctly
 (96.7%) with the Bash adapter, but it was slower and did not add a required capability in this
 workload.
@@ -48,7 +65,7 @@ keywords, repository search was both faster and more accurate in the measured sa
 
 ## Place helper models on CPU when useful
 
-A small embedding or intent model can be assigned to a second CPU Ollama process while a large
+A small embedding or helper model can be assigned to a second CPU Ollama process while a large
 generation model remains resident on the primary GPU-capable process. This can overlap independent
 work, but it does not make CPU inference fast. Apple silicon shares memory bandwidth; discrete-GPU
 systems have different transfer and contention costs and need their own benchmark.
@@ -63,3 +80,109 @@ this machine and workload, not a universal model-placement rule. Follow
 `bench-all` measures local performance. A quality benchmark measures correctness. FreeLlama needs
 both the generated task policy and local benchmark data before grading a route medium confidence.
 See the [CLI reference](CLI.md#earn-medium-routing-confidence) for the commands.
+
+## Model metadata
+
+The MCP `models` tool reads your Ollama instance and can enrich results with Ollama's public model
+library. Supported features and advertised use cases are metadata: they do not change routing
+confidence, and exact-model benchmarks and policy remain the quality evidence. The CLI's `models`
+command reports the core catalog and does not expose these enrichment flags.
+
+| Request | Result | External access |
+|---|---|---|
+| `models {view:"installed"}` | Managed inventory with a `knowledge` object for each model | Local backend data only |
+| `models {view:"installed", includeLibrary:true}` | Inventory enriched with public guidance for verified local tags | Fetches model family and tag pages from ollama.com |
+| `models {view:"detail", model:"gemma3:1b"}` | Ollama details, maximum context, and supported-use explanations | Selected Ollama endpoint only |
+| `models {view:"detail", model:"gemma3:1b", includeLibrary:true}` | Details plus public guidance after a tag and digest check | Selected Ollama endpoint and ollama.com |
+| `models {view:"library", query:"coding"}` | Public family search with descriptions and advertised feature chips | ollama.com and optional local inventory/profile reads |
+| `models {view:"library", model:"gemma3"}` | All public tags with size, digest prefix, context, modalities, and cloud status | ollama.com and optional local inventory/profile reads |
+
+
+| Option | Valid views | Default | Meaning |
+|---|---|---|---|
+| `includeLibrary` | `installed`, `detail` | `false` | Requests public metadata; the default makes no website calls |
+| `includeReadme` | Library step 2, or `detail` with `includeLibrary:true` | `false` | Returns README text, capped at 65,536 characters, with its format and truncation flag |
+| `includeVerbose` | `detail` | `false` | Returns Ollama's license and Modelfile fields independently of README enrichment |
+| `limit`, `cursor` | `raw`, library tag listings; `limit` also applies to library search | Tag/raw page size 20; search limit 10 | Pages large lists; the tag cursor rejects a changed tag list |
+| `ollamaEndpoint` | Direct Ollama views and library inventory reads | Configured Ollama endpoint | Chooses the Ollama instance; enriched managed inventory uses each model's assigned backend |
+
+An incompatible option returns an error before any lookup. Installed enrichment omits full READMEs;
+request one model's detail to read its README.
+
+### Supported features
+
+`knowledge.supportedFeatures` preserves the feature strings from the selected source
+(`supportedFeaturesSource`: `ollama_api` or `freellama_catalog`). Unknown future strings stay visible
+but do not create routing requirements. `knowledge.supportedUseCases` explains each recognized
+feature with `capability`, `use`, `reason`, and `basis:"supported_feature"`;
+`knowledge.quality` stays `"metadata_only"`.
+
+| Feature | Supported workflow | What to evaluate separately |
+|---|---|---|
+| `completion` | Text generation | Coding, summarization, reasoning, and instruction-following accuracy |
+| `tools` | Tool calls | Argument correctness, tool selection, and agent-loop reliability |
+| `vision` | Image input | OCR and visual understanding accuracy |
+| `audio` | Audio workflows advertised by the model | Supported audio task and accuracy |
+| `embedding` | Retrieval and similarity vectors | Retrieval quality on your documents |
+| `thinking` | A thinking mode | Reasoning accuracy, latency, and token cost |
+| `insert` | Suffix-based text insertion | Completion accuracy in the target code or text |
+| `image` | Image generation | Output quality and supported controls |
+
+
+### Public guidance and provenance
+
+For a verified installed model, `knowledge.library` contains the source page, tag-page URL,
+`fetchedAt`, `expiresAt`, `cached`, exact public tag metadata, and:
+
+| Field | Meaning |
+|---|---|
+| `description` | The public family description |
+| `advertisedFamilyCapabilities` | Family feature chips; these do not extend the local tag's capabilities |
+| `advertisedUseCases` | Recognized coding, tool-calling, image-understanding/OCR, reasoning, embedding, summarization, question-answering, translation, and long-context claims |
+| `advertisedUseCases[].evidence` | A bounded source excerpt, source URL, and fetch timestamp |
+| `usageNotes` | Bounded excerpts about requirements, restrictions, context, memory, languages, or query prefixes |
+| `readmeAvailable` | Whether a README was extracted |
+| `readme` | Requested text with `format` (`markdown` or `text`) and `truncated` |
+
+
+Advertised uses have `basis:"advertised"`, `scope:"family"`, and `quality:"unmeasured"`. Extraction
+skips fenced code and negative statements; an omitted use means no recognized claim, not absence.
+These labels are not the `run_task` task enum. Local capabilities filter advertised uses: a family
+vision chip cannot make `gemma3:1b` a vision candidate.
+
+Installed enrichment requires the exact public tag and a digest-prefix match of at least 12 hex
+characters. Custom names are never mapped by architecture or guessed base model; without a verified
+match, public guidance is withheld. Library step 2 describes public models before installation, so
+its guidance stays family-scoped; each `tags[]` entry carries its own modalities, context, digest
+prefix, and `cloud` flag. Search's `cloudAvailable` badge does not prove local downloads are absent.
+
+### Lookup states
+
+| `library.status` | Meaning | Next action |
+|---|---|---|
+| `not_requested` | Website enrichment was not requested | Set `includeLibrary:true` if external lookup is useful |
+| `matched` | Installed tag and public digest prefix match | Read the sourced guidance, then evaluate the exact tag |
+| `available` | Public family and tag pages were parsed in library step 2 | Inspect the exact tag before recommending a download |
+| `unverified` | Digest or current backend inventory could not be verified | Recheck local inventory; retain local feature information |
+| `digest_mismatch` | The public tag differs from the installed digest | Treat public guidance as unavailable for this installed model |
+| `tag_not_found` | The parsed public tag list has no exact matching tag | Check the source and custom-model provenance |
+| `unmatched` | The name cannot resolve to an exact public Ollama name | Keep use-case suitability unknown until provenance is verified |
+| `not_found` | A requested public page returned HTTP 404 | Check the source URL or custom name |
+| `unavailable` | Transport, timeout, HTTP refusal, or page-size limit prevented lookup | Retry later; local results remain available |
+| `parse_error` | HTML content or tag rows were not recognized | Inspect the source; do not conclude the model is absent |
+| `remote_model`, `remote_tag` | Remote Ollama fields or a hosted tag identify cloud execution | Select a local download tag for local execution |
+
+
+`localModel:false` flags detected remote entries; cloud tags never become local trial
+recommendations. This metadata is not processor-placement evidence.
+
+### Cache and request bounds
+
+Each MCP process caches at most 128 families: successes for one hour, failures for one minute;
+concurrent requests share one lookup and a restart clears the cache. Installed enrichment runs at
+most four family lookups concurrently. Requests use GET, refuse redirects, cap each page at 2 MiB,
+and time out after the smaller of 10 seconds and `FREELLAMA_MCP_FETCH_TIMEOUT_SECONDS`. They disclose
+the requested public family name to ollama.com. Download size does not establish runtime memory
+fit, placement, or quality, and model discovery never authorizes a pull. Sources:
+[model details API](https://docs.ollama.com/api-reference/show-model-details),
+[model-list API](https://docs.ollama.com/api/tags).

package/docs/MONITORING.md

@@ -0,0 +1,293 @@
+# Runtime status and live tuning
+
+Use runtime status to inspect how FreeLlama manages delegated work: per-backend queues and
+admission, memory pressure, resident models, execution usage, and resource holds. The operator can
+reload supported limits without a restart; the calling agent can use these observations to decide
+whether to wait, narrow a task, or handle it itself.
+
+The API endpoints in this reference are served under `/_freellama/v1/` and use the same bearer
+token as the rest of the control API when authentication is enabled.
+
+Open **`http://127.0.0.1:11435/_freellama/ui`** (the address `serve` prints at startup) for a live
+page of the same data, refreshed every two seconds: backends and queues, raw traffic, host RAM and
+VRAM, Ollama's settings, loaded models with their current tasks and load times, and the last
+eviction. The page itself is static and needs no token; when auth is on it asks for the bearer
+token and keeps it only for the browser tab.
+
+| Endpoint | CLI | MCP | What it returns |
+|---|---|---|---|
+| `GET /status` | `freellama status` | `doctor {view:"status"}` | Live view of every backend, the raw proxy, loaded models, host, Ollama settings, and today's usage |
+| `GET /usage?days=N` | `freellama usage --days N` | `doctor {view:"usage", days:N}` | Totals per day and per model (default 7 days) |
+| `GET /metrics` | | | Prometheus text exposition |
+| `GET /config` | `freellama config` | | Effective runtime settings and the source of each |
+| `POST /config/reload` | `freellama config --reload` | | Re-read the runtime file now; `422` keeps the last good values |
+| `GET /jobs` | `freellama jobs` | `task_jobs {action:"list"}` | Deferred job metadata without prompts or retained results |
+| `GET /jobs/:id` | `freellama jobs --id <uuid>` | `task_jobs {action:"get", jobId}` | One job receipt and its retained result or error |
+| `POST /jobs/:id/cancel` | `freellama jobs --id <uuid> --cancel` | `task_jobs {action:"cancel", jobId}` | Cancellation receipt after local permits are released |
+| `DELETE /jobs/:id` | `freellama jobs --id <uuid> --remove` | `task_jobs {action:"remove", jobId}` | Stops active work, releases local permits, and removes the retained record |
+
+## Read the live status
+
+`status` answers "why is my task waiting?" in one call:
+
+- **`backends.gpu` / `backends.cpu`**: admission (units in use, current limit, ceiling, queue depth,
+  oldest wait, refusal and timeout counters), the adaptive limiter state, and the circuit breaker
+  (`closed`, `open` with seconds remaining, or `half_open`).
+- **`raw_proxy`**: active and waiting raw streams against their cap and wait budget.
+- **`loaded_models`**: Ollama's `/api/ps` with total and VRAM bytes, context length, expiry, and
+  whether the model is pinned.
+- **`host`**: available and reserved RAM, whether the pressure gate is holding, and discrete-GPU
+  total and free memory (`nvidia-smi` or amdgpu sysfs) when available. Optional `gpu_activity`
+  reports driver counters separately from resident memory. On macOS, `thermal_pressure` is the
+  system thermal pressure level (`nominal`, `moderate`, `heavy`, `trapping`, `sleeping`); `heavy`
+  or worse holds CPU-assigned work and cold loads. Other platforms report `null`.
+- **`ollama`**: the Ollama server's effective `OLLAMA_*` settings with a source for each — the
+  running process's environment (Linux `/proc/<pid>/environ`, macOS `ps eww` for a same-user
+  process), `launchctl getenv` for the macOS app, FreeLlama's own environment, or Ollama's default.
+  A remote endpoint is never probed and is reported as unknown.
+- **`usage_today`**: tasks, errors, tokens, and busy time since midnight UTC.
+- **`task_jobs`**: deferred task IDs, selected models, configured backends, priorities, total
+  deadlines, states, and waiting reasons. States cover discovery, admission, resources, runner
+  transitions, loading, execution, and completion. Metadata excludes the original payload.
+
+Use `run_task {defer:true}` in MCP or `task --defer` in the CLI to get a job handle. A list or
+status request omits results; fetching one ID returns its retained result or structured failure.
+Terminal states are `completed`, `failed`, `cancelled`, and `expired`. Cancellation is per task,
+waits for local permits to be released, and does not unload a shared runner. Process-local job
+retention limits are documented in [CLI controls](CLI.md#execute-tasks).
+
+Removal uses the same exact job ID. It cancels active work before discarding the record, returns
+`{id, removed:true, status, scope:"process_memory"}`, and omits the saved result. The ID then returns
+`404` on lookup or another removal. Other jobs and installed models are unaffected. Local
+cancellation closes the upstream request; physical computation may continue on a shared runner.
+
+`ollama.config` describes the primary process; `ollama.cpu_config` describes the CPU process when
+configured. Process attribution matches `OLLAMA_HOST` to the selected endpoint, so separate
+parallelism and KV cache settings size each backend independently. Fallback sources remain visible
+when process settings cannot be observed.
+
+Process settings are cached for five seconds and refreshed on discovery or monitoring requests,
+including after an Ollama restart. Each config receipt includes `observed_at` and
+`observation_scope: "process_snapshot"`. Refresh also updates backend memory estimates and
+invalidates the use of measurements taken under different parallelism or KV cache settings.
+FreeLlama admission limits remain governed by its runtime configuration.
+
+`host.gpu_activity` is nullable. When available, it identifies its `source`,
+`scope: "observed_devices_all_processes"`, `window: "driver_defined"`, and individual `devices`.
+Each device has an identity, nullable `busy_percent`, and nullable `memory_busy_percent`. Apple
+devices may add `allocated_memory_bytes`, the driver's allocated memory across all processes.
+An observed zero means the driver reported zero; a missing or malformed counter remains unknown.
+Use `host.sample_age_ms` to check freshness.
+
+NVIDIA activity comes from the same bounded `nvidia-smi` query as memory. Apple Silicon activity
+comes from the accelerator's `PerformanceStatistics` in a bounded IORegistry read. Those Apple
+properties are driver-specific and may disappear or change; the read can cover only the devices
+within its retained output. AMD, Windows, unsupported drivers, and unavailable reads report unknown activity. The counters include other processes, use a driver-defined observation
+window, and do not attribute activity to a FreeLlama task or establish bandwidth or energy
+efficiency. They are diagnostics and do not alter admission or pressure decisions.
+
+Each backend's admission `queue_depth` includes both `slot_waiters` and `resource_waiters`.
+A resource waiter returns its execution slot after two seconds but remains in the bounded queue
+until capacity recovers, its deadline expires, or it is cancelled.
+
+## Record usage
+
+Every managed task updates in-memory usage totals and offers one record to a bounded writer queue.
+The record contains time, model, backend, task kind, priority, outcome, HTTP status, prompt and
+output tokens, duration, queue wait, load time, and output tokens per second. It contains no prompt
+or response text. One worker writes records in order; task completion does not wait for disk.
+
+The writer queue defaults to 128 records. Configure its positive startup-only capacity through
+`usage_queue_capacity` or `FREELLAMA_USAGE_QUEUE_CAPACITY`; runtime reload does not resize the worker.
+A full queue or unavailable worker drops the record; a disk
+failure records a failed write. Inspect `usage_ledger` in `status`, or `ledger` in `usage`, for
+`pending`, `failed`, `dropped`, `records`, `capacity`, and `last_error`. The durability contract is
+`asynchronous_best_effort`: pending records can be lost on a crash, and in-memory totals can exceed
+persisted totals. Error evidence remains visible after later successful writes.
+
+`serve` replays a bounded regular-file ledger at startup and rotates it to `usage.jsonl.1` at
+16 MiB. Successful writes contribute to restart totals; completion is not a zero-loss persistence
+promise. Embedded callers can use the bounded `flush_ledger` barrier for orderly teardown; it
+confirms attempted writes, not recovery of dropped/failed records or a filesystem sync.
+
+The ledger lives next to the feedback file in the platform data directory. Choose another path with
+`--usage-file` (or `FREELLAMA_USAGE_FILE`), or keep totals in memory only with `--ephemeral-usage`.
+
+## Scrape metrics
+
+`/metrics` exposes counters (`freellama_tasks_total`, `freellama_prompt_tokens_total`,
+`freellama_output_tokens_total`, `freellama_task_seconds_total`, `freellama_queue_wait_seconds_total`,
+`freellama_load_seconds_total`, `freellama_raw_requests_total`, `freellama_evictions_total`,
+`freellama_circuit_open_total`, `freellama_concurrency_limit_changes_total`) labelled by model,
+backend, task, and outcome, plus gauges for admission, queues, the raw proxy, host and GPU memory,
+loaded models, and `freellama_ollama_num_parallel`. Point Prometheus, Grafana Agent, or any
+OpenMetrics-compatible scraper at it.
+
+## Back-pressure
+
+FreeLlama waits for safe capacity within a deadline and bounds the retained backlog:
+
+| Situation | Status | `code` |
+|---|---|---|
+| Managed queue full | `429` | `admission_queue_full` |
+| Raw proxy over its cap after `raw_queue_wait_seconds` | `429` | |
+| Managed wait exceeded `max_queue_wait_seconds` (or the task's `max_wait_seconds`) | `503` | `admission_timeout` |
+| OS pressure, swapping, CPU, thermal, or required telemetry still holds when the waiting budget expires (low available RAM alone only warns; low RAM and swapping never hold an already-resident runner) | `503` | `resource_admission_unavailable` |
+| Backend circuit open | `503` | `upstream_circuit_open` |
+
+These capacity refusals carry a `Retry-After` header and `retry_after_seconds` in the body. For a queue it
+is estimated from the queue depth, the lanes that can run at once (current limit divided by the
+task's cost), and the backend's recent average service time, which excludes queue wait (1 to 120
+seconds). A task can ask for a shorter wait with `max_wait_seconds` (`maxWaitSeconds`
+in MCP); the server caps it at the configured maximum.
+
+## Circuit breaker
+
+After `breaker_failures` consecutive upstream failures on a backend (connection errors, `500`,
+`502`, or `504`), its circuit opens for `breaker_cooldown_seconds`; tasks routed there fail fast
+with `503` rather than queueing behind a broken Ollama. When the cooldown ends, one probe request is
+let through: success closes the circuit, failure reopens it. `4xx` responses such as an unknown
+model do not count. Set `breaker_failures = 0` to disable it.
+
+## Adaptive concurrency
+
+Managed task costs are configurable through the runtime file's `[task_costs]` table. Valid keys
+are `completion`, `coding`, `code_repair`, `tools`, `browser`, `vision`, `embedding`, and
+`long_context`. Each value is a positive integer through 4294967295. Omitted kinds retain their
+defaults: vision four, embeddings one, and other tasks two. Embeddings multiply their base by
+`ceil(input_items/4)`; the acquired charge cannot exceed the pool's current limit.
+
+For a qualified light OCR workload, `[task_costs]` with `vision = 2` permits two vision requests
+under a four-unit budget while leaving other task costs unchanged. This is an operator setting,
+not automatic quality or fit qualification. The memory governor and Ollama's final admission
+still apply.
+
+A task captures its cost when execution starts. Reload affects tasks that start afterward;
+already queued or active tasks keep their captured cost. `config` reports effective values and
+the runtime-file source; invalid keys, zero,
+or overflow reject the reload and retain the last good configuration. Preview advice, health
+costs, and execution receipts use the resolved policy. Health retains its summary aliases and
+exposes the complete task-kind map under `admission.costs.base_by_task`. Task kinds remain
+caller-declared; these weights are cooperative scheduling policy, not workload classification.
+See the
+[runtime example](https://github.com/bgauryy/FreeLlama/blob/main/freellama.runtime.example.toml) for the table syntax.
+
+With `adaptive_concurrency = "cpu"` (the default) or `"all"`, each pool's limit moves between 1 and
+its configured ceiling:
+
+- **Down (halve)** on an upstream error, host memory pressure (PSI or the pressure gate), or output
+  throughput below 70% of the matching execution profile's baseline. Decreases are at least
+  10 seconds apart.
+- **Up (+1 unit)** after five consecutive comparable healthy rate observations. Generation uses
+  output tokens per decode second; embeddings use input tokens per model-reported total second.
+  The profile records the metric kind, and unlike kinds cannot train or recover together.
+
+Only completed warm executions with verified placement and a comparable observed profile train
+speed baselines or healthy recovery. Identity includes model digest, explicit observed context,
+backend/process settings, effective controls, and admission class. Fresh endpoint-attributed
+process observations bracket candidate serial samples. Unknown identity, changed controls,
+parallel intervals, invalid rate observations, and incomplete or canceled execution break the
+recovery streak. Failures and observed memory pressure retain their safety decreases.
+
+Serial samples require one managed task throughout the interval: unchanged admitted/released
+counters, active charge, capacity, and policy epoch reject transient overlap or capacity changes.
+This avoids comparing serial request speed with legitimate parallel decoding. It does not prove
+that no external Ollama client ran concurrently. A successful parallel execution can therefore
+report verified placement and `feedback.accepted:true` while
+`execution.throughput_learning.eligible:false`; it contributes no comparable speed sample.
+
+Each backend retains at most one adaptive profile window per task kind. Status exposes
+`adaptive.profile_windows` keyed by task; `baseline_output_tokens_per_second` is a compatibility
+output-only summary by model and can omit additional task windows for the same model. Embedding
+windows report their explicit input-rate kind separately. Use the task windows for learning
+evidence. See [automatic feedback](CPU_GPU_ROUTING.md#understand-automatic-feedback) for
+persisted routing windows.
+
+The CPU backend is adaptive by default because oversubscribing it shows up as a throughput collapse
+rather than an error. Every change is counted in metrics and shown in `status`.
+
+## Model residency and eviction
+
+Omitted managed `keep_alive` values use finite retention informed by measured reuse and loading.
+Explicit values take precedence. For governed prewarming, profile compatibility, and retention
+settings, see [Scope history and model warming](SCOPES_AND_WARMING.md).
+
+When a model about to load needs room, FreeLlama unloads idle runners itself (`keep_alive: 0`)
+instead of letting the load wait, fail, or spill to CPU. It does this in three cases: host RAM
+can fall below the reserve, the host is already holding for low memory, or (discrete GPU) the
+model does not fit in free VRAM, where Ollama otherwise chooses its own victim.
+
+Which runners go is a small optimisation, in the spirit of llama-swap's `evict_costs` and LocalAI's
+busy-aware LRU:
+
+- **Never unloaded:** the model being loaded, `pinned_models`, `keep_alive: -1` runners, runners
+  Ollama is still loading, and any model with a FreeLlama task queued or running (unloading it forces an immediate reload). Busy status is checked again right before each unload.
+- **Not for a model bigger than the GPU:** it spills to the CPU whatever is unloaded, so VRAM is
+  left alone.
+- **Cost of unloading a runner** = its reload time (Ollama's measured `load_duration` from earlier
+  loads, or size ÷ 1.5 GB/s before the first one) × (1 + recent uses, decaying with a 30-minute
+  half-life; only tasks served count) × its `eviction_costs` weight (default 1).
+- **Choice:** the set of runners with the lowest total cost that frees enough memory; ties go to
+  fewer runners, then less memory freed beyond what is needed. Only the memory that matters is
+  counted: VRAM for a GPU fit, the host-RAM part for a spill, the whole runner on CPU or unified
+  memory.
+
+Every managed receipt that caused an eviction carries the plan under
+`memory_reservation.eviction`: what was unloaded, its cost and reload estimate, and what was kept
+and why. `status` and the page show the latest one that unloaded something. Eviction runs once per
+task, after admission and before the memory wait, and is never cancelled halfway by a deadline.
+Set `evict_idle_models = false` to leave residency entirely to Ollama.
+
+## Who decides what: FreeLlama and Ollama
+
+Two schedulers that both guess at memory can disagree. The split is:
+
+| Decision | Owner | How |
+|---|---|---|
+| Exact memory of a loaded runner | Ollama | Measured; FreeLlama reads it from `/api/ps` |
+| Whether a new load fits | Both, Ollama final | FreeLlama estimates from Ollama's past measurements, then Ollama loads or refuses |
+| Decoding slots per model | Ollama | `OLLAMA_NUM_PARALLEL`; FreeLlama's primary budget defaults to two weighted units per slot |
+| Admission, priority, queue limits, back-pressure | FreeLlama | Weighted pools, 429/503 with `Retry-After` |
+| Which idle runner to unload, and when | FreeLlama | The cost planner above, before Ollama has to choose |
+| Context size | Ollama by default | `num_ctx` is left to Ollama when its default covers the request |
+
+To keep the estimates honest, FreeLlama learns from every runner Ollama reports as resident,
+including ones loaded by raw clients, and saves those measurements to `footprints.json` next to the
+usage ledger. After a restart a model it has seen before is sized from Ollama's measurement, not a
+formula. Each measurement records the `OLLAMA_NUM_PARALLEL` and KV-cache type it was taken under and
+is used only while those match, so changing either setting falls back to the formula until Ollama
+measures again. A larger context than any measured one is sized as the measurement plus exactly the extra
+KV cache, when the model's KV shape is known.
+
+## Context sizing and Ollama's defaults
+
+Sending `num_ctx` on a request pins the runner to that size: a later request for the same model
+with a different or missing `num_ctx` forces Ollama to reload it, and Ollama's automatic
+out-of-memory shrink-and-retry only applies when the context was not requested explicitly.
+
+With `context_mode = "ollama_default"` (the default), FreeLlama sizes the context as before but
+omits `num_ctx` when Ollama's own default for the model is known, covers the request, and is at most
+`auto_context_max` (32768). The default is resolved in Ollama's order: the Modelfile's `num_ctx`,
+then `OLLAMA_CONTEXT_LENGTH`, then (discrete GPU only) Ollama's VRAM tier — 4096 below 24 GiB, 32768
+below 48 GiB, 262144 above. Receipts show `context_sizing.num_ctx_sent` and the default used.
+Memory estimates always use the context Ollama can allocate. Use `"explicit"` to always
+send `num_ctx`.
+
+## Change settings without a restart
+
+Put any of the settings above in a TOML file and start `serve --runtime-config <file>` (or set
+`FREELLAMA_RUNTIME_CONFIG`). Start from
+[`freellama.runtime.example.toml`](https://github.com/bgauryy/FreeLlama/blob/main/freellama.runtime.example.toml). `serve` checks the file every
+two seconds and applies changes to live admission pools; a waiting task sees the new limit at once.
+
+Precedence for every setting, strongest first: CLI flag, environment variable (where one exists),
+the runtime file, the built-in default. Only these settings have an environment variable:
+`FREELLAMA_MAX_QUEUE_WAIT_SECONDS`, `FREELLAMA_EVICT_IDLE_MODELS`, `FREELLAMA_USAGE_QUEUE_CAPACITY`,
+and the individual `FREELLAMA_SCOPE_*` and `FREELLAMA_KEEP_ALIVE_*` limits. Set every other key in
+the runtime file or with its CLI flag. A flag or variable pins a value, so editing the file cannot
+override it; `freellama config` shows which source won: `cli`, `env`, `file`, a default source
+such as `default` or `ollama_num_parallel`, or `env_invalid` when a set variable failed validation.
+A malformed file or an unknown key is rejected at reload and the previous values stay in force.
+
+The default GPU budget follows the Ollama server: 2 units per `OLLAMA_NUM_PARALLEL` slot (one chat
+per decoding slot), read from the running Ollama process where visible.

package/docs/PRODUCTION.md

@@ -1,17 +1,23 @@
 # Prepare FreeLlama for production
 
-This runbook prepares a single-operator or trusted-team FreeLlama deployment. FreeLlama now
-supports authenticated nonloopback listeners, but it is not a multi-tenant authorization or
-billing service. Put TLS and tenant-specific rate limits at an external ingress when traffic
-crosses a machine boundary.
+This runbook prepares the FreeLlama management layer for agents using local models in a
+single-operator or trusted-team deployment. Configure the Ollama backends, model evidence,
+authentication, and resource limits before agents submit work.
+
+FreeLlama supports authenticated nonloopback listeners. Put TLS and tenant-specific rate limits
+at an external ingress when traffic crosses a machine boundary; FreeLlama's bearer token does
+not provide tenant isolation or billing.
 
 ## Meet the prerequisites
 
 - Install one Ollama CLI and server version. Run `freellama doctor` and resolve any mismatch.
 - Install Node.js 20 or later for MCP and npm distribution use.
-- Use the release artifacts for your operating system, or build with Rust 1.85 or later.
+- Use the release artifacts for your operating system, or build with Rust 1.88 or later.
 - Preinstall and benchmark exact model tags. Discovery never authorizes a pull.
 - Keep the primary and optional CPU Ollama processes on distinct loopback sockets.
+- In a slim or distroless container, install the CA bundle (`ca-certificates`) or set
+  `SSL_CERT_FILE` if any upstream uses `https://`. Without one, FreeLlama starts with an empty trust
+  store and warns: plain-HTTP upstreams work, HTTPS upstreams fail their certificate check.
 
 After a tag has been published, install that exact checksummed release on macOS or Linux:
 
@@ -28,9 +34,14 @@ published tag:
 powershell -ExecutionPolicy Bypass -File .\scripts\install.ps1 -Version VERSION
 ```
 
-There is no repository-hosted release automation. Build and test every claimed target in a clean
+The repository has no CI or release workflows; they were removed (see git history). Build, verify,
+and publish artifacts with the [release procedure](../RELEASE.md). The publish verifier checks that
+a version tag matches the workspace version.
+Live E2E and hardware qualification remain separate operator checks; verify those receipts before tagging.
+
+For manual assembly, build and test every claimed target in a clean
 environment, place each executable and matching addon under `release-artifacts/<target>/`, run
-`yarn release:assemble release-artifacts release` and `yarn release:verify:publish`, then attach
+`node scripts/assemble-release.mjs release-artifacts release` and `yarn release:verify:publish`, then attach
 `release/SHA256SUMS` and binaries to the release. Publish the eight `@octocodeai/freellama-native-*` packages
 first, followed by the exact-version `@octocodeai/freellama` and `@octocodeai/freellama-mcp-server` packages. Configure
 registry credentials in the release environment; never store them in the repository.
@@ -52,9 +63,13 @@ swap, eviction churn, or unacceptable queueing.
 | `OLLAMA_MAX_QUEUE` | Workload-specific, bounded value | Limits Ollama's internal queue independently of FreeLlama admission |
 
 Do not set a global `OLLAMA_CONTEXT_LENGTH` merely to maximize the advertised window. Managed
-FreeLlama tasks send the smallest sufficient request-specific `num_ctx`; a larger context and a
-larger `OLLAMA_NUM_PARALLEL` multiply K/V-cache memory. Direct Ollama clients still follow the
-server's context configuration, which defaults to 4096 tokens in current Ollama.
+FreeLlama text tasks estimate a sufficient context and can omit `num_ctx` when Ollama's known default
+covers the request within the configured automatic bound. Explicit context requests send `num_ctx`.
+A larger context and a larger `OLLAMA_NUM_PARALLEL` multiply K/V-cache memory. See
+[context sizing](MONITORING.md#context-sizing-and-ollamas-defaults) for defaults and receipts.
+Direct Ollama clients still follow the
+server's context configuration. Current context documentation describes VRAM-tiered 4k/32k/256k
+defaults while the FAQ still says 4096, so treat `/api/ps` `context_length` after load as authority.
 
 `freellama doctor` returns `local_conservative_config_posture` as a **non-mutating** portable
 starting profile. It names the source of each value (`observed_process` or `configuration_hint`),
@@ -63,11 +78,13 @@ parallel stream, and a finite internal queue before benchmarking. It also report
 `host_runtime_signals` with source and permission scope. Treat unavailable GPU-memory, thermal, or
 power signals as unavailable; do not substitute host RAM or a guessed value.
 
-FreeLlama and Ollama have separate queues. FreeLlama acquires a weighted backend permit and waits
-for at most `--max-queue-wait-seconds`. An admitted request can then enter Ollama's internal queue,
-which is bounded by `OLLAMA_MAX_QUEUE`. Raw compatibility traffic bypasses FreeLlama admission and
-enters the primary Ollama queue directly. Set both limits from observed latency and overload
-behavior; changing one does not configure the other.
+FreeLlama and Ollama have separate queues. FreeLlama caps retained managed waiters per backend with
+`--max-queued-tasks`/`--cpu-max-queued-tasks`, then waits for at most
+`--max-queue-wait-seconds`. An admitted request can then enter Ollama's internal queue,
+which is bounded by `OLLAMA_MAX_QUEUE`. Raw compatibility traffic bypasses weighted managed
+admission and enters the primary Ollama queue through a separate one-stream default cap in `serve`.
+Set all limits from observed latency and overload behavior; changing one does not configure the
+others.
 
 The following flow shows where each production setting applies:
 
@@ -76,20 +93,22 @@ flowchart LR
     C["Client"] --> FQ["FreeLlama admission<br/>weighted budget + wait deadline"]
     FQ -->|"refused"| E503["503 server busy"]
     FQ -->|"admitted"| OQ["Ollama queue<br/>OLLAMA_MAX_QUEUE"]
-    RAW["Raw /api/* or /v1/*"] --> OQ
+    RAW["Raw /api/* or /v1/*"] --> RC["Raw stream cap<br/>default 1 in serve"]
+    RC -->|"refused"| E503
+    RC -->|"admitted until EOF/drop"| OQ
     OQ --> S["Ollama scheduler<br/>loaded models + parallel streams"]
     S --> M["Runner memory<br/>num_ctx x K/V cache"]
 ```
 
 These values are conservative starting points, not universal constants. NVIDIA, AMD, Apple, and
 CPU-only hosts must pass hardware validation before promotion. See
-[Ollama and FreeLlama optimization](OLLAMA_SYSTEM_OPTIMIZATION.md) for the ownership boundary and
+[Ollama and FreeLlama optimization](dev/OLLAMA_SYSTEM_OPTIMIZATION.md) for the ownership boundary and
 [Run models on CPU and GPU](CPU_GPU_ROUTING.md) for device-specific controls.
 
 Put the values in the service manager that owns each Ollama process: launchd on macOS, systemd or
 the container definition on Linux, and the Windows service wrapper. Shell exports and
 `launchctl setenv` are useful diagnostics but are login-session state, not a reboot-persistent
-production configuration. After a real service restart, rerun `doctor` and `scripts/check.sh` and
+production configuration. After a real service restart, rerun `doctor` and `skills/freellama/scripts/check.sh` and
 require the same visible values before admitting work.
 
 ## Create the security and state files
@@ -122,6 +141,18 @@ Only physically verified warm samples are saved. A corrupt or unsupported snapsh
 fail instead of silently discarding routing evidence. Use `--ephemeral-feedback` only for
 disposable tests.
 
+## Bound conversation history and residency
+
+Scope histories and deferred jobs are process-local and are discarded on restart. Configure
+`[scopes]` and `[warming]` limits in the runtime file from measured workload needs; finite history
+bounds and automatic retention remain separate from host-memory reserves. Warm loads use the same
+admission and placement checks as tasks. See [Scope history and model warming](SCOPES_AND_WARMING.md)
+for expected revisions, forks, explicit history reads, and residency precedence.
+
+Usage persistence is asynchronous best effort. Inspect pending, dropped, failed, and error evidence
+in `status.usage_ledger`; task success does not prove its usage record reached disk. See
+[Usage recording](MONITORING.md#record-usage) for replay, rotation, and shutdown boundaries.
+
 ## Configure MCP
 
 Set these values in the MCP host environment:
@@ -143,7 +174,7 @@ direct Ollama requests, and managed FreeLlama tasks send their own `num_ctx`.
 
 ## Verify lifecycle and placement
 
-`keep_alive:0` no longer sacrifices placement evidence. FreeLlama temporarily holds the runner,
+`keep_alive:0` preserves placement evidence. FreeLlama temporarily holds the runner,
 observes `/api/ps`, records eligible feedback, explicitly unloads it, and verifies that it is no
 longer resident before returning. Inspect both `execution.observation` and
 `execution.lifecycle.status`.
@@ -166,12 +197,9 @@ the Ollama process does not expose to FreeLlama.
 Run compile and deterministic test checks from clean checkouts on Windows, Linux, and macOS. Live
 accelerator acceptance is separate because build machines do not necessarily represent production
 GPU drivers or memory topology. Prepare Apple Metal, NVIDIA Linux, AMD Linux, and NVIDIA Windows
-hosts, then run `benchmark/hardware/run_validation.py` directly on each host.
+hosts, then run a hardware acceptance check directly on each host.
 
-Each environment supplies its exact GPU and CPU model tags and a running FreeLlama endpoint. The
-portable runner in `benchmark/hardware/run_validation.py` executes GPU coding and CPU embedding
-concurrently, verifies physical processor receipts, checks admission receipts, and writes a JSON
-artifact. Add `--vision-model` and `--vision-image` for that host's OCR/vision gate.
+The portable hardware runners were removed from the repository; they remain in git history.
 
 Do not mark a hardware row validated until its archived receipt says `verdict: accept`. Missing
 hardware is an explicit release gap, not a result that documentation can waive.
@@ -188,7 +216,7 @@ Promote a release only when all conditions pass:
 6. Restart testing proves feedback reloads and a corrupt snapshot fails startup.
 7. Immediate unload reports verified pre-unload placement and verified post-unload absence.
 8. Every claimed hardware row has a real archived acceptance receipt.
-9. The MCP schema remains eight tools and within its context budget.
+9. The MCP schemas match the published tool surface and pass its context-budget contract.
 
 Rollback by returning the listener to loopback, stopping the new service, and starting the prior
 checksummed binary with the same policy and feedback snapshot. Never downgrade across an unsupported