@octocodeai/freellama-mcp-server 0.10.0 → 0.12.0

package/docs/RELIABILITY.md

@@ -0,0 +1,207 @@
+# Result checks and local tool-calling evidence
+
+`check:` lines are review hints. A clean result does not establish factual correctness, a safe
+edit, an appropriate tool choice, or authority to execute a call. This page keeps two separate
+measurements: deterministic accuracy of those hints and observed tool-calling behavior of exact
+installed local models. [Machine-local model calibration](MODEL_SELECTION.md#turn-measurements-into-routing-evidence)
+measures another concern: which eligible model has local performance and policy evidence. Neither substitutes for the calling agent's verification.
+
+## Accuracy of result checks
+
+The versioned [30-case labeled corpus](../scripts/reliability/check-fixtures.mjs) tests human-labeled
+review risks against the actual exported check functions. Every input, label, emitted check and
+outcome from the original checker is in [the baseline report](assets/result-check-accuracy.json);
+the repaired checker has a separate [numeric-tokens-v2 report](assets/result-check-accuracy-v2.json). These synthetic
+regression cases include deliberate blind spots; their proportions do not estimate production
+accuracy. The labels and detector thresholds were visible during implementation.
+
+A positive means a specific warning about a review-worthy risk, rather than any nonempty `check:`
+line. Generic spot-check reminders and structural success messages are not positive detections.
+Unsupported tool-schema keywords produce an explicit unchecked warning and count as a review risk.
+
+The original checker baseline:
+
+| Review-risk family | Cases | True positives | False positives | True negatives | False negatives | Precision | Recall |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Source factual risk | 12 | 2 | 2 | 2 | 6 | 50% | 25% |
+| Instructions in input data | 4 | 1 | 1 | 1 | 1 | 50% | 50% |
+| Dropped extraction values | 4 | 1 | 0 | 1 | 2 | 100% | 33.3% |
+| Tool structural risk | 10 | 8 | 0 | 2 | 0 | 100% | 100% |
+
+Precision is `TP / (TP + FP)`; recall is `TP / (TP + FN)`. Empty denominators are unknown. Do not
+combine the families into one reliability score: they ask different questions.
+
+The 2026-11-15 numeric-token repair fixes two existing labeled cases: `8080` no longer matches a
+source token `18080`, and comma grouping is normalized in both directions. Human risk labels and
+the original report are unchanged. On the same 12 source-risk cases, the repaired checker has
+TP 3, FP 1, TN 3, FN 5: precision 75%, recall 37.5%. Other family counts are unchanged. These cases are
+now regression evidence after fixes, not held-out confirmation or production accuracy. The
+remaining genuine misses stay visible in both reports. Exact token matching can also warn about
+valid derived counts such as `×2` and `×1`; the checker does not verify arithmetic or counting.
+
+The source check looks for short backtick spans and numeric tokens somewhere in the supplied
+files. Reversed relationships, incorrect negation, unquoted invented names,
+and quotes longer than 60 characters can escape it. Correct derived totals and other formatting changes can still trigger warnings. The extraction check recognizes certain identifiers left null, but does
+not compare a wrong non-null value. The injection check catches selected instruction patterns;
+benign quotations can trigger it and paraphrases can evade it. A warning is a reason to inspect
+the answer, not proof of hallucination or a successful attack.
+
+Tool checks reject malformed native `response.message.tool_calls` containers and nonstring
+function names, and flag duplicate provided names as ambiguous. They validate known names, JSON-object arguments, nested properties, required fields,
+additional properties, array items, JSON types, enums, constants, numeric bounds and string/array
+length bounds. Objects allow additional properties unless their schema forbids them. Unsupported
+keywords, including `$ref`, `pattern`, `format`, `oneOf` and `anyOf`, are explicitly unchecked.
+Malformed values for supported validation keywords also produce unchecked warnings.
+This is a bounded structural checker, not a complete JSON Schema implementation or a tool policy.
+A structurally valid call can still name the wrong city or perform an unauthorized action.
+
+Reproduce the report with no Ollama, downloads or inference:
+
+```sh
+node scripts/reliability/result-check-accuracy.mjs --out /tmp/result-check-accuracy.json
+node --test scripts/reliability/*.test.mjs
+```
+
+The output file must not already exist. The offline test compares the generated report to the
+versioned current artifact and checks that the baseline human labels remain unchanged and keeps known false positives and false negatives visible. After changing
+a detector, review changed labels and outcomes, regenerate into a new file, then replace the
+artifact deliberately. Other unit tests cover confidence bands, nearest-match reminders,
+SEARCH/REPLACE matches, padding, fallback receipts and vision heuristics; these are contracts,
+not measured factual accuracy of those families.
+
+## Public tool-calling smoke suite
+
+The [runner](../scripts/reliability/tool-calling.mjs) calls Ollama directly with the frozen
+[tool schemas, prompts and deterministic grader](../scripts/reliability/tool-cases.mjs). It tests:
+
+- Choosing a weather tool among alternatives and returning the tool result.
+- Integer arguments and a second assistant turn after arithmetic output.
+- Nested object and array arguments under closed schemas.
+- Avoiding tools for a plain response and asking for a missing city.
+- Declining an unavailable action without inventing a tool.
+- Treating an instruction inside a simulated tool result as untrusted data.
+
+No real tool action is performed. A simulated result is supplied only after the expected tool
+and exact argument values match. The grader requires the requested final text, so inspect the raw
+transcript when a semantically similar answer fails. Passing the seven public cases once is a
+smoke result, not a population reliability estimate, a safety guarantee, or a model ranking.
+The runner does not measure FreeLlama admission, routing, MCP transport, or physical CPU/GPU
+placement. Tool choice and argument columns are first-turn outcomes on expected-call cases;
+follow-up is conditional on an accepted first call. Complete-case success includes both turns.
+
+[Ollama's tool-calling contract](https://docs.ollama.com/capabilities/tool-calling) leaves execution
+to the caller: preserve the assistant message, append each `{role:"tool", tool_name, content}`
+result, and send the updated conversation with the tools. It also documents parallel calls,
+agent loops and streamed accumulation. The present suite measures only single-call first turns
+and one follow-up; it has no live evidence for parallel calls, longer dependent loops, streamed
+tool calls or real tool side effects. [The chat API](https://docs.ollama.com/api/chat) defines the
+native `message`, `done` and tool fields used by the grader.
+
+Each run records exact installed tags, manifest digests, sizes, capabilities, Ollama and Node
+versions, harness/case SHA-256 hashes, Git commit and dirty files, host memory/load before and
+after, residency, all request/response bodies, errors and every planned denominator. An Ollama
+manifest digest identifies the tag manifest; it is not a GGUF file hash. No retry is hidden.
+New runs also record an acquisition UUID and strict grader version. The `tool-strict-v2` grader
+rejects malformed call containers and requires a question or request for the missing city rather
+than accepting any city mention. It is still a bounded oracle, not full semantic verification.
+Legacy reports replay their original v1 grader so historical scores are not silently rewritten.
+Infrastructure errors and timeouts remain non-passes in complete-case denominators; cases not
+attempted after a stop remain visible. HTTP abort does not prove inference has stopped.
+
+The runner requires an exact installed tag that reports `tools` capability. It never pulls,
+unloads, deletes, or starts a service. Requests are sequential with context 4096, temperature 0,
+seed 42, `think:false`, at most 256 generated tokens and a 45-second request timeout by default.
+`keep_alive:"60s"` lets Ollama expire the warmed model naturally. Response bodies are capped at
+2 MiB. Inspect current available memory, residency and other workloads before starting; disk
+size does not establish runtime fit. Check final residency before a subsequent run, especially
+after a timeout.
+
+```sh
+# Read-only inventory; every model remains unmeasured.
+node scripts/reliability/tool-calling.mjs --inventory --out /tmp/tool-inventory
+# Explicit bounded inference on one exact installed tag; output directory must be new.
+node scripts/reliability/tool-calling.mjs --model 'EXACT_INSTALLED_TAG' --out /tmp/tool-run
+# A declared repeat count runs every case that many times without automatic retries.
+node scripts/reliability/tool-calling.mjs --model 'EXACT_INSTALLED_TAG' --repeats 3 --out /tmp/tool-repeat
+```
+
+`--endpoint`, `--timeout-ms` and `--max-tokens` override the defaults and are preserved in the
+manifest. Changing them changes the evaluation conditions. Keep all audit files when sharing a
+result: `manifest.json`, `trials.jsonl`, `summary.json`, `TABLE.md`, and `final-state.json`; failed
+preflight runs retain `fatal.json`. The offline runner tests use fake HTTP responses and prove
+harness contracts only.
+
+## Measured on 2026-11-14
+
+Ollama 0.40.15 on macOS arm64, Node v26.4.0, 32 GiB RAM and 12 logical CPUs. Each selected model
+ran the same seven public cases once, with identical harness/case hashes and the default request
+settings above. Both started with no resident Ollama models. Estimated available memory was
+11.9 GB before LFM3-VL and 6.7 GB before Bonsai; macOS pressure level was 1. Other development
+work ran on the host and swap was already in use, so these runs support no latency or resource
+comparison. Models expired naturally; no pull, explicit unload or deletion was issued.
+
+| Exact installed tag | Complete strict cases | Tool choice | Exact arguments | Follow-up | Evidence |
+|---|---:|---:|---:|---:|---|
+| `lfm3-vl:2b-q4_K_M` | 5/7 | 4/4 | 4/4 | 3/4 | [Raw run](assets/tool-calling-2026-11-14-lfm3-vl/trials.jsonl), [manifest](assets/tool-calling-2026-11-14-lfm3-vl/manifest.json) |
+| `bonsai-2:9b` | 6/7 | 4/4 | 4/4 | 3/4 | [Raw run](assets/tool-calling-2026-11-14-bonsai-9b/trials.jsonl), [manifest](assets/tool-calling-2026-11-14-bonsai-9b/manifest.json) |
+| `bonsai-2:27b` | — | — | — | — | Unmeasured |
+| `nimble-v3:9b-q4_K_M` | — | — | — | — | Unmeasured |
+| `lfm3-omni:2b-q4_K_M` | — | — | — | — | Unmeasured |
+| `hf.co/openbmb/MiniCPM-OCR-1.0-gguf:Q4_K_M` | — | — | — | — | Unmeasured |
+| `hf.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB:Q4_K_M` | — | — | — | — | Unmeasured |
+| `embeddinggemma-2:740m-q8_0` | — | — | — | — | Unmeasured |
+| `lfm3-audio:2b-q4_K_M` | — | — | — | — | Unmeasured |
+| `phonon:en` | — | — | — | — | Unmeasured |
+| `strands-decider:2b-nvfp4` | — | — | — | — | Unmeasured |
+| `nomic-embed-text:v1.5` | — | — | — | — | Unmeasured |
+
+The [generated full table](assets/tool-calling-table.md) includes every installed tag and its
+manifest digest. A dash means no trial was run; installed status and advertised tools support
+are not reliability evidence. Non-tool models remain unmeasured rather than receiving a score
+for a task they do not advertise.
+
+All four expected first tool calls and arguments were correct for both measured models. Both
+also avoided unnecessary calls in all three non-call cases. Strict weather continuation failed
+because LFM3-VL returned `WEATHER:18` and Bonsai returned `WEATHER:18°C`, while the prompt requested
+exactly `WEATHER:18C`. LFM3-VL's other failure was an appropriate prose refusal to delete a file
+instead of the requested exact `UNAVAILABLE`. These are format failures, not observed dangerous
+tool execution or wrong weather values. The prompt-in-tool-result case passed once per model;
+that does not establish general injection resistance. The small exposed suite cannot rank the
+models or justify broad reliability percentages.
+
+Regrade the raw responses and regenerate the combined table without inference:
+
+```sh
+node scripts/reliability/tool-report.mjs \
+  --run docs/assets/tool-calling-2026-11-14-lfm3-vl \
+  --run docs/assets/tool-calling-2026-11-14-bonsai-9b \
+  --out /tmp/tool-calling-table.md
+```
+
+The reporter checks recorded initial prompts, tool schemas, model tags and options against the
+frozen case definition and manifest, and verifies every assistant/tool-result continuation. A
+manifest-recorded `--max-tokens` override is allowed; an unexplained changed request is rejected.
+It replays the recorded strict grader, verifies summaries and rejects pooling incompatible
+harness/case hashes, runtime versions, request limits or strict grader versions. Copied directory
+paths cannot multiply evidence: duplicate acquisitions are rejected by recorded run identity.
+These checks validate retained transcripts; hashes and captured requests do not independently
+attest physical model execution, unchanged weights, host identity or an untampered acquisition.
+Ordinary offline tests reproduce the checked-in table from the raw artifacts.
+
+## Post-hoc interpretation of strict failures
+
+The separate [`tool-format-adjudication-v1` report](assets/tool-outcomes-v1.json) applies narrow
+deterministic rules to the same 14 retained responses. It preserves all 11 historical strict
+passes and identifies the three failures described above as format-only: the reference Celsius
+temperature with an omitted/degree suffix, or a whole-response explicit inability to delete files
+without a tool call. Wrong temperatures, different units, extra calls, contrary claims or text
+outside those rules remain unresolved. The rules were written after reading these failures;
+this is explanatory regression evidence, not a new inference, an independent semantic quality
+study, a replacement score or a finding that every strict pass is factually correct.
+
+```sh
+node scripts/reliability/tool-outcomes.mjs \
+  --run docs/assets/tool-calling-2026-11-14-lfm3-vl \
+  --run docs/assets/tool-calling-2026-11-14-bonsai-9b \
+  --out /tmp/tool-outcomes-v1.json
+```

package/docs/RUST_RESOURCE_AUDIT.md

@@ -0,0 +1,65 @@
+# Rust resource and panic audit
+
+The resource boundary now measures Windows physical-memory headroom and requires that observation
+by default. It also propagates telemetry reader-thread creation failures after killing and reaping
+the child process. Test assertions using `unwrap()` were retained.
+
+## Scope and findings
+
+The initial production-library scan used `cargo clippy -p freellama-core --lib -- -W clippy::unwrap_used
+-W clippy::expect_used -W clippy::await_holding_lock`. It found no production `unwrap()` or
+`await_holding_lock` diagnostics and 30 explicit `expect()` sites at that audit stage. This is a lint census, not proof
+that every panic or deadlock is impossible. Deep review covered resource sampling, reservation
+accounting, and the holder cache; additional lock spot checks covered raw admission, scope history,
+telemetry, and the SQLite store.
+
+| Boundary | Finding and disposition |
+|---|---|
+| [Resource governor](../packages/rust-core/src/platform/resources/governor.rs) | Its three `expect()` calls state private invariants: the constant default policy validates, a sampling handle is installed before it is awaited, and a sample is saved before the cache is read. The Tokio state mutex serializes these transitions. Keep the explicit invariants. |
+| Governor synchronous accounting mutex | All lock acquisitions recover poisoned guards. They protect generation/reservation updates and do not span an await. A fault-injection test poisons the lock and verifies both permit release and conservative reservation decisions. |
+| Governor sampling await | The Tokio mutex intentionally spans the shared sampling handle await so cancellation leaves one reusable in-flight sample. Existing cancellation and concurrent-reservation tests cover the affected behavior. A panicking sampler becomes `sampler_failed` with unknown memory and refused admission. |
+| [Bounded subprocess](../packages/rust-core/src/platform/resources/sampling.rs) | Reader creation previously used `thread::spawn`, which can panic after a child was started. It now uses fallible `Builder::spawn`; failure kills/reaps the child and returns the I/O error. Reader-buffer locks recover poison and guard only bounded buffer updates. Existing deadline, output-cap, and inherited-pipe tests pass. OS thread-exhaustion injection was not run. |
+| [Memory holders](../packages/rust-core/src/platform/resources/holders.rs) | The diagnostic cache recovers poisoned locks and releases its mutex before OS commands and `spawn_blocking` awaits. Holder output does not authorize admission. |
+| Store, telemetry, scope and raw-admission spot checks | Their synchronous mutex helpers recover poisoned guards. Inspected async paths release these guards before awaits. This does not establish every application-level lock ordering or runtime contention property. |
+
+## Windows memory contract
+
+[The Windows collector](../packages/rust-core/src/platform/resources/windows.rs) runs one
+unprivileged, noninteractive PowerShell CIM query with a 1.5-second command deadline and the shared
+64-KiB output cap. It accepts only successful command output containing one JSON object.
+[Microsoft defines](https://learn.microsoft.com/en-us/windows/win32/cimwin32prov/win32-operatingsystem)
+`TotalVisibleMemorySize` and `FreePhysicalMemory` in KiB. Conversion uses checked multiplication;
+the total must be positive and available memory must not exceed it. Missing, malformed, negative,
+fractional, overflowing, or inconsistent readings cannot authorize a load under `require-memory`.
+Observed zero remains zero.
+
+The collector does not invent CPU-load, OS-pressure, thermal, swap, or GPU observations. These stay
+unknown, so `require-all` holds work on Windows. Operators can explicitly select `best-effort` for
+unknown-memory admission. See [monitoring](MONITORING.md) for the receipt fields and policy.
+
+## Verification and limits
+
+`cargo test -p freellama-core --lib platform::resources` passed 59 tests with one live-host test
+ignored on the macOS development host. The new fixtures exercise valid Windows conversion,
+unknown metrics, bad/missing/overflowing input, zero headroom, policy defaults, poisoned accounting,
+and sampler panic. Those tests are grouped in
+[`resources/tests/windows.rs`](../packages/rust-core/src/platform/resources/tests/windows.rs) and
+[`resources/governor/audit_tests.rs`](../packages/rust-core/src/platform/resources/governor/audit_tests.rs);
+the existing resource contracts remain enabled in `resources/tests.rs`. The strict production-library check
+`cargo clippy -p freellama-core --lib -- -D clippy::unwrap_used -D clippy::await_holding_lock`
+passed. A subsequent local workspace check with
+`cargo clippy --workspace --all-targets --all-features -- -D warnings` also passed after engine work
+settled; that gate does not enable `expect_used` or prove every `expect()` safe.
+The explicit read-only live-host smoke also passed on macOS in 60 ms after the subprocess change.
+
+The [CI workflow](../.github/workflows/ci.yml) configures the actual PowerShell/CIM smoke on
+Windows, in addition to cross-platform parser fixtures. No completed hosted Windows result is
+recorded by this local audit. Actual Windows startup, CIM permissions, command timing, and hardware
+behavior need that run. Reproduce the read-only host smoke there with:
+
+```sh
+cargo test -p freellama-core --lib live_host_resource_observation -- --ignored --nocapture
+```
+
+No model was loaded, downloaded, or executed for these checks. The audit does not claim complete
+panic freedom, measured contention, or live Windows GPU qualification.

package/docs/SCOPES_AND_WARMING.md

@@ -5,6 +5,11 @@ model needs preparation before related work. FreeLlama manages these controls th
 admission and memory checks as task execution. The calling agent owns conversation instructions,
 task dependencies, and verification.
 
+Scope history is a core facility; the selected engine still validates every retained message.
+The [llama.cpp adapter](LLAMACPP.md) accepts only text messages and its fixed context contract.
+The warming, eviction, preload and adaptive keep-alive behavior in this guide applies to Ollama;
+external-engine warm and unload requests are refused.
+
 ## Ownership
 
 | State | Owner | Lifetime and meaning |
@@ -85,8 +90,9 @@ not append history.
 
 Limits reject an oversized transcript instead of silently deleting earlier instructions. If a task
 cannot fit, narrow its input, create a compact replacement scope, or fork a suitable snapshot. The
-caller owns any semantic summary and its verification. Stored history also contributes to managed
-context sizing; the selected model's context boundary still applies. Scoped image or audio input
+caller owns any semantic summary and its verification. Stored history also contributes to Ollama-managed
+context sizing; the selected model's context boundary still applies. llama.cpp uses its configured
+fixed window and does not verify exact input-token fit. Scoped image or audio input
 requires explicit `context_tokens` (`contextTokens` through MCP). Its context receipt estimates
 text and reports total multimodal token cost as unknown. Encoded media is retained in history,
 but its bytes are not counted as text tokens. Text plus the finite output reserve must fit the
@@ -226,8 +232,9 @@ The runtime file's `[preload]` table loads models without a caller asking:
   say which devices that Ollama process may allocate on, and nothing in FreeLlama verifies it
   (`gpu_capacity_scope` stays `host_aggregate_not_verified_for_backend`), so a speculative load
   could displace work on a pool it cannot see. The skip reason is `GPU device eligibility for this
-  Ollama backend is unverified` (`opportunistic_warm_skipped`). Unified-memory hosts and the CPU
-  backend, where host memory is the budget, still predict. Explicit warms, startup preloads and
+  Ollama backend is unverified` (`opportunistic_warm_skipped`). A local primary backend on a
+  unified-memory or CPU-only host can still predict using host memory as its budget. A dedicated
+  secondary CPU backend does not predict. Explicit warms, startup preloads and
   managed tasks on a discrete GPU run through ordinary admission.
 
 Each preload is a background warm under the usual admission, memory and pressure rules, and the
@@ -244,7 +251,7 @@ npx @octocodeai/freellama warm --model MODEL_TAG \
 
 ## Adaptive keep-alive
 
-When a managed request omits `keep_alive`, FreeLlama selects a finite duration from measured reuse
+When an Ollama-managed request omits `keep_alive`, FreeLlama selects a finite duration from measured reuse
 and load cost, within the runtime `[warming]` bounds. Explicit per-call values take precedence,
 including immediate unload (`"0"`) and indefinite residency (`"-1"`). Use
 `--keep-alive=-1` in the CLI so the negative value remains attached to its flag. The configured eviction policy and available

package/docs/TESTING.md

@@ -25,7 +25,7 @@ yarn build
 ```
 
 The root build compiles the host Rust release binary, NAPI addon, and MCP JavaScript bundles
-(server and research adapters). The native `.node` addon remains external to the JavaScript bundle because it is selected
+(server, the agent CLI behind `freellama run`, `chat`, `gateway`…, and research adapters). The native `.node` addon remains external to the JavaScript bundle because it is selected
 by platform triple at runtime. Published packages resolve it from an OS/CPU/libc-specific optional
 dependency rather than compiling Rust during installation.
 
@@ -61,6 +61,64 @@ lifecycle test is disabled unless the operator explicitly sets `FREELLAMA_TEST_P
 `qwen2.5:0.5b` and `FREELLAMA_TEST_PULL_SIZE_BYTES` to the exact reported download size. The test
 rechecks that size and reads every installed-model page before allowing the round trip.
 
+## Continuous integration
+
+[`.github/workflows/ci.yml`](../.github/workflows/ci.yml) configures the deterministic tiers on every push,
+every pull request and on demand (`gh workflow run CI`), on Ubuntu, macOS and Windows:
+`yarn install --immutable`, `yarn lint:rust` (Ubuntu), `yarn build`, `yarn typecheck`,
+and `yarn test:rust`. Ubuntu and macOS run the full `yarn test:js` and `yarn test:release`; the latter's fake
+executable fixtures require a POSIX shell. The native build must reproduce the committed TypeScript
+declarations. Windows additionally runs the read-only live host-resource observation to exercise
+PowerShell/CIM, beyond parser fixtures.
+
+Windows runs the workflow's explicit JavaScript allowlist: launcher and PowerShell installer,
+agent CLI, gateway/auth/TLS/response limits, job calibration/selection, and result/tool-call checker
+contracts. Their execution uses Node, local HTTP/TLS or PowerShell fixtures, not Unix shell tools.
+This is a portable-contract lane, not the full JavaScript suite: it excludes the POSIX research and
+coder execution/path suites, the shell installer, and the benchmark/reliability script suites.
+In particular, coder execution names `/bin/sh`; restricted research shell discovery and workspace
+path semantics remain POSIX-oriented. Git Bash being installed is not a native Windows adapter
+port. Schema/checker coverage does not qualify those adapters on Windows. Linux/macOS retain the
+complete suites; native Windows adapter support needs its own implementation and validation.
+The workflow selects Node 22 and the stable Rust channel; local development uses the compiler
+in `rust-toolchain.toml`. Inspect the hosted run's toolchain and results when qualifying a release.
+The configured matrix is not evidence that each lane has completed successfully; this change was
+inspected and its selected contracts run locally on macOS, not executed on native Windows. CI uses no
+Ollama and no models, so it does not replace inference or hardware qualification;
+run the live tiers before a release (`yarn verify:production`).
+
+The local macOS verification snapshot of 2026-11-15 passed the release build, TypeScript checks,
+strict Rust lint, 781 Rust tests, 1,474 JavaScript/evaluation checks and 51 packaging contracts.
+Three JavaScript tests were skipped and four Rust tests ignored. The 21 selected cookbook,
+packaged-resource and isolated-endpoint integration contracts also passed. This records local
+deterministic checks, not a completed `verify:production`, a hosted Linux/Windows run, or a new
+inference/hardware qualification. The [end-to-end review](OLLAMA_REVIEW.md#validation-of-this-revision)
+records the repaired baseline failures and the limits of these results.
+
+`yarn test:reliability` reproduces the labeled result-check accuracy report and exercises the
+tool-calling grader offline. It is included in `yarn test:js`. See [reliability evidence](RELIABILITY.md)
+for false positives, false negatives, live model results, raw transcripts and reproduction commands.
+The [Rust resource audit](RUST_RESOURCE_AUDIT.md) records the production panic/lock review and its limits.
+
+## Comparative benchmark
+
+Comparing FreeLlama with Ollama alone and other local stacks is a separate, manual programme:
+protocol, measured results and limits are in [Benchmark](BENCHMARK.md). The recorded 2026-11-14
+Apple M2 Pro run covers warm extraction, two- and four-client contention, and cold-request
+observations for one hash-verified model. llama.cpp's startup-loaded first request is marked
+non-comparable. Separate feature observations cover queue priority, workflow prompts,
+reused searches/sweeps, deferred jobs and memory refusal. These are historical results for the
+recorded builds, configurations and workloads; they do not qualify every current routing or engine
+path. The cross-stack memory-pressure comparison and agent-token meter remain unrun. See the
+[benchmark limits](BENCHMARK.md#limits) and
+[incomplete workflow-trial denominators](BENCHMARK.md#built-in-prompts-measured).
+
+`yarn test:bench` checks the sequential vendor runner, streamed timing, exact JSON grading and
+host-counter parsing using deterministic fixtures. It is included in `yarn test:js`. These checks
+require no model downloads or inference and do not qualify live performance. Use
+`node scripts/bench/preflight.mjs --hash-model` for the separate live prerequisite report and
+follow [`scripts/bench/README.md`](../scripts/bench/README.md) for vendor installation and execution.
+
 ## Model metadata contracts
 
 Model metadata unit contracts cover README extraction, complete tag pages, family-versus-variant
@@ -75,7 +133,7 @@ The fixture tests perform no inference, pulls, or deletions. See the
 
 ```bash
 cargo fmt --all --check
-cargo clippy --all-targets -- -D warnings
+cargo clippy --workspace --all-targets --all-features -- -D warnings
 ```
 
 Routing regressions run in the default Rust suite; none are hidden behind `#[ignore]`. Add a
@@ -95,6 +153,21 @@ The local-agent tier runs all context fitting, compaction, pagination, repeat-su
 strict action-shape contracts. It is included in `yarn test:all`, so adapter regressions cannot be
 missed by the root release matrix.
 
+Focused source modules keep contracts close to their owner: benchmark fixtures live in
+`src/model_bench/tests.rs`, profile-matching fixtures in `src/model_bench/provenance.rs`, Windows
+memory fixtures in `src/platform/resources/tests/windows.rs`, and panic/poison fault injection in
+`src/platform/resources/governor/audit_tests.rs` (paths relative to `packages/rust-core/`). These
+are ordinary enabled tests. Other resource tests remain in `resources/tests.rs`; this was a
+targeted split, not a claim that every large test file has been reorganized.
+
+Benchmark fixtures reject foreign hosts/endpoints and legacy reports, bind scores to artifact
+digests, and keep primary-endpoint throughput out of CPU-assigned models. They also cover
+existing-runner preservation and failure/identity checks before benchmark cleanup in
+`packages/rust-core/tests/model_bench_contract.rs`. The managed
+llama.cpp tests use a mock HTTP server; they establish its bounded text contract, not live
+llama-server performance or observed processor placement. See the
+[engine contract](LLAMACPP.md) and [machine calibration](MODEL_SELECTION.md#turn-measurements-into-routing-evidence).
+
 `yarn test:all` does not run formatting, Clippy, a release build, or package inspection. Use
 `yarn verify:production` for the complete local promotion gate.
 
@@ -144,12 +217,31 @@ Usage-ledger unit contracts cover ordered writes, bounded queue overflow, failed
 validation, replay, and flush behavior. `monitoring_contract` also checks that a blocked ledger path
 does not hold task completion. Persistence remains best effort, not a zero-loss promise.
 
-MCP `serve.test.ts` checks startup permission failure, retry, child-lifecycle isolation, and the
-background start at launch. The
+MCP `serve.test.ts` checks startup permission failure, retry, child-lifecycle isolation, the
+background start at launch, and the detached serve the agent commands start (log file, release once ready, log
+tail on an early exit). `agent-cli.test.ts` covers `freellama run` flag mapping and output, registry
+parity with the MCP server's tool list, virtual models, and the gateway's Ollama, OpenAI and Anthropic
+protocols, including a client that leaves mid-stream. The
 protocol integration contracts verify that spawn failure leaves the MCP connection usable and that
 fixture-backed library requests fail on unexpected product errors. External catalog availability
 is separate from these deterministic checks.
 
+`agent-cli-auth.test.ts` uses token-protected local HTTP fixtures to exercise terminal streams,
+residency/config reads, real-model lists, passthrough and virtual SSE. It also checks that absolute
+request URLs stay on the configured serve origin and generation redirects are refused.
+The HTTPS passthrough fixture requires a real TLS handshake and verifies that an untrusted local
+certificate returns 502 before the server receives HTTP credentials; it changes no TLS defaults.
+Response-budget fixtures count UTF-8 wire bytes, accept the exact limit, cancel oversized
+NDJSON/SSE/JSON and error bodies even without a newline, and fail after partial terminal output.
+They also require raw real-tag passthrough to remain byte-preserving above the rewrite budget.
+CLI regressions require `done:true` before reporting stream success, preserve explicit managed controls
+under `--stream`, and retain a deferred lookup's error and retryable exit status.
+Installer fixtures are platform-specific: `install-script.test.ts` exercises the POSIX installer
+on Unix, and `install-powershell.test.ts` exercises Windows checksum acceptance/refusal with local
+release files and a replaced HTTP cmdlet. Each suite skips the other OS; neither performs a
+network install or runs the fixture executable. Launcher tests select `.exe` on Windows so a
+built release is checked there rather than silently skipped. Windows execution still requires its lane.
+
 Launcher fixtures use `process.execPath` to avoid an extra PATH-resolved shell wrapper. After the
 MCP fixture's real executable-permission refusal, a test-only transport launches that exact fixture
 script through the same Node executable. Real child processes, HTTP listeners, signals, and exit
@@ -218,6 +310,18 @@ tier, and each writes a dated report to the git-ignored `.octocode/local-test/`.
 | `scripts/local-test/assist.mjs` | Issue to code, duplicates, triage and OCR on a real repository |
 | `scripts/local-test/media.mjs` | Speech round trip, multimodal image search, embedding comparison through MCP |
 | `scripts/local-test/octocode-agent.mjs` | A local model driving Octocode through `delegate_research` |
+| `scripts/local-test/workflows.mjs` | Each text workflow (summarize-text, summarize-log, summarize-diff, extract-json, explain-code, review-code, fix-bug) on each candidate model through `freellama run -w -m`. Graded on facts present, nothing invented, length, filler, exact JSON fields and the planted bug. Pass `--models a,b`, `--workflows x,y`, `--out file.json`. Re-run it after changing a workflow prompt or a catalog pick |
+
+Workflow eval of 2026-11-10 (Apple M2 Pro, 12 cases, so treat it as directional). The second
+run used the reworded prompts and the shipped sampling presets (the first forced temperature 0):
+
+| Workflow | `lfm3:1.7b` first → second | `bonsai-2:9b` first → second | Change made |
+|---|---|---|---|
+| summarize-text | 0/3 → 3/3 | 3/3 → 3/3 | No-padding prompt; `prefer: "capable"` |
+| summarize-log | 0/2 → 1/2 (one run looped to 1,688 words) | 2/2 → 2/2 | Line-by-line format, no-padding closing; already `prefer: "capable"` |
+| explain-code, review-code | 3/3 → 3/3 | 1/3 → 3/3 | Callers only when shown; quote the faulty code instead of line numbers the input lacks |
+| summarize-diff, extract-json, fix-bug | all pass | all pass | Diff risks only when the diff shows them |
+| **All** | **7/12 → 11/12** | **10/12 → 12/12** | |
 
 ## Verify release packages
 
@@ -231,10 +335,12 @@ yarn release:smoke            # packed-install smoke of the host platform packag
 
 Build every claimed CLI/native target in a clean environment, assemble `SHA256SUMS`, run the full
 deterministic suite, and inspect every platform package dry run before publishing explicitly.
-Publish the eight `@octocodeai/freellama-native-*` packages before `@octocodeai/freellama` and `@octocodeai/freellama-mcp-server`, all
+Publish the five `@octocodeai/freellama-native-*` packages listed in
+[`scripts/release-platforms.mjs`](../scripts/release-platforms.mjs) before `@octocodeai/freellama` and `@octocodeai/freellama-mcp-server`, all
 at the exact same version, then the crates with `cargo publish --workspace` (`--dry-run` first packages
 and compiles both crates from their published sources). A successful local pack dry run proves package contents; it does not
-prove registry publication or another hardware class.
+prove registry publication or another hardware class. Both release scripts require
+`packages/mcp/dist/agent-cli.js` (the agent commands the `freellama` launcher runs) in the MCP package, and the CLI package to depend on the MCP package of the same version.
 
 ## Test one layer while iterating
 

package/docs/TUNING.md

@@ -1,10 +1,10 @@
 # Make local models faster and lighter
 
-Moved from the root README.
+Start with an installed model selected using [machine-local calibration](MODEL_SELECTION.md#turn-measurements-into-routing-evidence)
+and held-out quality checks. Tuning cannot turn throughput evidence into correctness evidence.
+The Ollama settings below do not configure the separate [managed llama.cpp adapter](LLAMACPP.md).
 
-## Make local models faster and lighter
-
-Decode speed is fixed by the model and hardware. A task's real cost is mostly around decoding: cold
+Steady-state decode speed depends on the model, hardware and competing work. A task's real cost is mostly around decoding: cold
 loads, reloads, re-evaluating a prompt the runner already saw, unneeded thinking and output, and
 running out of memory. Best payoff first:
 
@@ -30,10 +30,14 @@ running out of memory. Best payoff first:
 3. **Spend output only where it helps.** Completion, tools, browser and vision tasks send
    `think:false` (Ollama otherwise enables thinking on capable models); coding, code-repair and
    long-context tasks send `think:"medium"` to thinking models. `stopOnRepetition` cut `glm-ocr` on 12
-   report strips from 12,948 to 6,452 output tokens with every label still read.
+   report strips from 12,948 to 6,452 output tokens with every label still read in that dated sample. Each text workflow
+   also caps its answer (`num_predict`, about 8 times the median its eval answers needed, 300–2,000
+   tokens) when thinking is off, and keeps `stopOnRepetition` on with it, so a model that loops stops
+   in seconds; a caller's own `num_predict` wins. End to end, workflows ran at 13–27 tokens/s on
+   `bonsai-2:9b` and Nemotron-3.5-Bolt here: the answer's length, not the prompt, sets the time.
 4. **Unload instead of swapping.** A load that does not fit waits instead of swapping, idle models are
-   unloaded under memory pressure, and `unload` frees a big model before a switch without ever
-   loading it.
+   unloaded under memory pressure according to operator policy. An agent may use `unload` for a
+   model it warmed or one the user named; it frees that model without loading it.
 5. **Shrink memory per runner.** Quantize the KV cache, keep `OLLAMA_NUM_PARALLEL=1` on a laptop
    (every slot multiplies KV memory by the context), and bound llama-server's hidden prompt cache
    with `LLAMA_ARG_CACHE_RAM`; see [Tune Ollama](#tune-ollama-for-local-models).
@@ -51,7 +55,7 @@ FreeLlama never changes Ollama's settings. These matter most, per Ollama's docs
 | Setting | What it does | Suggested start |
 |---|---|---|
 | `OLLAMA_NUM_PARALLEL` | Decoding slots per model, each with its own prompt cache; RAM grows with slots times context ([Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.mdx)). Small gain on Apple silicon; some engines run one request at a time ([MLX engine](https://github.com/ollama/ollama/issues/17280)) | `1`, raised only after measuring. FreeLlama's budget follows it |
-| `OLLAMA_FLASH_ATTENTION` + `OLLAMA_KV_CACHE_TYPE=q8_0` | A quantized KV cache roughly halves its memory and needs flash attention ([Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.mdx)); with `OLLAMA_FLASH_ATTENTION` unset, Ollama enables it automatically on supported backends. See [KV cache and flash attention](#kv-cache-and-flash-attention) | On, when the model's quality holds |
+| `OLLAMA_FLASH_ATTENTION` + `OLLAMA_KV_CACHE_TYPE=q8_0` | A quantized KV cache roughly halves its memory and needs flash attention ([Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.mdx)); with `OLLAMA_FLASH_ATTENTION` unset, Ollama enables it automatically on supported backends. See [KV cache and flash attention](#kv-cache-and-flash-attention) | `freellama doctor`'s conservative starter profile reports `f16`; switch to `q8_0` after checking quality on your workload (`scripts/local-test/workflows.mjs`) |
 | Context length | Ollama defaults to 4k below 24 GiB of VRAM and recommends at least 64k for agent and coding work ([context length](https://github.com/ollama/ollama/blob/main/docs/context-length.mdx)) | Size per task. FreeLlama keeps Ollama's default when it covers the prompt, otherwise sizes text tasks to the prompt; `contextTokens` overrides |
 | `OLLAMA_MAX_LOADED_MODELS` | How many runners stay resident. Unset and `0` mean 3 per GPU, then Ollama unloads by count | More than the model memory budget can hold, so the budget decides by size and reload cost. `1` reloads on every switch |
 | `keep_alive` | How long a runner stays loaded after use | Leave it out. FreeLlama picks a finite time from measured reuse |

package/docs/assets/benchmark-results-2026-11-14.html

@@ -0,0 +1,4 @@
+<!doctype html>
+<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>FreeLlama benchmark · pooled run orders</title><style>
+:root{color-scheme:light;--ink:#16242d;--muted:#53636d;--line:#d7dfe4;--accent:#15657b;--paper:#fff;--wash:#f3f6f8}*{box-sizing:border-box}body{margin:0;background:var(--wash);color:var(--ink);font:15px/1.5 system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif}main{max-width:1200px;margin:0 auto;padding:32px 28px 48px}header{padding-bottom:20px;border-bottom:2px solid var(--ink)}h1{font-size:30px;letter-spacing:-.03em;margin:0 0 10px}h2{font-size:20px;letter-spacing:-.02em;margin:0 0 8px}p{margin:8px 0}.subhead,.note,.muted,small{color:var(--muted)}.verdict{font-size:17px;font-weight:600}section{margin-top:28px}.table-scroll{overflow-x:auto;background:var(--paper);border:1px solid var(--line);margin-top:16px}table{border-collapse:collapse;width:100%;font-variant-numeric:tabular-nums}th,td{padding:14px 12px;text-align:left;border-bottom:1px solid var(--line);vertical-align:top}thead th{background:#edf2f5;font-size:13px;font-weight:600;white-space:nowrap}thead small{font-weight:400}tbody tr:last-child>*{border-bottom:0}tbody th{min-width:180px;font-weight:600}td{min-width:120px}.unknown{color:var(--muted);font-size:13px}small{display:block;font-size:12px;font-weight:400;margin-top:4px}.row-note{max-width:250px;line-height:1.35}.measured{font-weight:600}.note{font-size:13px;max-width:100ch}.controls{display:flex;align-items:center;gap:12px;margin:14px 0}select{font:inherit;color:var(--ink);background:var(--paper);border:1px solid #8b9ca8;border-radius:4px;padding:7px 10px}select:focus-visible{outline:3px solid #9bd2e1;outline-offset:2px}.empty{padding:16px;background:#edf2f5;color:var(--muted);border-left:3px solid #9aaab5}.plot{margin:14px 0;max-width:880px}.plot figcaption{font-size:13px;color:var(--muted);margin-bottom:8px}.plot svg{display:block;width:100%;max-width:760px;background:var(--paper);border:1px solid var(--line);padding:10px}.plot text,.samples text{font:12px system-ui,sans-serif;fill:var(--ink)}.plot text.muted,.samples text.muted{fill:var(--muted)}.plot .plot-name{font-weight:600}.plot .plot-value{font-variant-numeric:tabular-nums}.track{stroke:#d1dbe1;stroke-width:2}.point{fill:var(--accent);stroke:var(--paper);stroke-width:1}.samples{width:300px;height:58px;display:block}.sanity-table th,.sanity-table td{vertical-align:middle}.resource-line{display:flex;gap:36px;flex-wrap:wrap;border-top:1px solid var(--line);padding-top:16px;margin-top:20px}.resource-line dt{font-size:12px;color:var(--muted)}.resource-line dd{margin:3px 0 0;font-weight:600}details{margin-top:24px;color:var(--muted);font-size:13px}summary{cursor:pointer}details li{overflow-wrap:anywhere;margin:6px 0}footer{margin-top:28px;padding-top:16px;border-top:1px solid var(--line);font-size:12px;color:var(--muted)}[hidden]{display:none!important}@media(max-width:640px){main{padding:22px 16px}h1{font-size:25px}th,td{padding:11px 10px}.resource-line{gap:18px}body{font-size:14px}}
+</style></head><body><main><header><h1>FreeLlama benchmark · pooled run orders</h1><p class="verdict">2 run orders. No ranking or winner is inferred.</p><p class="subhead">Pooled p50/p95 use every verified measured trial from each order's raw trials.jsonl; warm-ups are excluded.</p></header><section aria-labelledby="pooled-heading"><h2 id="pooled-heading">Pooled across orders</h2><p class="note">Compare the gap between systems with the drift between orders before claiming a difference. — means Not measured.</p><div class="table-scroll"><table><thead><tr><th scope="col">System / per-order status</th><th scope="col">Orders pooled</th><th scope="col">Verified / attempted</th><th scope="col">Quality<small>% correct</small></th><th scope="col">Task time<small>ms · p50 / p95</small></th><th scope="col">TTFT<small>ms · p50 / p95</small></th></tr></thead><tbody><tr><th scope="row">Ollama alone<small>run-forward: Complete · run-reversed: Complete</small></th><td>2 of 2</td><td>60 / 60</td><td class="measured">100</td><td class="measured">271.96 / 289.82<small>n=60</small></td><td class="measured">93.35 / 97.29<small>n=60</small></td></tr><tr><th scope="row">FreeLlama<small>run-forward: Complete · run-reversed: Complete</small></th><td>2 of 2</td><td>60 / 60</td><td class="measured">100</td><td class="measured">273.73 / 302.84<small>n=60</small></td><td class="unknown"><span class="missing" role="img" aria-label="Not measured">—</span></td></tr><tr><th scope="row">FreeLlama (streamed)<small>run-forward: Complete · run-reversed: Complete</small></th><td>2 of 2</td><td>60 / 60</td><td class="measured">100</td><td class="measured">272.95 / 298.87<small>n=60</small></td><td class="measured">94.62 / 97.31<small>n=60</small></td></tr><tr><th scope="row">llama.cpp<small>run-forward: Complete · run-reversed: Complete</small></th><td>2 of 2</td><td>60 / 60</td><td class="measured">100</td><td class="measured">335.79 / 375.13<small>n=60</small></td><td class="measured">128.02 / 144.29<small>n=60</small></td></tr><tr><th scope="row">llama-swap<small>run-forward: Complete · run-reversed: Complete</small></th><td>2 of 2</td><td>60 / 60</td><td class="measured">100</td><td class="measured">331.83 / 371.27<small>n=60</small></td><td class="measured">129.73 / 137.92<small>n=60</small></td></tr><tr><th scope="row">LocalAI<small>run-forward: Complete · run-reversed: Complete</small></th><td>2 of 2</td><td>60 / 60</td><td class="measured">100</td><td class="measured">801.76 / 910.41<small>n=60</small></td><td class="measured">538.67 / 610.52<small>n=60</small></td></tr><tr><th scope="row">oMLX<small>run-forward: Not run · no raw trials · run-reversed: Not run · no raw trials</small></th><td>0 of 2</td><td>0 / 0</td><td class="unknown"><span class="missing" role="img" aria-label="Not measured">—</span></td><td class="unknown"><span class="missing" role="img" aria-label="Not measured">—</span></td><td class="unknown"><span class="missing" role="img" aria-label="Not measured">—</span></td></tr></tbody></table></div></section><section aria-labelledby="order-0"><h2 id="order-0">Order 1: run-forward</h2><p class="subhead">2026-11-14T15:49:12.314Z · S1 warm sequential generation; FreeLlama raw admission and managed completion reported separately</p><div class="table-scroll"><table><thead><tr><th scope="col">System / outcome</th><th scope="col" title="Exact-case pass rate; all measured attempts remain in the denominator.">Quality<small>% correct</small></th><th scope="col" title="Client wall time for verified answers. Failures remain in outcome counts.">Task time<small>ms · median</small></th><th scope="col" title="Verified completions divided by the entire measured request window.">Throughput<small>verified tasks/min</small></th><th scope="col" title="All measured local prompt and output tokens, including failed effort. Requires complete token coverage. Paid-agent cost is separate.">Local tokens<small>tokens / verified task</small></th><th scope="col" title="Whole-host CPU capacity from cumulative OS counters; includes other applications.">Host CPU<small>% busy · mean</small></th><th scope="col" title="Lowest reported available host memory. This is not process RSS; missing counters remain unknown.">Memory<small>GiB available · min</small></th></tr></thead><tbody><tr><th scope="row">Ollama alone<small>Complete · n=30 · 30 verified</small><small>Cohort: gguf · declared</small></th><td class="measured" title="Exact-case pass rate; all measured attempts remain in the denominator.">100<small>n=30</small></td><td class="measured" title="Client wall time for verified answers. Failures remain in outcome counts.">267.82<small>p95 280.22 ms · n=30</small></td><td class="measured" title="Verified completions divided by the entire measured request window.">221.39<small>n=30</small></td><td class="measured" title="All measured local prompt and output tokens, including failed effort. Requires complete token coverage. Paid-agent cost is separate.">97<small>n=30</small></td><td class="measured" title="Coverage 100%">37.2<small>n=28</small></td><td class="measured" title="macOS FreeLlama estimate (free + bounded reclaimable), not a kernel value">5.88<small>n=28</small></td></tr><tr><th scope="row">FreeLlama<small>Complete · n=30 · 30 verified</small><small>Cohort: gguf · declared</small></th><td class="measured" title="Exact-case pass rate; all measured attempts remain in the denominator.">100<small>n=30</small></td><td class="measured" title="Client wall time for verified answers. Failures remain in outcome counts.">273.08<small>p95 290.39 ms · n=30</small></td><td class="measured" title="Verified completions divided by the entire measured request window.">217.48<small>n=30</small></td><td class="measured" title="All measured local prompt and output tokens, including failed effort. Requires complete token coverage. Paid-agent cost is separate.">97<small>n=30</small></td><td class="measured" title="Coverage 100%">33<small>n=29</small></td><td class="measured" title="macOS FreeLlama estimate (free + bounded reclaimable), not a kernel value">5.99<small>n=29</small></td></tr><tr><th scope="row">FreeLlama (streamed)<small>Complete · n=30 · 30 verified</small><small>Cohort: gguf · declared</small></th><td class="measured" title="Exact-case pass rate; all measured attempts remain in the denominator.">100<small>n=30</small></td><td class="measured" title="Client wall time for verified answers. Failures remain in outcome counts.">272.73<small>p95 285.25 ms · n=30</small></td><td class="measured" title="Verified completions divided by the entire measured request window.">217.46<small>n=30</small></td><td class="measured" title="All measured local prompt and output tokens, including failed effort. Requires complete token coverage. Paid-agent cost is separate.">97<small>n=30</small></td><td class="measured" title="Coverage 100%">34.09<small>n=28</small></td><td class="measured" title="macOS FreeLlama estimate (free + bounded reclaimable), not a kernel value">5.91<small>n=28</small></td></tr><tr><th scope="row">llama.cpp<small>Complete · n=30 · 30 verified</small><small>Cohort: gguf · declared</small></th><td class="measured" title="Exact-case pass rate; all measured attempts remain in the denominator.">100<small>n=30</small></td><td class="measured" title="Client wall time for verified answers. Failures remain in outcome counts.">311.03<small>p95 346 ms · n=30</small></td><td class="measured" title="Verified completions divided by the entire measured request window.">188.23<small>n=30</small></td><td class="measured" title="All measured local prompt and output tokens, including failed effort. Requires complete token coverage. Paid-agent cost is separate.">98<small>n=30</small></td><td class="measured" title="Coverage 100%">7.13<small>n=27</small></td><td class="measured" title="macOS FreeLlama estimate (free + bounded reclaimable), not a kernel value">5.55<small>n=27</small></td></tr><tr><th scope="row">llama-swap<small>Complete · n=30 · 30 verified</small><small>Cohort: gguf · declared</small></th><td class="measured" title="Exact-case pass rate; all measured attempts remain in the denominator.">100<small>n=30</small></td><td class="measured" title="Client wall time for verified answers. Failures remain in outcome counts.">331.09<small>p95 379.07 ms · n=30</small></td><td class="measured" title="Verified completions divided by the entire measured request window.">178.89<small>n=30</small></td><td class="measured" title="All measured local prompt and output tokens, including failed effort. Requires complete token coverage. Paid-agent cost is separate.">98<small>n=30</small></td><td class="measured" title="Coverage 100%">21.59<small>n=31</small></td><td class="measured" title="macOS FreeLlama estimate (free + bounded reclaimable), not a kernel value">3.83<small>n=31</small></td></tr><tr><th scope="row">LocalAI<small>Complete · n=30 · 30 verified</small><small>Cohort: gguf · declared</small></th><td class="measured" title="Exact-case pass rate; all measured attempts remain in the denominator.">100<small>n=30</small></td><td class="measured" title="Client wall time for verified answers. Failures remain in outcome counts.">851.17<small>p95 936.1 ms · n=30</small></td><td class="measured" title="Verified completions divided by the entire measured request window.">71.42<small>n=30</small></td><td class="measured" title="All measured local prompt and output tokens, including failed effort. Requires complete token coverage. Paid-agent cost is separate.">97<small>n=30</small></td><td class="measured" title="Coverage 100%">47.52<small>n=67</small></td><td class="measured" title="macOS FreeLlama estimate (free + bounded reclaimable), not a kernel value">4.39<small>n=67</small></td></tr><tr><th scope="row">oMLX<small>Not run · n=0</small></th><td class="unknown" title="Exact-case pass rate; all measured attempts remain in the denominator."><span class="missing" role="img" aria-label="Not measured">—</span></td><td class="unknown" title="Client wall time for verified answers. Failures remain in outcome counts."><span class="missing" role="img" aria-label="Not measured">—</span></td><td class="unknown" title="Verified completions divided by the entire measured request window."><span class="missing" role="img" aria-label="Not measured">—</span></td><td class="unknown" title="Incomplete or absent token coverage"><span class="missing" role="img" aria-label="Not measured">—</span></td><td class="unknown" title="Whole-host CPU capacity from cumulative OS counters; includes other applications."><span class="missing" role="img" aria-label="Not measured">—</span></td><td class="unknown" title="Lowest reported available host memory. This is not process RSS; missing counters remain unknown."><span class="missing" role="img" aria-label="Not measured">—</span></td></tr></tbody></table></div></section><section aria-labelledby="order-1"><h2 id="order-1">Order 2: run-reversed</h2><p class="subhead">2026-11-14T15:50:37.982Z · S1 warm sequential generation; FreeLlama raw admission and managed completion reported separately</p><div class="table-scroll"><table><thead><tr><th scope="col">System / outcome</th><th scope="col" title="Exact-case pass rate; all measured attempts remain in the denominator.">Quality<small>% correct</small></th><th scope="col" title="Client wall time for verified answers. Failures remain in outcome counts.">Task time<small>ms · median</small></th><th scope="col" title="Verified completions divided by the entire measured request window.">Throughput<small>verified tasks/min</small></th><th scope="col" title="All measured local prompt and output tokens, including failed effort. Requires complete token coverage. Paid-agent cost is separate.">Local tokens<small>tokens / verified task</small></th><th scope="col" title="Whole-host CPU capacity from cumulative OS counters; includes other applications.">Host CPU<small>% busy · mean</small></th><th scope="col" title="Lowest reported available host memory. This is not process RSS; missing counters remain unknown.">Memory<small>GiB available · min</small></th></tr></thead><tbody><tr><th scope="row">Ollama alone<small>Complete · n=30 · 30 verified</small><small>Cohort: gguf · declared</small></th><td class="measured" title="Exact-case pass rate; all measured attempts remain in the denominator.">100<small>n=30</small></td><td class="measured" title="Client wall time for verified answers. Failures remain in outcome counts.">273.99<small>p95 300.95 ms · n=30</small></td><td class="measured" title="Verified completions divided by the entire measured request window.">195.94<small>n=30</small></td><td class="measured" title="All measured local prompt and output tokens, including failed effort. Requires complete token coverage. Paid-agent cost is separate.">97<small>n=30</small></td><td class="measured" title="Coverage 100%">34.41<small>n=10</small></td><td class="measured" title="macOS FreeLlama estimate (free + bounded reclaimable), not a kernel value">4.31<small>n=10</small></td></tr><tr><th scope="row">FreeLlama<small>Complete · n=30 · 30 verified</small><small>Cohort: gguf · declared</small></th><td class="measured" title="Exact-case pass rate; all measured attempts remain in the denominator.">100<small>n=30</small></td><td class="measured" title="Client wall time for verified answers. Failures remain in outcome counts.">275.04<small>p95 344.78 ms · n=30</small></td><td class="measured" title="Verified completions divided by the entire measured request window.">194.95<small>n=30</small></td><td class="measured" title="All measured local prompt and output tokens, including failed effort. Requires complete token coverage. Paid-agent cost is separate.">97<small>n=30</small></td><td class="measured" title="Coverage 100%">33.85<small>n=11</small></td><td class="measured" title="macOS FreeLlama estimate (free + bounded reclaimable), not a kernel value">4.5<small>n=11</small></td></tr><tr><th scope="row">FreeLlama (streamed)<small>Complete · n=30 · 30 verified</small><small>Cohort: gguf · declared</small></th><td class="measured" title="Exact-case pass rate; all measured attempts remain in the denominator.">100<small>n=30</small></td><td class="measured" title="Client wall time for verified answers. Failures remain in outcome counts.">273.16<small>p95 300.01 ms · n=30</small></td><td class="measured" title="Verified completions divided by the entire measured request window.">215.49<small>n=30</small></td><td class="measured" title="All measured local prompt and output tokens, including failed effort. Requires complete token coverage. Paid-agent cost is separate.">97<small>n=30</small></td><td class="measured" title="Coverage 100%">35.24<small>n=10</small></td><td class="measured" title="macOS FreeLlama estimate (free + bounded reclaimable), not a kernel value">4.55<small>n=10</small></td></tr><tr><th scope="row">llama.cpp<small>Complete · n=30 · 30 verified</small><small>Cohort: gguf · declared</small></th><td class="measured" title="Exact-case pass rate; all measured attempts remain in the denominator.">100<small>n=30</small></td><td class="measured" title="Client wall time for verified answers. Failures remain in outcome counts.">345.2<small>p95 397.39 ms · n=30</small></td><td class="measured" title="Verified completions divided by the entire measured request window.">168.69<small>n=30</small></td><td class="measured" title="All measured local prompt and output tokens, including failed effort. Requires complete token coverage. Paid-agent cost is separate.">98<small>n=30</small></td><td class="measured" title="Coverage 100%">19.09<small>n=11</small></td><td class="measured" title="macOS FreeLlama estimate (free + bounded reclaimable), not a kernel value">5.38<small>n=11</small></td></tr><tr><th scope="row">llama-swap<small>Complete · n=30 · 30 verified</small><small>Cohort: gguf · declared</small></th><td class="measured" title="Exact-case pass rate; all measured attempts remain in the denominator.">100<small>n=30</small></td><td class="measured" title="Client wall time for verified answers. Failures remain in outcome counts.">331.83<small>p95 371.27 ms · n=30</small></td><td class="measured" title="Verified completions divided by the entire measured request window.">177.61<small>n=30</small></td><td class="measured" title="All measured local prompt and output tokens, including failed effort. Requires complete token coverage. Paid-agent cost is separate.">98<small>n=30</small></td><td class="measured" title="Coverage 100%">34.43<small>n=12</small></td><td class="measured" title="macOS FreeLlama estimate (free + bounded reclaimable), not a kernel value">4.23<small>n=12</small></td></tr><tr><th scope="row">LocalAI<small>Complete · n=30 · 30 verified</small><small>Cohort: gguf · declared</small></th><td class="measured" title="Exact-case pass rate; all measured attempts remain in the denominator.">100<small>n=30</small></td><td class="measured" title="Client wall time for verified answers. Failures remain in outcome counts.">751.03<small>p95 886.6 ms · n=30</small></td><td class="measured" title="Verified completions divided by the entire measured request window.">79.96<small>n=30</small></td><td class="measured" title="All measured local prompt and output tokens, including failed effort. Requires complete token coverage. Paid-agent cost is separate.">97<small>n=30</small></td><td class="measured" title="Coverage 100%">32.05<small>n=26</small></td><td class="measured" title="macOS FreeLlama estimate (free + bounded reclaimable), not a kernel value">4.07<small>n=26</small></td></tr><tr><th scope="row">oMLX<small>Not run · n=0</small></th><td class="unknown" title="Exact-case pass rate; all measured attempts remain in the denominator."><span class="missing" role="img" aria-label="Not measured">—</span></td><td class="unknown" title="Client wall time for verified answers. Failures remain in outcome counts."><span class="missing" role="img" aria-label="Not measured">—</span></td><td class="unknown" title="Verified completions divided by the entire measured request window."><span class="missing" role="img" aria-label="Not measured">—</span></td><td class="unknown" title="Incomplete or absent token coverage"><span class="missing" role="img" aria-label="Not measured">—</span></td><td class="unknown" title="Whole-host CPU capacity from cumulative OS counters; includes other applications."><span class="missing" role="img" aria-label="Not measured">—</span></td><td class="unknown" title="Lowest reported available host memory. This is not process RSS; missing counters remain unknown."><span class="missing" role="img" aria-label="Not measured">—</span></td></tr></tbody></table></div></section><footer>Evidence report. No ranking, aggregate score, estimated result, or winner is inferred. This file contains no external requests or dependencies.</footer></main><script id="benchmark-data" type="application/json">{"title":"FreeLlama benchmark · pooled run orders","orders":["run-forward","run-reversed"],"pooled":[{"id":"ollama","name":"Ollama alone","orders":[{"label":"run-forward","vendor_id":"ollama","status":"complete","raw_trials":30},{"label":"run-reversed","vendor_id":"ollama","status":"complete","raw_trials":30}],"orders_pooled":2,"attempted":60,"verified":60,"quality":100,"task_time":{"n":60,"p50":271.9590829999943,"p95":289.8182919999963},"ttft":{"n":60,"p50":93.3453329999902,"p95":97.28612500000008}},{"id":"freellama","name":"FreeLlama","orders":[{"label":"run-forward","vendor_id":"freellama","status":"complete","raw_trials":30},{"label":"run-reversed","vendor_id":"freellama","status":"complete","raw_trials":30}],"orders_pooled":2,"attempted":60,"verified":60,"quality":100,"task_time":{"n":60,"p50":273.7295000000013,"p95":302.83750000000146},"ttft":{"n":0,"p50":null,"p95":null}},{"id":"freellama-stream","name":"FreeLlama (streamed)","orders":[{"label":"run-forward","vendor_id":"freellama-stream","status":"complete","raw_trials":30},{"label":"run-reversed","vendor_id":"freellama-stream","status":"complete","raw_trials":30}],"orders_pooled":2,"attempted":60,"verified":60,"quality":100,"task_time":{"n":60,"p50":272.95179199999984,"p95":298.87129200000345},"ttft":{"n":60,"p50":94.6217500000057,"p95":97.30845799999952}},{"id":"llama-cpp","name":"llama.cpp","orders":[{"label":"run-forward","vendor_id":"llama-cpp","status":"complete","raw_trials":30},{"label":"run-reversed","vendor_id":"llama-cpp","status":"complete","raw_trials":30}],"orders_pooled":2,"attempted":60,"verified":60,"quality":100,"task_time":{"n":60,"p50":335.78712500000256,"p95":375.12854200000584},"ttft":{"n":60,"p50":128.02329100000497,"p95":144.29299999999785}},{"id":"llama-swap","name":"llama-swap","orders":[{"label":"run-forward","vendor_id":"llama-swap","status":"complete","raw_trials":30},{"label":"run-reversed","vendor_id":"llama-swap","status":"complete","raw_trials":30}],"orders_pooled":2,"attempted":60,"verified":60,"quality":100,"task_time":{"n":60,"p50":331.8255419999987,"p95":371.2747079999972},"ttft":{"n":60,"p50":129.73000000001048,"p95":137.92079199999716}},{"id":"localai","name":"LocalAI","orders":[{"label":"run-forward","vendor_id":"localai","status":"complete","raw_trials":30},{"label":"run-reversed","vendor_id":"localai","status":"complete","raw_trials":30}],"orders_pooled":2,"attempted":60,"verified":60,"quality":100,"task_time":{"n":60,"p50":801.7590839999993,"p95":910.4146250000049},"ttft":{"n":60,"p50":538.673375000013,"p95":610.5174159999879}},{"id":"omlx","name":"oMLX","orders":[{"label":"run-forward","vendor_id":null,"status":"not_run","raw_trials":null},{"label":"run-reversed","vendor_id":null,"status":"not_run","raw_trials":null}],"orders_pooled":0,"attempted":0,"verified":0,"quality":null,"task_time":{"n":0,"p50":null,"p95":null},"ttft":{"n":0,"p50":null,"p95":null}}]}</script></body></html>