@octocodeai/freellama-mcp-server 0.1.0 → 0.5.0

package/docs/dev/FINDINGS_AND_POSITIONING.md

@@ -0,0 +1,290 @@
+# FreeLlama findings and open-source comparison
+
+Reviewed on October 3, 2026, for developers and local-AI operators evaluating the `0.2.1` candidate.
+
+```mermaid
+flowchart LR
+    Caller["Agent selects task controls and optional scope"] --> Core["FreeLlama qualifies model and checks resources"]
+    Core -->|"admitted"| Engine["Ollama loads models and executes inference"]
+    Core -->|"held"| Wait["Bounded queue or refusal"]
+    Wait -->|"resources recover"| Core
+    Engine --> Receipt["Routing, timing, and placement receipt"]
+    Receipt --> History["Commit successful scoped history"]
+```
+
+The diagram describes managed tasks. Raw Ollama API requests retain the primary-backend compatibility path.
+
+**Feature and logic assessment: 8.5/10 for local Ollama agent workflows.**
+The [feature assessment](#feature-and-logic-assessment) excludes speed, utilization, thermal measurements, packaging, and release readiness.
+The earlier evidence-based control-plane assessment remains 8/10 under its broader rubric.
+Real inference throughput, hardware efficiency, and sustained thermal behavior remain unqualified for this candidate on the latest Mac.
+This review combines repository contracts, retained local receipts, an independent SOL examination, and primary project documentation.
+It does not establish a speed ranking among projects.
+
+## What the findings establish
+
+| Finding | Established behavior | Evidence boundary |
+|---|---|---|
+| Model discovery and qualification | FreeLlama checks installed models, capabilities, policy, and requested controls. | Model names alone do not establish fit, placement, or task quality. |
+| Preview and execution | A preview returns a decision without submitting inference or reserving capacity. Execution checks admission again. | A preview does not reserve later admission. |
+| Resource control | Managed tasks use weighted admission, bounded queues, deadlines, cancellation, and host resource holds. | This controls submitted work; it does not schedule hardware kernels or prove optimal utilization. |
+| Per-task controls | Callers select model, task profile, runtime controls, priority, and placement requirements through supported contracts. | Ollama and its runners retain final execution authority. |
+| CPU/GPU routing | Explicit assignments can direct eligible tasks to separate Ollama backends. Returned receipts distinguish configuration from observed residency. | Overlap remains unqualified for this candidate on the latest Mac; an older M4 measurement is summarized below. |
+| Placement feedback | Runtime feedback uses bounded samples and comparable work units before changing a placement preference. | These checks do not establish optimal placement for every task or machine. |
+| Independent work | Batches dispatch caller-declared independent tasks with bounded fairness. | FreeLlama does not infer task dependencies from prompts. |
+| Scoped history | Scope IDs retain managed chat history, support forks, and reject stale revisions. Successful scoped execution commits history within storage limits. | Scope storage is process-local and does not own an engine KV cache. |
+| Model affinity and warming | Sessions retain model affinity. Managed warming checks resources and requests residency from Ollama. | Cached context, future admission, and placement require separate checks. |
+| Research delegation | Bounded adapters provide read-only code lookup, pagination, context fitting, repair, and repeat suppression. | The caller retains decomposition, judgment, mutation authority, and verification. An escalation result must be discarded. |
+| Compatibility | Native Ollama APIs pass through to the primary upstream. | Managed routing contracts apply to managed calls. |
+
+The architecture is coherent for agents that must decide whether local delegation is admissible.
+Its value lies in explicit decisions, bounded execution, and inspectable evidence.
+The reviewed receipts do not establish faster decoding or a universal model-quality improvement.
+
+See [Architecture](../ARCHITECTURE.md), [CPU/GPU routing](../CPU_GPU_ROUTING.md),
+[Scopes and warming](../SCOPES_AND_WARMING.md), and [MCP contracts](../../packages/mcp/README.md).
+
+## Warm models and separate task context
+
+Warm models and scoped history solve separate problems.
+Ollama supports preloading and `keep_alive` to retain a loaded model. [Ollama FAQ](https://docs.ollama.com/faq)
+FreeLlama adds managed resource checks and receipts around those operations.
+
+A scope retains application-level history for one task or related tasks.
+A session records model affinity.
+Neither handle reserves an Ollama runner or owns its KV cache.
+Scope forks preserve independent histories, while revision checks prevent stale writes.
+Model switching can reuse retained history without guaranteeing engine cache reuse.
+
+These features can reduce repeated caller setup and cold loading for related work.
+Their net benefit remains workload-dependent: retained history consumes context, and warm models consume memory.
+Measure cold loading separately from warm execution before claiming a throughput improvement.
+Use [Scopes and warming](../SCOPES_AND_WARMING.md) for the operation contract.
+
+## Repairs and audit findings
+
+| Area | Completed repair or examination | Remaining limit |
+|---|---|---|
+| Research confinement | Confirmed writing and outside-workspace escapes were repaired. Valid quoted `jq`, bounded `sed`, and output-only `xargs` fixtures passed independently. | The checks are not a general proof for every internal behavior of each allowed program. |
+| Agent context | Full observations remain available for pagination. Context fitting preserves pinned content, compacts older observations, and suppresses repeated execution. | Task accuracy must still be measured with representative questions and installed models. |
+| MCP instructions | Contracts clarify preview payloads, error handling, opaque cursors, scope revisions, lifecycle approval, and research escalation. | The discovery-size gate measures a character-based estimate, not billed tokens or model reliability. |
+| Health helper | Checks distinguish intentional ephemeral feedback, resource holds, and authenticated remote transport from contract errors. | Model metadata age does not establish last use. |
+| Dependencies | Compatible updates resolved six reported advisories across three transitive dependencies during the audit. | Audit results describe the checked dependency set and available advisory database at that time. |
+| Package contents | Gates check runtime files, adapters, both license texts, versions, and native artifacts across ten public npm packages. | Automatic optional-native selection from the published registry remains unqualified for this unpublished candidate. |
+| Artifact assembly | Assembly validates all eight artifact pairs before replacing outputs and rejects unsafe output paths. | Compilation and binary headers do not prove foreign runtime execution. |
+| Native artifacts | Two stale musl addons were rejected and replaced. All sixteen candidate artifacts received build, timestamp, architecture, and hash checks. | Foreign runtime and driver checks remain pending. |
+| Registry gate | All ten exact `0.2.1` versions were absent during the read-only registry check. | The check neither reserves versions nor establishes account publication permissions. |
+| Release workflow | CI precedes artifact builds; target checks load native addons, including an Alpine musl check. | Changed workflows were not executed remotely for this working-tree candidate. |
+| Examples | The installer passed controlled checksum fixtures. The RAG example passed indexing and query plumbing with synthetic vectors. | Remote installation and real retrieval quality remain separate checks. |
+
+Dependency resolutions changed to `fast-uri` 3.1.8, `hono` 4.13.12, and `ip-address` 10.7.3.
+The final production and fresh-consumer audits reported zero advisories at that time.
+
+The captured MCP surface contained eleven tools and 17,981 characters of instructions and tool definitions.
+The existing gate estimated 4,495 tokens, compared with a baseline of 4,498.
+This small reduction does not establish a meaningful inference or reliability improvement.
+
+One full validation attempt recorded a ten-second launcher timeout.
+Direct binary and launcher reruns completed in 31–61 ms, and the unchanged full validation passed afterward.
+The timeout cause remains unclassified; repeated startup measurements are still needed.
+
+An extra MCP confidence preflight remains a possible optimization.
+It also protects older servers, so removal requires a compatibility decision and measured latency evidence.
+Execution revalidation, resource sampling, and permit-release handoffs remain necessary boundaries.
+
+See [Testing](../TESTING.md), [Production](../PRODUCTION.md), and [Release procedure](../../RELEASE.md) for their owning contracts.
+
+## Historical candidate verification
+
+These results describe the retained prepublication audit snapshot, not a fresh validation of every subsequent repository edit.
+Later logo and documentation changes occurred after the captured build and packed-consumer checks.
+The new performance guide was absent from the examined package.
+Run the current release gates again before publishing the changed candidate.
+
+| Check | Captured result |
+|---|---|
+| Rust formatting, strict Clippy, release build, and native build | Passed |
+| Rust tests | 386 passed |
+| JavaScript unit tests | 152 passed |
+| Python adapter and health-helper tests | 58 passed |
+| Context and pagination assertions | 94 passed |
+| Hardware harness contracts | 77 passed |
+| Package and registry contracts | 40 passed |
+| MCP integration | 111 passed; two real-inference cases skipped |
+| End-to-end tests | Seven passed; four lifecycle or inference cases skipped |
+| Separate control-timing and telemetry checks | Four passed |
+| Packed consumer | Nineteen operation checks; thirty-six exact source-to-installed file matches |
+
+Test counts establish exercised contracts, not physical performance or universal feature correctness.
+Controlled upstream timings do not qualify real inference throughput.
+The packed consumer installed the matching host native package explicitly because the candidate was unpublished.
+
+An independent SOL examiner used the actual SDK with the packed MCP server.
+It checked eleven tools, fifteen documentation resources, diagnosis, status, and empty job inventory.
+All fourteen captured operator guides matched their source bytes at examination time.
+The owned child process exited after the exam.
+
+| Operation | One measured duration |
+|---|---:|
+| SDK initialization | 104.79 ms |
+| Tool discovery | 10.55 ms |
+| Resource discovery | 1.04 ms |
+| Individual resource reads | 0.42–1.31 ms |
+| Diagnosis summary | 292.82 ms |
+| Status | 39.20 ms |
+| Job inventory | 1.23 ms |
+| Complete examination | 467.70 ms |
+
+These durations describe one control-plane examination. They do not establish sustained latency or model throughput.
+
+## Earlier measured workloads
+
+Earlier guides retain physical results from other snapshots and workloads.
+They establish feasibility within their measured limits, not a fresh performance qualification for this candidate or Mac.
+
+| Earlier measurement | Reported result | Boundary |
+|---|---|---|
+| Warmed CPU/GPU overlap on a 48 GB Apple M4 Pro | Three sequential trials: 34.586, 38.988, and 37.997 seconds. Three parallel trials: 27.499, 28.233, and 38.167 seconds. | Median speedup was 1.346 times; one parallel trial was slower. This is overlap feasibility, not sustained utilization. |
+| Placement mismatch guard | An explicitly CPU-assigned Qwen runner still reported GPU residency. FreeLlama rejected feedback and a later request requiring observed placement. | Configured intent differs from physical evidence. |
+| Grounded delegation context | Six questions used 59,208 source tokens and returned 1,742 tokens: 97.1% context offload. | One workload; returned-token reduction is not runtime or billed-cost savings. |
+| Raw Ollama prefix reuse | A 2,462-token prefix took 18,631 ms cold, 281 ms warm, and 285 ms after another conversation interjected. | Engine cache reuse was observed on that path; scope IDs do not own or reserve it. |
+| Structured versus Bash research | The documented thirty-question Qwen suite had equal pass rates, with about 2.8 times more time and 4.7 times more input tokens for Octocode. | A structured tool did not improve efficiency on that measured suite. |
+| Code retrieval and judgment | Historical fixtures reported imperfect semantic-search recall and about 67% code-review accuracy. | Similarity and generated review findings require verification; they cannot authorize deletion or approve changes. |
+
+See [the measured CPU/GPU result](../CPU_GPU_ROUTING.md#interpret-the-measured-mac-result),
+[token economics](../ECONOMICS.md), [adapter caveats](../../benchmark/local/docs/07-adapter-contracts.md), and
+[historical delegation evidence](../../skills/freellama/assets/evidence/task-delegation.md).
+This review did not rerun those workloads or independently regrade their quality results.
+
+## Latest check on this Mac
+
+The quick check started isolated, owned Ollama and FreeLlama services and inspected the approved installed embedding model.
+Three route previews held under host memory pressure.
+The check submitted zero inference requests and downloaded zero models.
+Both owned services stopped after the examination.
+
+| Observation | Measured result | Interpretation |
+|---|---|---|
+| Host | Apple M2 Pro, 32 GiB RAM, twelve logical CPUs | One machine, not cross-platform qualification |
+| Available-memory estimate | About 5.050 GB during host sampling; 3.691 GB at the final preview | Admission remained held under the configured policy |
+| Recovery reserve | 6.872 GB | A recovery threshold, not this model's measured allocation |
+| Latest hold reason | `low_available_memory` | Normal OS memory-pressure status did not override FreeLlama's reserve |
+| Swap counters | Zero swap-outs and eight swap-in pages in the final sampled interval | This interval does not establish an absence of swapping overall |
+| GPU activity | Host samples of 28%, 33%, and 28% over about 2.4 seconds | Driver-defined activity across all processes; no FreeLlama attribution |
+| Thermal, power, and energy | Unknown; `powermetrics` required administrator access | No temperature or thermal-efficiency result |
+| Embedding model | `nomic-embed-text:v1.5`, installed size 274,302,450 bytes | Installation was verified; execution and physical placement were not qualified |
+| Inference trials | Zero | No throughput, quality, or CPU/GPU overlap result |
+
+Values marked GB use decimal gigabytes; the installed 32 GiB uses binary units.
+Earlier audit observations also held admission under low memory and active swapping.
+The latest check establishes resource refusal and cleanup, not an inference failure or a defective model.
+
+Local receipts remain under ignored `.octocode/tmp/` paths:
+`prepublish/final-audit.md`, `prepublish/final-test-counts.json`,
+`prepublish/independent-packed-readonly.json`, `quick-local/receipt.json`, and `quick-local-host.json`.
+They are local audit artifacts, not required installation files.
+Use [Local performance testing](LOCAL_PERFORMANCE_TESTING.md) for the repeatable measurement procedure.
+
+## Comparison with open-source projects
+
+Primary documentation was inspected on October 3, 2026.
+This table compares documented roles and mechanisms; none of these projects received a common-hardware benchmark in this review.
+Related projects can combine engine optimization, routing, and orchestration within their ecosystems.
+The categories describe emphasis, not exclusive feature ownership.
+
+| Project | Documented role and mechanisms | FreeLlama's position |
+|---|---|---|
+| [Ollama](https://docs.ollama.com/faq) | Local model runtime with residency controls, parallel requests, memory-dependent loading, and request queueing. | FreeLlama adds agent qualification, host admission, task scopes, and receipts around managed Ollama calls. |
+| [llama.cpp server](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) | Inference server with parallel slots, continuous batching, GPU offload, and slot prompt-cache save/restore. | FreeLlama scopes store application history with revisions and forks; engine prompt-cache snapshots store a different kind of state. Direct llama.cpp integration requires separate engineering and qualification. |
+| [vLLM](https://github.com/vllm-project/vllm) | Inference serving with PagedAttention, continuous batching, prefix caching, and distributed parallelism. | FreeLlama `0.2.1` targets Ollama; it does not implement these engine mechanisms or qualify a direct vLLM backend. |
+| [SGLang](https://github.com/sgl-project/sglang) | Inference serving for language and multimodal models. Its ecosystem includes hierarchical KV caching and deployment gateways. | FreeLlama focuses on bounded local delegation over Ollama. SGLang illustrates engine and deployment capabilities beyond that current integration. |
+| [LiteLLM Router](https://docs.litellm.ai/docs/routing) | Deployment routing supports weighted selection, least-busy, latency and cost strategies, session affinity, retries, fallbacks, and usage limits. | Both projects have routing and affinity concepts. FreeLlama focuses on installed Ollama eligibility, host admission, scoped history, and observed placement. |
+| [RouteLLM](https://github.com/lm-sys/RouteLLM) | Trained prompt routers select stronger or weaker models using calibrated quality-cost thresholds. | FreeLlama uses explicit policy and available evidence for local eligibility. Adding learned prompt routing requires a separate quality dataset and evaluation. |
+
+**Positioning judgment:** FreeLlama is a specialized control plane for developer agents operating local Ollama workloads.
+It is complementary to Ollama and narrower than general provider gateways or inference frameworks.
+Its most defensible contribution is an explicit delegation contract that combines admission, task history, and verifiable receipts.
+Competitive advantage in useful throughput remains a hypothesis requiring matched real workloads.
+
+Ollama already owns warming and runtime scheduling.
+FreeLlama must demonstrate additional workload value rather than claim these mechanisms as unique inventions.
+Engine-level batching, cache reuse, and device execution should remain engine responsibilities unless a measured integration justifies a change.
+See [Product positioning](../PRODUCT_POSITIONING.md) for the audience and product boundaries.
+
+## Feature and logic assessment
+
+This assessment answers the feature-only comparison requested after the broader audit.
+It uses reviewed contracts and documented mechanisms, excluding performance, hardware measurements, packaging, and publication readiness.
+Scores are subjective design judgments for developer agents using local Ollama workloads.
+They do not rank the other projects by a purpose they were not designed to serve.
+
+| Dimension | Score | Reason and limit |
+|---|---:|---|
+| Agent workflow and authority | 9/10 | Preview, qualification, explicit controls, bounded delegation, and error handling form a clear contract. The caller still owns task dependencies and judgment. |
+| Admission and execution logic | 8.5/10 | Weighted fairness, resource holds, deadlines, cancellation, and execution revalidation compose coherently. Operator policy and memory estimates remain part of admission. |
+| Task context and residency | 8.5/10 | Revisions, forks, finite history, affinity, and warming have separate ownership. Scopes and deferred jobs do not survive a service restart. |
+| Adaptive decisions | 7.5/10 | Comparable observed feedback can influence eligible backend choices without overriding explicit controls. It is not a learned prompt-quality router or general hardware scheduler. |
+| Explanation and evidence | 9/10 | Receipts distinguish eligibility, configured placement, observed placement, and feedback acceptance. Engine cache state remains outside scope ownership. |
+
+Each dimension carries equal weight: the average is **8.5/10**.
+A score of 10 requires complete, coherent behavior within this stated purpose, including lifecycle and recovery semantics.
+Additional features count only when they improve that purpose; engine responsibilities need not move into FreeLlama.
+
+### Decision logic compared directly
+
+| Decision | FreeLlama's mechanism | Related project mechanism |
+|---|---|---|
+| Which model or deployment? | Qualify installed candidates against task, capability, context, policy, and evidence requirements. | [RouteLLM](https://github.com/lm-sys/RouteLLM) uses trained prompt routers and calibrated strong/weak thresholds. [LiteLLM](https://docs.litellm.ai/docs/routing) offers multiple deployment-selection strategies. |
+| Can work start? | Check weighted capacity and host resources; queue, refuse, or execute within bounded deadlines. | [Ollama](https://docs.ollama.com/faq) manages runtime loading and queues requests when models cannot load. Its runtime checks complement FreeLlama's admission checks. |
+| What context survives a switch? | Replay bounded application history through scope IDs; fork histories and reject stale writes. | [llama.cpp](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) can save and restore slot prompt caches. Application history and engine cache snapshots are different contracts. |
+| How are active requests scheduled inside a model? | Submit admitted requests through Ollama and leave engine scheduling to the runner. | [vLLM](https://github.com/vllm-project/vllm) implements continuous batching, PagedAttention, and prefix caching. [SGLang's ecosystem](https://github.com/sgl-project/sglang) includes hierarchical KV caching and deployment gateways. |
+| When can adaptation override a preference? | Only comparable eligible feedback can influence automatic choices; explicit model, affinity, and policy retain authority. | [LiteLLM](https://docs.litellm.ai/docs/routing) supports latency, cost, load, and custom routing strategies. [RouteLLM](https://github.com/lm-sys/RouteLLM) calibrates query-level quality-cost tradeoffs. |
+
+**Judgment:** FreeLlama offers a strong combination of local agent controls, context ownership, and evidence semantics.
+Its current specialization is narrower than provider gateways and separate from inference-engine optimization.
+The comparison establishes feature differences; it does not establish exclusive capabilities or global superiority.
+
+Three feature gaps limit the score:
+
+1. **Restart recovery:** scopes and deferred jobs are process-local. Optional durable storage needs explicit privacy, expiry, and recovery contracts.
+2. **Task-quality adaptation:** backend timing feedback does not learn prompt-specific answer quality across models.
+3. **Backend breadth:** direct llama.cpp, vLLM, and SGLang integrations need adapters that preserve controls, cancellation, and evidence semantics.
+
+These are potential extensions, not confirmed correctness defects or authorization to implement them.
+The strongest priority depends on whether users need long-lived tasks, quality-based selection, or additional inference backends.
+
+## Historical assessment and remaining qualification work
+
+The independent SOL judge retained the overall **8/10** rating after inspecting the latest receipts.
+This is a judgment of the bounded control plane, not a weighted average or a hardware-performance score.
+
+| Dimension | Rating | Basis |
+|---|---:|---|
+| Architecture | 8/10 | Clear ownership and execution boundaries; optimization decisions still need workload evidence |
+| Agent contracts | 8.5/10 | Explicit shapes, authority, error handling, and evidence semantics; measured agent reliability remains incomplete |
+| Documentation | 8/10 | Operational guides and examples; this review corrects a stale MCP tool count |
+| Package engineering | 8.5/10 | Historical artifact, integrity, and consumer checks; current release and foreign execution checks remain pending |
+| Real throughput evidence completeness | 3/10 | Harnesses and controlled checks exist; no admissible matched real-inference trial in the latest qualification |
+| Hardware-utilization evidence completeness | 3/10 | Host telemetry exists; no attributed sustained inference, verified device overlap, or thermal qualification for this candidate on the latest Mac |
+
+Evidence-completeness scores describe measurement coverage, not slow execution or poor utilization.
+The earlier 7/10 and 6/10 evidence ratings overstated qualification relative to the retained physical receipts.
+For this rubric, 3 means instrumentation and contracts exist; 5 requires a real pilot; 7 requires repeated matched trials.
+A score of 9 requires sustained, attributable results across representative workloads and documented operating limits.
+A score of 10 also requires reproducibility beyond this single host.
+
+The following evidence is needed before increasing the ratings:
+
+1. Run matched direct-Ollama and managed trials under an admissible host with identical models, inputs, context, concurrency, and residency.
+2. Measure correct requests per minute, aggregate tokens per second, failures, queue waits, and p50/p95 latency against declared limits.
+3. Separate cold loading from warm execution, then increase concurrency to locate useful saturation rather than maximum busy time.
+4. Verify CPU/GPU overlap with observed placement and actual runner telemetry. Attribute activity wherever the platform permits.
+5. Capture sustained early and late throughput, clocks, thermal pressure, and available power telemetry. Record missing temperature readings explicitly.
+6. Repeat startup measurements and investigate the unclassified ten-second timeout.
+7. Rerun current release gates and fresh packed-consumer checks after all edits. Execute foreign native and installer checks on target runtimes.
+8. Evaluate research accuracy and context retention on held-out tasks. Use identical workloads before making competitor performance claims.
+
+Use [Local performance testing](LOCAL_PERFORMANCE_TESTING.md) and the
+[hardware benchmark contracts](../../benchmark/hardware/README.md) for measurement.
+Use the [local adapter benchmark](../../benchmark/local/README.md) for research quality.
+The [release procedure](../../RELEASE.md) owns publication readiness.

package/docs/dev/LOCAL_PERFORMANCE_TESTING.md

@@ -0,0 +1,298 @@
+# Test throughput, CPU/GPU activity, and thermal behavior on macOS
+
+Use this procedure to measure the resource-management cost and benefit of offloading tasks
+through FreeLlama on your Mac. It compares direct and managed Ollama execution for useful output,
+request latency, device activity, and thermal behavior under a declared workload.
+Run every command from the repository root.
+
+```mermaid
+flowchart TD
+    Prepare["Prepare services and fixed inputs"] --> Ready{"Resource checks clear?"}
+    Ready -->|"No"| Wait["Free resources, wait, and check again"]
+    Wait --> Ready
+    Ready -->|"Yes"| Warm["Warm exact model and verify placement"]
+    Warm --> Capture["Start host and thermal capture"]
+    Capture --> Trials["Run matched direct and managed trials"]
+    Trials --> Review["Check correctness, latency, activity, and pressure"]
+    Review --> Next{"Another concurrency setting?"}
+    Next -->|"Yes"| Prepare
+    Next -->|"No"| Cleanup["Retain receipts and stop owned services"]
+```
+
+Each result applies to the measured model, workload, configuration, and machine.
+
+## 1. Prepare the machine and inputs
+
+Connect mains power and keep the cooling environment consistent.
+Close unused applications and heavy builds.
+Record the macOS version, hardware, Ollama version, model digest, context, output limit, and concurrency.
+Use independently checked expected answers and representative input lengths.
+Choose latency and overhead limits before running the test.
+
+Do not disable resource admission to obtain a passing result.
+An active swapping or memory-pressure hold means the machine needs preparation before inference.
+Existing swap occupancy alone does not prove current swapping; inspect counter changes.
+
+Create a directory for this experiment:
+
+```bash
+mkdir -p .octocode/hardware/local-test
+```
+
+The earlier local audit prepared eight OCR cases under `.octocode/tmp/throughput-qualification/heldout-eight/`.
+If that directory exists, copy its inputs:
+
+```bash
+cp .octocode/tmp/throughput-qualification/heldout-eight/workload.json \
+   .octocode/tmp/throughput-qualification/heldout-eight/comparison.json \
+   .octocode/tmp/throughput-qualification/heldout-eight/*.png \
+   .octocode/hardware/local-test/
+```
+
+Those ignored fixtures are absent from a clean checkout and published packages.
+If they are absent, create a workload using the [hardware manifest example](../../benchmark/hardware/README.md#measure-a-useful-gpu-workload).
+Supply at least two distinct cases with independently checked expected answers.
+Image paths resolve relative to the workload file.
+For text generation, omit images and select a supported text task.
+Do not change expected answers to make a model pass.
+
+Inspect `workload.json` before using the example commands below.
+They assume the prepared OCR model `glm-ocr:latest`, concurrency `2`, and context `4096`.
+Confirm the exact tag is installed; its name does not prove installation or memory fit.
+This procedure never authorizes a model download.
+
+Create `comparison.json` beside the workload if it is missing:
+
+```json
+{
+  "schema_version": 1,
+  "workload": "workload.json",
+  "repeats_per_phase": 20,
+  "minimum_measured_seconds": 120,
+  "maximum_overhead_ratio": 1.15,
+  "latency_slo_ms": 500
+}
+```
+
+These are example limits, not universal performance targets.
+Select limits appropriate to your workload before inference.
+The runner performs at least three alternating direct/managed pairs.
+Its repeated cases increase timing samples, not the number of independent quality cases.
+
+## 2. Start isolated services
+
+Use unused loopback ports and dedicated terminals.
+Do not reuse a running production service for this experiment.
+If necessary, build the local CLI with `yarn build:rust`.
+
+In terminal A, start Ollama:
+
+```bash
+OLLAMA_HOST=127.0.0.1:11439 \
+OLLAMA_NO_CLOUD=1 \
+OLLAMA_NUM_PARALLEL=2 \
+OLLAMA_MAX_LOADED_MODELS=1 \
+OLLAMA_CONTEXT_LENGTH=4096 \
+ollama serve
+```
+
+Set `OLLAMA_NUM_PARALLEL` to the workload's concurrency before starting Ollama.
+The comparison requires an observed process setting, not an inferred default.
+
+Create `.octocode/hardware/local-test/runtime.toml`:
+
+```toml
+context_mode = "explicit"
+adaptive_concurrency = "off"
+```
+
+In terminal B, start the gateway:
+
+```bash
+./target/release/freellama serve \
+  --listen 127.0.0.1:11438 \
+  --upstream http://127.0.0.1:11439 \
+  --max-concurrent-tasks 8 \
+  --runtime-config .octocode/hardware/local-test/runtime.toml \
+  --ephemeral-feedback \
+  --ephemeral-usage
+```
+
+This sample freezes adaptive concurrency for a controlled comparison.
+The eight-unit budget permits two default four-unit vision tasks, subject to resource checks.
+It is an experiment setting, not a recommended production default.
+Record it, and test deployment defaults separately.
+For another task or concurrency, choose a matching weighted budget using the [routing guide](../CPU_GPU_ROUTING.md).
+
+If you enable gateway authentication, supply `--auth-token-file` to the gateway and each benchmark runner.
+Keep tokens out of manifests and telemetry logs.
+
+## 3. Check admission and warm the model
+
+Inspect current gateway health and status:
+
+```bash
+curl --fail --silent --show-error http://127.0.0.1:11438/_freellama/v1/health
+curl --fail --silent --show-error http://127.0.0.1:11438/_freellama/v1/status
+OLLAMA_HOST=127.0.0.1:11439 ollama list
+```
+
+Require an idle backend, fresh host observations, and normal memory pressure.
+Wait until resource holds clear before warming.
+The comparison runner independently validates its supported resource conditions.
+Treat a refusal as a diagnostic result, not throughput evidence.
+
+Warm the exact installed profile:
+
+```bash
+./target/release/freellama warm \
+  --endpoint http://127.0.0.1:11438 \
+  --model glm-ocr:latest \
+  --task vision \
+  --context-tokens 4096 \
+  --execution-preference prefer-gpu \
+  --min-placement-evidence configured \
+  --keep-alive 300s \
+  --timeout-seconds 60
+OLLAMA_HOST=127.0.0.1:11439 ollama ps
+```
+
+Review the warm receipt's placement observation.
+The initial configured requirement permits a cold load; the comparison requires verified GPU residency afterward.
+Require the installed/resident digest and context to match the workload.
+Start the benchmark before the warm keepalive expires.
+
+The `100% GPU` label in `ollama ps` describes model placement.
+It does not mean the GPU is busy for every instant. See the [Ollama FAQ](https://docs.ollama.com/faq).
+
+## 4. Capture telemetry while running inference
+
+In terminal C, start a ten-minute macOS capture:
+
+```bash
+sudo /usr/bin/powermetrics \
+  --samplers tasks,cpu_power,gpu_power,thermal \
+  --show-process-gpu \
+  -i 1000 -n 600 \
+  --output-file .octocode/hardware/local-test/power-thermal.txt
+```
+
+Run this command yourself with administrator access.
+On the inspected Mac, the unprivileged tool reports that it requires the superuser.
+It captures available process activity, CPU/GPU clocks, estimated subsystem power, and thermal pressure.
+The selected samplers may omit temperature readings in degrees Celsius.
+If a sampler is unavailable, retain the error and report that measurement as unknown.
+
+Run the [host sampler](../../benchmark/hardware/sample_host.py) in another terminal for memory and process observations.
+Find the exact PIDs and executable paths for the owned gateway, Ollama server, and inference runners.
+Repeat `--process PID=/absolute/executable` for those processes.
+For example, replace both placeholders before execution:
+
+```bash
+python3 benchmark/hardware/sample_host.py \
+  --process PID=/absolute/executable \
+  --duration-seconds 180 \
+  --interval-seconds 1 \
+  --output .octocode/hardware/local-test/host-observations.json
+```
+
+The host sampler supports at most 180 seconds per invocation.
+Use successive captures for a longer experiment, with separate output files.
+Process CPU and RSS exclude children; include the actual runner processes.
+In its interval CPU scale, 100% means one logical CPU.
+RSS does not equal physical memory footprint.
+GPU driver activity covers all processes; close unrelated GPU workloads and retain the idle baseline.
+Subsystem power estimates do not establish per-task energy.
+
+In terminal D, run the comparison:
+
+```bash
+python3 benchmark/hardware/run_comparison.py \
+  --endpoint http://127.0.0.1:11438 \
+  --ollama-endpoint http://127.0.0.1:11439 \
+  --comparison .octocode/hardware/local-test/comparison.json \
+  --output .octocode/hardware/local-test/comparison-receipt.json
+```
+
+Both paths use the same engine, logical payloads, context, and output controls.
+The runner warms each path and shares the resident runner and prefix cache.
+It does not establish independent cache states or measure time to first token.
+Keep model, runtime settings, and goldens unchanged within a run.
+
+## 5. Read the results and check sustained behavior
+
+Inspect `verdict`, `failures`, `completion`, and `performance` in the JSON receipt.
+Accept a run only when every planned request completes correctly and the declared gates pass.
+Keep rejected and incomplete receipts.
+
+| Measurement | What to examine |
+|---|---|
+| Useful throughput | Correctness-qualified requests/minute and qualified output tokens/second |
+| Responsiveness | Request wall latency p50/p95, sample counts, and latency-objective pass rate |
+| Gateway overhead | Median paired managed/direct wall ratio; above one means extra duration |
+| Waiting | Managed queue and total admission wait; direct queue breakdown remains unknown |
+| Model execution | Load, prefill, and decode durations, with reported token counts |
+| CPU/GPU activity | Process CPU, available GPU activity, and clocks during matched workload windows |
+| Memory | Pressure, compressor occupancy, reservation levels, and swap-counter changes |
+| Thermal behavior | Thermal pressure and clocks alongside early/late performance |
+
+Small-sample p95 values are descriptive; retain their sample counts.
+High CPU/GPU activity alone does not establish efficient execution.
+Prefer higher correct throughput at the required latency, with acceptable resource and thermal behavior.
+
+The copied example has a five-minute whole-run budget.
+A ten-minute telemetry capture does not extend that workload.
+For a sustained thermal experiment, freeze a separate plan before execution.
+Choose enough cases/repeats for continuous measured work and an adequate `run_budget_seconds` in `workload.json`.
+Set `minimum_measured_seconds` to the required measured duration, such as `600`.
+The comparison rejects an unmet duration target; it does not automatically loop until that target.
+Keep the plan within the runner's 10,000-request and 3,600-second whole-run limits.
+Ensure telemetry overlaps the entire measured interval.
+
+Compare early and late request windows at the same concurrency.
+Interpret clock changes alongside thermal pressure and workload phase; clock variation alone does not prove thermal throttling.
+If the host reports no thermal state, leave thermal status unknown.
+
+Measure cold loading in a separate experiment before warming.
+Retain load duration and total wall time without mixing them into the warm baseline.
+To test another concurrency, create a separate plan and restart the owned Ollama with the matching parallelism.
+Keep other settings comparable and record every change.
+Higher parallelism can require more context-cache memory. See [Ollama concurrency settings](https://docs.ollama.com/faq).
+
+## 6. Test simultaneous CPU and GPU work separately
+
+Configure a dedicated CPU upstream and exact CPU model assignment using the [CPU/GPU setup guide](../CPU_GPU_ROUTING.md).
+Ensure both installed models can coexist under the resource policy.
+Replace `EXACT_INSTALLED_GPU_TAG` with your qualified generation tag:
+
+```bash
+python3 benchmark/hardware/run_validation.py \
+  --endpoint http://127.0.0.1:11438 \
+  --gpu-model EXACT_INSTALLED_GPU_TAG \
+  --cpu-model nomic-embed-text:v1.5 \
+  --trials 3 \
+  --minimum-speedup 1.05 \
+  --output .octocode/hardware/local-test/cpu-gpu-receipt.json
+```
+
+Confirm `nomic-embed-text:v1.5` is installed and assigned to the CPU upstream first.
+This runner warms both tasks and compares matched sequential/parallel pairs.
+It requires observed CPU/GPU placement and correct responses.
+The default 1.05 speedup gate is configurable; declare the chosen target before execution.
+This short overlap check does not qualify sustained saturation or general model quality.
+
+Use [fixed-arrival load tests](../../benchmark/hardware/README.md#check-fixed-arrival-load-and-queue-ownership) for burst and queue behavior.
+Use real resident mode for physical measurements.
+Synthetic controlled mode supplies no inference throughput or physical placement proof.
+
+## 7. Retain evidence and clean up
+
+Keep input hashes, model/runtime identities, manifests, JSON receipts, telemetry, and failure logs together.
+Record the workload interval so reviewers can align it with telemetry timestamps.
+After pending jobs finish or cancel, stop only the services started for this experiment.
+Use Ctrl-C in their dedicated terminals.
+Verify no owned tasks or waiters remain before stopping the gateway.
+An inference keepalive expiry or service stop does not delete installed models.
+
+See the [hardware harness reference](../../benchmark/hardware/README.md) for schemas, bounds, and qualification details.
+See [production checks](../PRODUCTION.md) for promotion requirements beyond this local experiment.