@octocodeai/freellama-mcp-server 0.1.0

package/docs/PRODUCT_POSITIONING.md

@@ -0,0 +1,291 @@
+# Product positioning
+
+FreeLlama is a **governed local-model delegation layer for developers who run AI agents on
+Ollama**. It helps an agent decide whether local inference is appropriate and select an eligible
+installed model. It executes within machine and policy limits and returns evidence about what happened.
+
+The shortest accurate description is:
+
+> Ollama runs local models. FreeLlama helps agents use them deliberately.
+
+This positioning is intentionally narrower than “AI gateway” and more useful than “Ollama proxy.”
+FreeLlama preserves Ollama's API, but its distinctive work is the decision and evidence layer around
+inference: model qualification, fail-closed routing, admission, CPU/GPU placement, grounded
+research, and execution receipts.
+
+## Product category decision
+
+| Question | Decision |
+|---|---|
+| Product category | Local-model delegation control plane |
+| Primary audience | Developers operating coding or research agents with Ollama |
+| Core job | Move suitable work to local models without silently lowering quality or losing control of the machine |
+| Primary alternative | Call Ollama directly and implement routing, verification, and resource coordination in every agent |
+| Product promise | Make local delegation explicit, bounded, observable, and measurable |
+| Not the product | A chat UI, an Ollama fork, a hosted inference service, or a generic multi-provider gateway |
+
+“Trustworthy local-model delegation” is the clearest value phrase. “Localhost control plane” is an
+accurate architectural phrase, but it supports the explanation rather than leads it.
+
+## How the product fits
+
+```mermaid
+flowchart LR
+    U["Developer"] --> A["AI agent or application"]
+    A -->|"MCP, CLI, Rust, or control API"| F["FreeLlama"]
+    F --> D{"Delegate?\nWhich model?\nWhich backend?"}
+    D -->|"qualified GPU task"| G["Primary Ollama\nGPU-capable runner"]
+    D -->|"explicit CPU assignment"| C["Secondary Ollama\nCPU runner"]
+    D -->|"guard fails"| X["Refuse before inference"]
+    G --> E["Answer + execution receipt"]
+    C --> E
+    X --> E
+    E --> A
+```
+
+[Ollama](https://docs.ollama.com/api/introduction) owns model execution and its API. FreeLlama
+owns the policy around managed use of that runtime. The [Model Context Protocol
+(MCP)](https://modelcontextprotocol.io/specification/2025-06-18/architecture) makes those controls
+available to compatible agent hosts as seven bounded tools. Native Ollama requests can still pass
+through unchanged to the primary upstream.
+
+See [FreeLlama architecture](ARCHITECTURE.md) for the complete responsibility and request-flow
+contracts.
+
+## Audience hierarchy
+
+### Primary: developers operating AI agents
+
+The strongest user is a developer who already uses a capable cloud or local agent and wants to
+delegate bounded work to Ollama. Examples include grounded repository research, embeddings, OCR,
+bulk transforms, and helper-model tasks.
+
+Their job is not merely “run a model.” It is:
+
+1. Determine whether a local model is suitable for the task.
+2. Avoid sending work to an incapable, unmeasured, or oversized model.
+3. Keep large files, vectors, images, and intermediate tool output out of the orchestrator's
+   context.
+4. Coordinate concurrent work without uncontrolled model loads or memory pressure.
+5. Receive citations, verdicts, placement, queue, and timing signals that the agent can inspect.
+
+FreeLlama fits this user because its MCP server, CLI, agent skill, and control API expose exactly
+those decisions. The core job-to-be-done is:
+
+> When my agent encounters expensive but bounded work, decide whether a local model can handle it,
+> run it under explicit limits, and give the agent enough evidence to trust or reject the result.
+
+### Secondary: Ollama power users and local-AI operators
+
+An operator with several installed models can use FreeLlama to diagnose environment drift, inspect
+residency, preview routes, coordinate admission, assign selected helpers to a CPU backend, and turn
+benchmark results into routing policy.
+
+This audience can use the CLI without an MCP host. The relevant value is machine-aware operation,
+not agent integration alone. FreeLlama discovers the current host and observes Ollama state instead
+of embedding assumptions from the development Mac. See [model selection](MODEL_SELECTION.md) and
+[CPU/GPU routing](CPU_GPU_ROUTING.md).
+
+### Secondary: developers embedding local-model policy
+
+Library and tool authors can use the Rust core, Node native boundary, or HTTP control API instead
+of rebuilding discovery, routing, admission, and evaluation. This is an integration audience, not
+the clearest first-run story.
+
+### Adjacent: Ollama developers and contributors
+
+FreeLlama can help Ollama contributors as a downstream compatibility and workload testbed. Its
+diagnostics expose version drift, native passthrough tests protect API compatibility, and its
+dual-process experiments document observed runner placement.
+
+Ollama developers are not the primary audience. FreeLlama does not replace Ollama's scheduler,
+runner, model storage, accelerator backends, or token generation. It is a sidecar that exercises
+and governs Ollama from the client side. Describe the relationship as complementary:
+
+> Ollama makes local inference possible; FreeLlama makes agent delegation to it governable.
+
+### Not a target: mainstream users
+
+FreeLlama is not a mainstream local-AI product. It has no end-user chat interface. Meaningful use
+assumes familiarity with Ollama and its model tags. Users also need a terminal or MCP host and an
+understanding of policy or benchmark concepts.
+[Open WebUI](https://github.com/open-webui/open-webui) already addresses approachable self-hosted
+AI interfaces.
+
+A separate desktop or web interface can sit on top of FreeLlama as a new product layer. Keep that
+layer from blurring the current audience or forcing consumer concerns into the control plane.
+
+## The gap FreeLlama fills
+
+Calling Ollama directly is the correct path when the caller already knows the exact model
+and request. FreeLlama becomes useful when the caller has to make and defend a delegation decision.
+
+| Direct Ollama question | FreeLlama question |
+|---|---|
+| Which model tag receives this request? | Which installed model qualifies for this task and confidence threshold? |
+| Did the request return HTTP 200? | Was it admitted, where did it execute, and what did it cost? |
+| Can the model call tools? | Did bounded research read evidence and produce verifiable citations? |
+| Can Ollama load this model? | Does it fit the discovered machine and current resident workload? |
+| Can I set request options? | Can this managed model use the explicitly assigned CPU backend without changing raw traffic? |
+
+This distinction also separates FreeLlama from a broad gateway such as
+[LiteLLM](https://github.com/BerriAI/litellm). LiteLLM unifies many providers and adds centralized
+authentication, budgets, spend tracking, and load balancing. FreeLlama is Ollama-specific,
+machine-aware, and centered on agent delegation evidence. Its bearer boundary supports a trusted
+operator or team; it does not attempt LiteLLM's tenant, spend, or provider-management scope.
+
+[RouteLLM](https://github.com/lm-sys/RouteLLM) is the closer routing-specific comparison. It trains
+and evaluates routers that select between stronger and weaker model endpoints. FreeLlama does not
+implement a learned prompt router. It applies deterministic local eligibility and operator policy,
+then adds host fit, admission, physical CPU/GPU placement evidence, and grounded-research receipts.
+
+## The product loop
+
+The flexible part of FreeLlama is not an unconstrained agent choosing hardware. It is a guarded
+feedback loop with explicit operator and policy boundaries.
+
+```mermaid
+flowchart TD
+    T["Agent describes task, objective, and constraints"] --> P["Preview route"]
+    P --> Q{"Capability, policy, confidence,\nand capacity qualify?"}
+    Q -->|"no"| R["Refuse with reasons"]
+    Q -->|"yes"| A["Admit on selected backend"]
+    A --> O["Ollama executes"]
+    O --> V["Return result, placement, queue,\nand timing evidence"]
+    V --> F["Record normalized warm feedback"]
+    F -.->|"auto; fastest or balanced only\nafter enough samples"| P
+```
+
+An agent can request `auto`, `prefer_cpu`, or `prefer_gpu`, but the request remains a preference.
+Explicit model choice, session affinity, capability requirements, policy, confidence, and operator
+CPU assignments remain authoritative. Runtime feedback only influences eligible `auto` routes for
+speed-oriented objectives after comparable warm samples exist; it never overrides a quality
+objective. This design lets the system adapt to another machine without turning resource placement
+into an opaque autonomous decision.
+
+## Message hierarchy
+
+Use one message at each level:
+
+| Context | Message |
+|---|---|
+| Category | Local-model delegation control plane |
+| One line | FreeLlama helps AI agents use Ollama deliberately, with routing, resource limits, and evidence. |
+| Problem | Local inference is direct to call but hard to delegate to safely across different models and machines. |
+| Promise | Move suitable work local without silently lowering quality or destabilizing the runtime. |
+| Mechanism | Qualify, preview, admit, execute, and return a receipt. |
+| Boundary | Ollama runtime underneath; FreeLlama policy around it. |
+
+Good headline directions are outcome-first:
+
+- **Use local models without lowering your standards.**
+- **Give your AI agent a governed path to Ollama.**
+- **Delegate locally. Keep the evidence.**
+
+Avoid “AI orchestration platform.” It is broad enough to hide the product. Avoid leading with
+“proxy,” because raw passthrough is a compatibility feature rather than the principal value.
+
+## Build credible attention
+
+Create interest through short, reproducible contrasts rather than market-size or
+speed claims.
+
+### Demonstration 1: refuse before wasting work
+
+Show an agent previewing a task, rejecting an unqualified model, and returning the reason without
+an inference call. The story is not “the router is smart”; it is “a weak local model cannot quietly
+produce a confident answer.”
+
+### Demonstration 2: preserve frontier context
+
+Give `delegate_research` a narrow repository question. Show the local adapter reading allowlisted
+files and returning an answer, citations, and a verification verdict instead of returning all source
+material and intermediate tool output to the orchestrator.
+
+One measured six-question run reduced 59,208 source tokens to 1,742 returned tokens, or 97.1%
+context offload. That is a single measured workload, not a universal savings rate. The durable
+claim is context isolation; see [token economics](ECONOMICS.md).
+
+### Demonstration 3: use the machine as a system
+
+Run a large GPU completion and a small CPU helper concurrently, then show both placement receipts.
+On the measured 48 GB Apple M4 Pro, three warmed trials reduced median wall time from 37.997 to
+28.233 seconds, a 1.346-times speedup. One of the three parallel trials was slower than sequential,
+so the claim is “these workloads can overlap,” not “CPU/GPU mode is always faster.” See the full
+[CPU/GPU measurement](CPU_GPU_ROUTING.md#interpret-the-measured-mac-result).
+
+These demonstrations map directly to three memorable benefits: **trust, context, and resources**.
+Turn them into a short terminal recording, copyable README recipes, and versioned benchmark
+receipts. Each demo needs the command, machine profile, model tags, policy, result, and
+failure conditions.
+
+## Adoption path
+
+The first-use sequence needs to reveal value before asking for sophisticated configuration:
+
+```mermaid
+flowchart LR
+    I["Install beside Ollama"] --> D["doctor\nsee machine and drift"]
+    D --> P["preview\nsee a route and reasons"]
+    P --> T["run one managed task\nsee the receipt"]
+    T --> G["delegate research\nsee citations and verdict"]
+    G --> B["benchmark\nmeasure local models"]
+    B --> Y["policy\nallow proven routes"]
+    Y --> R["resource tuning\nCPU/GPU and admission"]
+```
+
+The product must not require a new user to understand every tool, KV-cache setting, or routing
+objective before the first managed task. Advanced controls become relevant after the first receipt
+makes the control loop visible.
+
+## Claim and naming guardrails
+
+| Do not say | Say instead |
+|---|---|
+| Free local inference | Local inference uses hardware, power, memory, and time; FreeLlama can preserve paid frontier context. |
+| Automatically picks the optimal model | Routes among eligible installed models using explicit policy and available evidence. |
+| Makes Ollama faster | Can improve workload efficiency; it does not increase one model's decode speed. |
+| Runs CPU and GPU in parallel on every machine | Can overlap explicitly assigned workloads when the host and Ollama backends support them; measure the result. |
+| Works the same on any hardware | Uses platform-specific host discovery on macOS, Linux, and Windows. Portable defaults do not remove the need to observe accelerator behavior. |
+| Replaces Ollama | Runs beside Ollama and preserves its native API. |
+| For everyone | Built first for developers and local-AI operators. |
+| Llama models only | Works with qualifying models exposed by Ollama; “Llama” is the product name, not a model-family restriction. |
+
+Use the word “Free” to refer to reducing reliance on metered frontier inference, not zero-cost
+computing. Readers can misread “Llama” as Meta Llama-only, so put the Ollama relationship and model
+agnosticism within Ollama near the first description.
+
+## Product decision
+
+Narrow the story before broadening the product:
+
+1. Lead with developers operating AI agents, not all developers.
+2. Make trustworthy delegation the category-defining job.
+3. Use MCP as the primary agent integration and the CLI as the operator surface.
+4. Treat Ollama power users and embedders as secondary audiences.
+5. Treat Ollama contributors as an adjacent ecosystem audience, not the main customer.
+6. Do not pursue a mainstream chat interface until the developer onboarding loop is clear and
+   repeatable.
+
+This decision matches the code that exists today. The Rust core, CLI, MCP server, proxy, agent
+skill, benchmark harnesses, and policy system all support a developer control plane. Positioning it
+as a consumer app promises a product surface the repository does not provide.
+
+## Related documentation
+
+- [Architecture](ARCHITECTURE.md): ownership boundaries and request flows
+- [MCP server](../packages/mcp/README.md): six agent-facing tools and their contracts
+- [CLI](CLI.md): operator commands and control-plane workflows
+- [Model selection](MODEL_SELECTION.md): evidence, qualification, and recommendation
+- [CPU/GPU routing](CPU_GPU_ROUTING.md): portable configuration, feedback, and measured limits
+- [Testing](TESTING.md): verification tiers and live-runtime boundaries
+- [Token economics](ECONOMICS.md): measured context isolation and its limits
+
+## Sources
+
+- [Ollama API introduction](https://docs.ollama.com/api/introduction)
+- [Ollama tool calling](https://docs.ollama.com/capabilities/tool-calling)
+- [Model Context Protocol architecture](https://modelcontextprotocol.io/specification/2025-06-18/architecture)
+- [Open WebUI](https://github.com/open-webui/open-webui)
+- [LiteLLM](https://github.com/BerriAI/litellm)
+- [RouteLLM](https://github.com/lm-sys/RouteLLM)

package/docs/TESTING.md

@@ -0,0 +1,125 @@
+# Test FreeLlama
+
+FreeLlama separates fast deterministic checks from live-system tests. Run the narrowest relevant
+tier during development, then run the full matrix before release.
+
+```mermaid
+flowchart LR
+    U["Unit and Rust contracts"] --> A["Local-agent context<br/>and action contracts"]
+    A --> T["Type check, format, and Clippy"]
+    T --> I["MCP integration against live Ollama"]
+    I --> E["End-to-end tools against serve and Ollama"]
+    E --> L["Optional live CPU/GPU workload evaluation"]
+```
+
+## Install and build
+
+```bash
+yarn install
+yarn build
+```
+
+The root build compiles the host Rust release binary, NAPI addon, and single-file MCP JavaScript
+bundle. The native `.node` addon remains external to the JavaScript bundle because it is selected
+by platform triple at runtime. Published packages resolve it from an OS/CPU/libc-specific optional
+dependency rather than compiling Rust during installation.
+
+## Run the test tiers
+
+| Tier | Command | External requirements |
+|---|---|---|
+| JavaScript unit | `yarn test` | None |
+| JavaScript watch | `yarn test:watch` | None |
+| Rust contracts | `yarn test:rust` | Rust toolchain |
+| Local-agent contracts | `yarn test:agents` | Python 3 standard library |
+| TypeScript | `yarn typecheck` | None after install |
+| MCP integration | `yarn test:integration` | Ollama on `127.0.0.1:11434` |
+| MCP end to end | `yarn test:e2e` | Ollama, `freellama serve`, and required models |
+| All configured tiers | `yarn test:all` | Requirements of every included tier |
+| Production verification | `yarn verify:production` | Build, formatting, strict Clippy, all test tiers, and package verification |
+
+The end-to-end suite checks behavior rather than schema shape: confidence refusal happens before
+generation, embedding vectors are withheld by default, impossible models are excluded by memory,
+and models with unusable research evidence are refused without tool calls. Tests that require a
+specific unavailable model report a skip reason.
+
+## Run static Rust checks
+
+```bash
+cargo fmt --all --check
+cargo clippy --all-targets -- -D warnings
+```
+
+Routing regressions run in the default Rust suite; none are hidden behind `#[ignore]`. Add a
+focused contract test for each repaired behavior and keep it enabled in ordinary release checks.
+The local-agent tier runs all context fitting, compaction, pagination, repeat-suppression, and
+strict action-shape contracts. It is included in `yarn test:all`, so adapter regressions cannot be
+missed by the root release matrix.
+
+`yarn test:all` does not run formatting, Clippy, a release build, or package inspection. Use
+`yarn verify:production` for the complete local promotion gate.
+
+Before claiming platform support, run the deterministic Rust and JavaScript/native lanes from a
+clean checkout on macOS, Linux, and Windows. The machine profile contract test requires a positive
+CPU count, host-memory value, and free-disk value on each machine; it reports unified memory only
+on Apple-silicon macOS. This catches an OS branch that compiles but returns no usable capacity to
+recommendations. Record the commands, toolchain versions, commit, and results with the release.
+
+`platform_contract` also pins the resource-control loop: a CPU preference is honored only for an
+operator-assigned eligible model, an explicit model overrides that hint and reports the fallback,
+two warm samples do not steer `auto`, the third sample on both backends enables a token-normalized
+comparison, differences of 10% or less remain noise, and one GPU plus one CPU request overlap even
+when both admission pools contain one unit.
+
+## Validate CPU and GPU concurrency
+
+Start the two Ollama processes and FreeLlama as described in
+[CPU and GPU model routing](CPU_GPU_ROUTING.md). Verify placement with both backend `/api/ps`
+responses, then compare matched sequential and concurrent requests. Record at least three warmed
+trials and use the median so one cold load does not decide the result.
+
+The local validated workload used a CPU-assigned `nomic-embed-text:latest` request and a resident
+GPU `qwen3.8:27b-mlx` completion. It measured a 1.346-times median speedup with successful responses,
+zero FreeLlama queue wait, `size_vram: 0` for the CPU runner, and positive `size_vram` for the GPU
+runner.
+
+The placement guard recorded 19,175,677,668 GPU-resident bytes for Qwen, identical GPU output
+length across all trials, correct upstream receipts, and primary Ollama 0.33.2 through raw
+passthrough. A separate small-helper trial returned the CPU embedding in 60 ms while the cold GPU
+completion continued to 7.391 seconds.
+
+After rebuilding the release, a fresh concurrent smoke check completed both managed requests in
+8.303 seconds. It returned HTTP 200, zero queue wait, the expected upstream receipts, `OK` from
+Qwen, one embedding from Nomic, positive Qwen VRAM, zero Nomic VRAM, and primary Ollama 0.33.2
+through raw passthrough.
+
+For promotion, use the portable [hardware acceptance runner](../benchmark/hardware/README.md) and
+archive its JSON receipt with the release evidence. Compilation alone does not validate a driver,
+physical placement, shared-memory contention, or OCR quality.
+
+## Verify release packages
+
+```bash
+# Collect each target's `freellama` executable and `freellama.<target>.node`
+# under release-artifacts/<target>/, then stage all eight platform packages.
+yarn release:assemble release-artifacts release
+yarn release:verify:publish
+```
+
+Build every claimed CLI/native target in a clean environment, assemble `SHA256SUMS`, run the full
+deterministic suite, and inspect every platform package dry run before publishing explicitly.
+Publish the eight `@octocodeai/freellama-native-*` packages before `@octocodeai/freellama` and `@octocodeai/freellama-mcp-server`, all
+at the exact same version. A successful local pack dry run proves package contents; it does not
+prove registry publication or another hardware class.
+
+## Test one layer while iterating
+
+```bash
+yarn build:native
+yarn workspace @octocodeai/freellama-mcp-server build
+yarn workspace @octocodeai/freellama-mcp-server test
+yarn workspace @octocodeai/freellama test
+```
+
+The MCP integration and end-to-end setup rebuilds its bundle. The CLI package tests its launcher and
+published-file contract; Rust tests own the executable's command and routing behavior.

package/native/index.d.ts

@@ -0,0 +1,158 @@
+// Hand-written type declarations for the native module (see index.js for why these are
+// hand-written rather than tool-generated). Mirrors packages/rust-core/src/napi.rs's public function signatures
+// exactly — keep the two in sync if you change one. `npm run build` no longer overwrites this
+// file (it previously did, silently emptying it — see package.json's build script comment).
+
+/**
+ * Runs `freellama doctor` against Ollama directly — no running `freellama serve` required.
+ * Cross-checks the Ollama CLI and server versions and confirms the endpoint is reachable.
+ * @param endpoint Ollama endpoint, defaults to http://127.0.0.1:11434
+ * @returns Pretty-printed JSON string
+ */
+export function doctor(endpoint?: string | undefined | null): Promise<string>;
+
+/**
+ * Machine profile (chip, host memory, memory kind, CPU count, and disk) from `freellama serve`.
+ * @param endpoint FreeLlama serve endpoint, defaults to http://127.0.0.1:11435
+ */
+export function machine(endpoint?: string | undefined | null): Promise<string>;
+
+/** Current managed-platform health, including admission and session bounds. */
+export function health(endpoint?: string | undefined | null): Promise<string>;
+
+/** Create a bounded, idle-expiring model-affinity handle; it stores no prompt history or KV. */
+export function createSession(endpoint?: string | undefined | null): Promise<string>;
+
+/** Release a model-affinity handle after related work has completed. */
+export function deleteSession(
+  endpoint: string | undefined | null,
+  sessionId: string,
+): Promise<void>;
+
+/**
+ * Installed-model inventory with capabilities, residency, and advertised context.
+ * @param endpoint FreeLlama serve endpoint, defaults to http://127.0.0.1:11435
+ */
+export function listModels(endpoint?: string | undefined | null): Promise<string>;
+
+/**
+ * Deterministic model selection for a task.
+ * @param endpoint FreeLlama serve endpoint, defaults to http://127.0.0.1:11435
+ * @param task e.g. "completion" | "code_repair" | "vision" | "embedding"
+ * @param objective "fastest" | "balanced" | "quality", defaults to "balanced"
+ * @param model Optional explicit model name to force
+ * @param sessionId Optional session id for model affinity
+ * @param contextTokens Optional minimum context window required
+ * @param requiredCapabilities Optional list of capabilities the model must advertise (e.g. ["vision"])
+ * @param minConfidence Optional floor ("low" | "medium"); the core gate refuses below it
+ * @param executionPreference Guarded backend hint: "auto" | "prefer_cpu" | "prefer_gpu"
+ * @param minPlacementEvidence "configured" (default) or fail-closed physical "observed"
+ * @param minPlacementEvidence "configured" (default) or fail-closed physical "observed"
+ */
+export function route(
+  endpoint: string | undefined | null,
+  task: string,
+  objective?: string | undefined | null,
+  model?: string | undefined | null,
+  sessionId?: string | undefined | null,
+  contextTokens?: number | undefined | null,
+  requiredCapabilities?: Array<string> | undefined | null,
+  minConfidence?: string | undefined | null,
+  executionPreference?: string | undefined | null,
+  minPlacementEvidence?: string | undefined | null,
+): Promise<string>;
+
+/**
+ * Side-effect-free install recommendation for a task. Never runs `ollama pull` itself.
+ * @param endpoint FreeLlama serve endpoint, defaults to http://127.0.0.1:11435
+ * @param task e.g. "completion" | "code_repair" | "vision" | "embedding"
+ * @param objective "fastest" | "balanced" | "quality", defaults to "balanced"
+ * @param model Optional explicit model name to filter to
+ * @param contextTokens Optional minimum context window required
+ * @param requiredCapabilities Optional list of capabilities the model must advertise (e.g. ["vision"])
+ * @param executionPreference Guarded backend hint: "auto" | "prefer_cpu" | "prefer_gpu"
+ * @param minPlacementEvidence "configured" (default) or fail-closed physical "observed"
+ */
+export function recommend(
+  endpoint: string | undefined | null,
+  task: string,
+  objective?: string | undefined | null,
+  model?: string | undefined | null,
+  contextTokens?: number | undefined | null,
+  requiredCapabilities?: Array<string> | undefined | null,
+  executionPreference?: string | undefined | null,
+  minPlacementEvidence?: string | undefined | null,
+): Promise<string>;
+
+/**
+ * Routes AND executes a chat/generate/embed call in one shot — unlike `route`/`recommend`, this
+ * actually does work. Provide `prompt` for a single message, or `messages` (an array of
+ * `{role, content}` objects) for multi-turn history. For task "embedding", set `input` instead.
+ * @param endpoint FreeLlama serve endpoint, defaults to http://127.0.0.1:11435
+ * @param task e.g. "completion" | "code_repair" | "vision" | "embedding"
+ * @param objective "fastest" | "balanced" | "quality", defaults to "balanced"
+ * @param model Optional explicit model name to force
+ * @param sessionId Optional session id for model affinity
+ * @param contextTokens Optional minimum context window required
+ * @param requiredCapabilities Optional list of capabilities the model must advertise
+ * @param prompt Single-turn user message
+ * @param images Base64-encoded images (no data-URI prefix) attached to the `prompt` message — pair with requiredCapabilities: ["vision"]
+ * @param messages Multi-turn message history, wins over `prompt` if both are set
+ * @param input Embedding input (string or array of strings), for task "embedding"
+ * @param tools Optional tool/function definitions for function-calling tasks
+ * @param keepAlive Overrides Ollama's model residency window, e.g. "0" or "-1"
+ * @param minConfidence Optional floor ("low" | "medium"); the core gate refuses before generation
+ * @param executionPreference Guarded backend hint: "auto" | "prefer_cpu" | "prefer_gpu"
+ */
+export function runTask(
+  endpoint: string | undefined | null,
+  task: string,
+  objective?: string | undefined | null,
+  model?: string | undefined | null,
+  sessionId?: string | undefined | null,
+  contextTokens?: number | undefined | null,
+  requiredCapabilities?: Array<string> | undefined | null,
+  prompt?: string | undefined | null,
+  images?: Array<string> | undefined | null,
+  messages?: unknown | undefined | null,
+  input?: unknown | undefined | null,
+  tools?: unknown | undefined | null,
+  keepAlive?: string | undefined | null,
+  minConfidence?: string | undefined | null,
+  executionPreference?: string | undefined | null,
+  minPlacementEvidence?: string | undefined | null,
+): Promise<string>;
+
+/**
+ * Object-based managed task API. Supports the full typed task request including lossless message
+ * history and request_options (format, think, runtime options, logprobs).
+ * @param endpoint FreeLlama serve endpoint, defaults to http://127.0.0.1:11435
+ * @param request Complete `/_freellama/v1/tasks` request object
+ */
+export function runTaskRequest(
+  endpoint: string | undefined | null,
+  request: unknown,
+): Promise<string>;
+
+/**
+ * Executes explicitly independent tasks with a bounded, priority-fair dispatcher. Individual
+ * task failures are returned as result entries so unaffected sibling work is preserved.
+ * @param endpoint FreeLlama serve endpoint, defaults to http://127.0.0.1:11435
+ * @param request Complete `/_freellama/v1/task-batches` request object
+ */
+export function runTaskBatchRequest(
+  endpoint: string | undefined | null,
+  request: unknown,
+): Promise<string>;
+
+/**
+ * Converts a free-text natural-language intent into a route.
+ * @param endpoint FreeLlama serve endpoint, defaults to http://127.0.0.1:11435
+ * @param text Free-text description of what's needed
+ * @param sessionId Optional session id for model affinity
+ */
+export function naturalRoute(
+  endpoint: string | undefined | null,
+  text: string,
+  sessionId?: string | undefined | null,
+): Promise<string>;

package/native/index.js

@@ -0,0 +1,45 @@
+// Hand-written glue: napi-rs CLI did not emit JS bindings for this crate/cli pair, so this
+// file re-exports the compiled `.node`. Rebuild with `npm run build` from the repo root.
+// See packages/mcp/README.md.
+const { existsSync } = require("node:fs");
+const path = require("node:path");
+const { createRequire } = require("node:module");
+const requireFromHere = createRequire(__filename);
+
+// napi-rs filenames: `freellama.<platform>-<arch>[-<abi>].node`. ABI suffix is present on
+// Linux/Windows, absent on macOS. On Linux try both gnu and musl — Node's glibc probe does not
+// have to match how the addon was compiled.
+const ABI_SUFFIXES = {
+  linux: process.report?.getReport()?.header?.glibcVersionRuntime ? ["-gnu", "-musl"] : ["-musl", "-gnu"],
+  win32: ["-msvc"],
+};
+
+function candidateIds() {
+  const { platform, arch } = process;
+  const suffixes = ABI_SUFFIXES[platform] ?? [];
+  return [...new Set(suffixes.concat("").map((suffix) => `${platform}-${arch}${suffix}`))];
+}
+
+const candidates = candidateIds();
+const found = candidates.find((id) => existsSync(path.join(__dirname, `freellama.${id}.node`)));
+
+if (!found) {
+  const failures = [];
+  for (const id of candidates) {
+    const packageName = `@octocodeai/freellama-native-${id}`;
+    try {
+      module.exports = requireFromHere(packageName);
+      return;
+    } catch (error) {
+      failures.push(`${packageName}: ${error?.message ?? error}`);
+    }
+  }
+  throw new Error(
+    `freellama: no usable native addon for ${process.platform}-${process.arch}.\n` +
+      `Tried local development artifacts and optional packages: ${candidates.map((id) => `@octocodeai/freellama-native-${id}`).join(", ")}.\n` +
+      "Reinstall without --omit=optional. From a source checkout, run `yarn build:native`.\n" +
+      `Load failures:\n  ${failures.join("\n  ")}`,
+  );
+}
+
+module.exports = require(`./freellama.${found}.node`);

package/native/package.json

@@ -0,0 +1,7 @@
+{
+  "name": "freellama-native",
+  "private": true,
+  "type": "commonjs",
+  "main": "index.js",
+  "types": "index.d.ts"
+}