virtuai-cli 1.0.1__tar.gz → 1.0.3__tar.gz

{virtuai_cli-1.0.1 → virtuai_cli-1.0.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: virtuai-cli
-Version: 1.0.1
+Version: 1.0.3
 Summary: Run VirtuAI deep agents on your local machine
 Author-email: uCloudStore <lmoreno@ucloudstore.com>
 License: Proprietary
@@ -21,7 +21,9 @@ Requires-Dist: typer>=0.12
 Requires-Dist: rich>=13.0
 Requires-Dist: textual>=0.86
 Provides-Extra: local
-Requires-Dist: langchain-ollama>=0.2; extra == "local"
+Requires-Dist: langchain-ollama>=0.3.4; extra == "local"
+Requires-Dist: llmfit>=1.1.16; extra == "local"
+Requires-Dist: langchain-openai>=0.3; extra == "local"
 Provides-Extra: browser
 Requires-Dist: browser-use>=0.1.40; extra == "browser"
 
@@ -41,7 +43,7 @@ Or with [pipx](https://pipx.pypa.io/) (recommended for CLI tools — handles PAT
 pipx install virtuai-cli
 ```
 
-For **local model inference** (running agents against [Ollama](https://ollama.com) on your machine — see below), install the `local` extra:
+For **local model inference** (running agents against [Ollama](https://ollama.com) or [oMLX](https://github.com/jundot/omlx) on your machine — see below), install the `local` extra:
 
 ```bash
 pip install 'virtuai-cli[local]'      # or: pipx install 'virtuai-cli[local]'
@@ -89,27 +91,58 @@ virtuai ask --json "find any bugs" > events.jsonl
 
 `ask` is non-interactive: it prints the response to stdout, then exits.
 
-## Local model inference (Ollama)
+## Local model inference
 
-Agents can run inference on **your own machine** via [Ollama](https://ollama.com) instead of a cloud provider — useful for cost, data locality, or running open models. Only the model turns are proxied to your machine over the CLI's WebSocket; the agent loop, tools, knowledge bases, and chat history stay on the server.
+Agents can run inference on **your own machine** — with [Ollama](https://ollama.com), or [oMLX](https://github.com/jundot/omlx) on Apple Silicon — instead of a cloud provider: for cost, data locality, or open models. Only the model calls go to the machine; the agent loop, tools, knowledge bases and chat history stay on the server. It works from **every channel** — web chat, Slack, Telegram, WhatsApp, Google Chat, OpenCode — not only the TUI, as long as a runner that can serve the model is connected.
 
-To use it:
+### Set up
 
-1. Install the CLI with the local extra (pulls `langchain-ollama`):
+1. Install the CLI with the local extra (pulls `langchain-ollama`, `langchain-openai` for oMLX, and `llmfit`):
    ```bash
    pip install 'virtuai-cli[local]'
    ```
-2. Install [Ollama](https://ollama.com), start it, and pull a **tool-capable** model:
+2. Install Ollama **0.40 or newer** and pull a validated model:
    ```bash
-   ollama serve
-   ollama pull qwen3:8b          # instruct/agentic models work best for tool calling
+   ollama pull qwen3:8b
    ```
-3. In the VirtuAI portal, set the agent's **Provider** to **Local (Ollama)**, enter the model tag you pulled, and pick a **cloud fallback** model.
-4. Connect the CLI (`virtuai chat`, `virtuai ask`, or `virtuai run`) — model turns are now answered by your local Ollama.
+   Ollama before 0.40 held the same models in two to three times the memory; `virtuai run` warns when it finds one.
+3. In the VirtuAI portal, set the agent's **Provider** to **Local (Ollama)** (or **Local (oMLX)**), enter the model, and pick a **cloud fallback** model.
+4. Keep a runner connected: `virtuai run`, or `virtuai chat` / `virtuai ask`, which start their own.
 
-The CLI talks to Ollama at `http://localhost:11434` by default; override with the `OLLAMA_BASE_URL` environment variable.
+### Which models
 
-If the CLI isn't connected, Ollama isn't running, or the model isn't pulled, the agent **falls back to its configured cloud model** and shows a notice. For reliable tool use, prefer instruct/agentic models (e.g. `qwen3:*`, `qwen2.5:*`); some code-tuned or non-instruct models emit tool calls as plain text and won't drive the agent's tools.
+Validated — every one of 10 tool-calling tasks right, 8k context, 16 GB Apple M3 with Ollama 0.40:
+
+| Model | Memory | Median answer |
+|---|---|---|
+| `qwen3:8b` | 6.6 GB | 2.2 s |
+| `qwen2.5:7b` | 5.0 GB | 3.5 s |
+| `qwen2.5-coder:7b` | ~5 GB | 4.4 s |
+| `qwen2.5:14b` | 10.3 GB | 6.0 s |
+
+Other models run, but are not vouched for. `qwen3:4b` is accurate but slow (it reasons before every call). Gemma 3 is never used: Ollama cannot call tools with it, so the cloud fallback answers.
+
+### Whose turns this machine answers
+
+```bash
+virtuai run                              # your own turns only (the default)
+virtuai run --serve-models workspace     # anyone's in the workspace: a shared Mac mini or GPU box
+virtuai run --serve-models none          # tools only, no model turns
+```
+
+Each model call goes to the runner of the person the turn is for; if they have none, to a runner started with `--serve-models workspace`; if there is none either, the agent's **cloud fallback** answers that call and every channel shows "Modelo local no disponible: responde …". A runner that fails before it answers (Ollama stopped, the model unloaded) costs the call a cloud answer, not an error.
+
+> **Upgrading from 1.0.2:** with a current server, an older runner serves only the person who paired it. A machine that serves a whole team needs 1.0.3 and `--serve-models workspace`.
+
+### What the runner tells the server
+
+At connect and then every minute: the models it has, the Ollama version, what is loaded and how much memory it really took. With `llmfit` installed it also estimates which pulled models fit this machine, so the server does not send a model to a runner that cannot hold it until it has actually run there. A model that spills to the CPU gets half the context on its next call on that machine (down to 4k).
+
+The context sent to Ollama is 8k by default (set on the server); `OLLAMA_NUM_CTX` overrides it on a machine with less memory. For qwen3 the runner turns Ollama's thinking mode off: left on, it reasoned for minutes before every tool call.
+
+### oMLX
+
+The runner finds oMLX through its own settings (`~/.omlx/settings.json`: port and API key) or `VIRTUAI_OMLX_URL` / `OMLX_API_KEY`; the key never leaves the machine. Measured with `Qwen3-8B-4bit`: 10/10, 2.5 s per answer — on par with Ollama for one person. Its case is a shared host serving several people at once; raise `VIRTUAI_MAX_CONCURRENT_MODEL` there, since the runner otherwise sends one generation at a time.
 
 ## Environment variables
 
@@ -124,11 +157,15 @@ mean "allow more commands at once" silently launched more browsers.
 | `VIRTUAI_ISOLATE_SESSIONS` | off | Give each conversation its own directory under the workdir, and each person their own browser. Turn it on when one runner serves several people. Also set by `--isolate-sessions`. |
 | `VIRTUAI_MAX_NAMESPACES` | `5` | Browsers held at once — one per person. Each is a real headful Chrome (roughly 0.4–1 GB resident), so size this to the machine's memory: an OOM-killed Chrome is a worse failure than a clean refusal. Only applies with isolation on. Also set by `--max-namespaces`. |
 | `VIRTUAI_MAX_CONCURRENT_WORK` | `4` | Commands running at once. Each conversation is already capped at one by its own queue, so in practice this is how many conversations may be executing at the same instant. |
-| `VIRTUAI_MAX_CONCURRENT_MODEL` | `1` | Local model generations at once. Ollama does one well; its own parallelism lives in `OLLAMA_NUM_PARALLEL` on your machine, which the server cannot see. |
+| `VIRTUAI_MAX_CONCURRENT_MODEL` | `1` | Local model generations at once. Ollama does one well; its own parallelism lives in `OLLAMA_NUM_PARALLEL` on your machine, which the server cannot see. Raise it for oMLX on a shared host, which batches several at once. |
 | `VIRTUAI_MAX_QUEUED_FRAMES` | `64` | Requests waiting for a slot. Past this the runner answers "busy" rather than queueing without bound. |
 | `VIRTUAI_SESSION_IDLE_TIMEOUT` | off (`0`) | Seconds before an idle browser stops holding its slot. Off by default: a slot normally frees only when its browser closes or the runner restarts. |
 | `VIRTUAI_SANDBOX_OWNER` | unset | The one person this runner serves, reported to the server so a browser action reaches *their* browser. Leave it unset on a laptop or a container shared by a team: the server then routes to it for anyone, which is the behaviour that existed before per-person sandboxes. Set it when a workspace has a sandbox per person, or actions land on whichever runner connected last. |
+| `VIRTUAI_SERVE_MODELS` | `own` | Whose local-model turns this runner answers: `own`, `workspace` or `none`. Also set by `--serve-models`. |
 | `OLLAMA_BASE_URL` | `http://localhost:11434` | Where to reach the local Ollama. |
+| `OLLAMA_NUM_CTX` | unset | The context to ask Ollama for, instead of the server's (8k). Lower it on a machine with little memory. |
+| `VIRTUAI_OMLX_URL` | from `~/.omlx/settings.json`, else `http://127.0.0.1:8000` | Where to reach oMLX. |
+| `OMLX_API_KEY` | from `~/.omlx/settings.json` | oMLX's API key. Stays on this machine. |
 
 Server-side, `VIRTUAI_ENV_WAIT_SECONDS` (default `45`) is how long the server
 holds a message waiting for a runner to reconnect before telling the user the
@@ -143,7 +180,7 @@ environment is not responding.
 | `virtuai opencode`      | Merge this workspace's agents into the local OpenCode TUI config |
 | `virtuai pair <code>`    | Pair this machine with a workspace |
 | `virtuai unpair`         | Revoke the current pairing |
-| `virtuai run`            | Headless WebSocket runner — let the web app/channels invoke local tools without opening the TUI |
+| `virtuai run`            | Headless runner — let the web app and channels use this machine's tools and local models without opening the TUI (`--serve-models own\|workspace\|none`) |
 | `virtuai status`         | Show pairing and connection status |
 | `virtuai logs`           | Show recent commands the agent executed locally |
 | `virtuai login`          | Authenticate with your VirtuAI account (browser flow) |

{virtuai_cli-1.0.1 → virtuai_cli-1.0.3}/README.md

@@ -14,7 +14,7 @@ Or with [pipx](https://pipx.pypa.io/) (recommended for CLI tools — handles PAT
 pipx install virtuai-cli
 ```
 
-For **local model inference** (running agents against [Ollama](https://ollama.com) on your machine — see below), install the `local` extra:
+For **local model inference** (running agents against [Ollama](https://ollama.com) or [oMLX](https://github.com/jundot/omlx) on your machine — see below), install the `local` extra:
 
 ```bash
 pip install 'virtuai-cli[local]'      # or: pipx install 'virtuai-cli[local]'
@@ -62,27 +62,58 @@ virtuai ask --json "find any bugs" > events.jsonl
 
 `ask` is non-interactive: it prints the response to stdout, then exits.
 
-## Local model inference (Ollama)
+## Local model inference
 
-Agents can run inference on **your own machine** via [Ollama](https://ollama.com) instead of a cloud provider — useful for cost, data locality, or running open models. Only the model turns are proxied to your machine over the CLI's WebSocket; the agent loop, tools, knowledge bases, and chat history stay on the server.
+Agents can run inference on **your own machine** — with [Ollama](https://ollama.com), or [oMLX](https://github.com/jundot/omlx) on Apple Silicon — instead of a cloud provider: for cost, data locality, or open models. Only the model calls go to the machine; the agent loop, tools, knowledge bases and chat history stay on the server. It works from **every channel** — web chat, Slack, Telegram, WhatsApp, Google Chat, OpenCode — not only the TUI, as long as a runner that can serve the model is connected.
 
-To use it:
+### Set up
 
-1. Install the CLI with the local extra (pulls `langchain-ollama`):
+1. Install the CLI with the local extra (pulls `langchain-ollama`, `langchain-openai` for oMLX, and `llmfit`):
    ```bash
    pip install 'virtuai-cli[local]'
    ```
-2. Install [Ollama](https://ollama.com), start it, and pull a **tool-capable** model:
+2. Install Ollama **0.40 or newer** and pull a validated model:
    ```bash
-   ollama serve
-   ollama pull qwen3:8b          # instruct/agentic models work best for tool calling
+   ollama pull qwen3:8b
    ```
-3. In the VirtuAI portal, set the agent's **Provider** to **Local (Ollama)**, enter the model tag you pulled, and pick a **cloud fallback** model.
-4. Connect the CLI (`virtuai chat`, `virtuai ask`, or `virtuai run`) — model turns are now answered by your local Ollama.
+   Ollama before 0.40 held the same models in two to three times the memory; `virtuai run` warns when it finds one.
+3. In the VirtuAI portal, set the agent's **Provider** to **Local (Ollama)** (or **Local (oMLX)**), enter the model, and pick a **cloud fallback** model.
+4. Keep a runner connected: `virtuai run`, or `virtuai chat` / `virtuai ask`, which start their own.
 
-The CLI talks to Ollama at `http://localhost:11434` by default; override with the `OLLAMA_BASE_URL` environment variable.
+### Which models
 
-If the CLI isn't connected, Ollama isn't running, or the model isn't pulled, the agent **falls back to its configured cloud model** and shows a notice. For reliable tool use, prefer instruct/agentic models (e.g. `qwen3:*`, `qwen2.5:*`); some code-tuned or non-instruct models emit tool calls as plain text and won't drive the agent's tools.
+Validated — every one of 10 tool-calling tasks right, 8k context, 16 GB Apple M3 with Ollama 0.40:
+
+| Model | Memory | Median answer |
+|---|---|---|
+| `qwen3:8b` | 6.6 GB | 2.2 s |
+| `qwen2.5:7b` | 5.0 GB | 3.5 s |
+| `qwen2.5-coder:7b` | ~5 GB | 4.4 s |
+| `qwen2.5:14b` | 10.3 GB | 6.0 s |
+
+Other models run, but are not vouched for. `qwen3:4b` is accurate but slow (it reasons before every call). Gemma 3 is never used: Ollama cannot call tools with it, so the cloud fallback answers.
+
+### Whose turns this machine answers
+
+```bash
+virtuai run                              # your own turns only (the default)
+virtuai run --serve-models workspace     # anyone's in the workspace: a shared Mac mini or GPU box
+virtuai run --serve-models none          # tools only, no model turns
+```
+
+Each model call goes to the runner of the person the turn is for; if they have none, to a runner started with `--serve-models workspace`; if there is none either, the agent's **cloud fallback** answers that call and every channel shows "Modelo local no disponible: responde …". A runner that fails before it answers (Ollama stopped, the model unloaded) costs the call a cloud answer, not an error.
+
+> **Upgrading from 1.0.2:** with a current server, an older runner serves only the person who paired it. A machine that serves a whole team needs 1.0.3 and `--serve-models workspace`.
+
+### What the runner tells the server
+
+At connect and then every minute: the models it has, the Ollama version, what is loaded and how much memory it really took. With `llmfit` installed it also estimates which pulled models fit this machine, so the server does not send a model to a runner that cannot hold it until it has actually run there. A model that spills to the CPU gets half the context on its next call on that machine (down to 4k).
+
+The context sent to Ollama is 8k by default (set on the server); `OLLAMA_NUM_CTX` overrides it on a machine with less memory. For qwen3 the runner turns Ollama's thinking mode off: left on, it reasoned for minutes before every tool call.
+
+### oMLX
+
+The runner finds oMLX through its own settings (`~/.omlx/settings.json`: port and API key) or `VIRTUAI_OMLX_URL` / `OMLX_API_KEY`; the key never leaves the machine. Measured with `Qwen3-8B-4bit`: 10/10, 2.5 s per answer — on par with Ollama for one person. Its case is a shared host serving several people at once; raise `VIRTUAI_MAX_CONCURRENT_MODEL` there, since the runner otherwise sends one generation at a time.
 
 ## Environment variables
 
@@ -97,11 +128,15 @@ mean "allow more commands at once" silently launched more browsers.
 | `VIRTUAI_ISOLATE_SESSIONS` | off | Give each conversation its own directory under the workdir, and each person their own browser. Turn it on when one runner serves several people. Also set by `--isolate-sessions`. |
 | `VIRTUAI_MAX_NAMESPACES` | `5` | Browsers held at once — one per person. Each is a real headful Chrome (roughly 0.4–1 GB resident), so size this to the machine's memory: an OOM-killed Chrome is a worse failure than a clean refusal. Only applies with isolation on. Also set by `--max-namespaces`. |
 | `VIRTUAI_MAX_CONCURRENT_WORK` | `4` | Commands running at once. Each conversation is already capped at one by its own queue, so in practice this is how many conversations may be executing at the same instant. |
-| `VIRTUAI_MAX_CONCURRENT_MODEL` | `1` | Local model generations at once. Ollama does one well; its own parallelism lives in `OLLAMA_NUM_PARALLEL` on your machine, which the server cannot see. |
+| `VIRTUAI_MAX_CONCURRENT_MODEL` | `1` | Local model generations at once. Ollama does one well; its own parallelism lives in `OLLAMA_NUM_PARALLEL` on your machine, which the server cannot see. Raise it for oMLX on a shared host, which batches several at once. |
 | `VIRTUAI_MAX_QUEUED_FRAMES` | `64` | Requests waiting for a slot. Past this the runner answers "busy" rather than queueing without bound. |
 | `VIRTUAI_SESSION_IDLE_TIMEOUT` | off (`0`) | Seconds before an idle browser stops holding its slot. Off by default: a slot normally frees only when its browser closes or the runner restarts. |
 | `VIRTUAI_SANDBOX_OWNER` | unset | The one person this runner serves, reported to the server so a browser action reaches *their* browser. Leave it unset on a laptop or a container shared by a team: the server then routes to it for anyone, which is the behaviour that existed before per-person sandboxes. Set it when a workspace has a sandbox per person, or actions land on whichever runner connected last. |
+| `VIRTUAI_SERVE_MODELS` | `own` | Whose local-model turns this runner answers: `own`, `workspace` or `none`. Also set by `--serve-models`. |
 | `OLLAMA_BASE_URL` | `http://localhost:11434` | Where to reach the local Ollama. |
+| `OLLAMA_NUM_CTX` | unset | The context to ask Ollama for, instead of the server's (8k). Lower it on a machine with little memory. |
+| `VIRTUAI_OMLX_URL` | from `~/.omlx/settings.json`, else `http://127.0.0.1:8000` | Where to reach oMLX. |
+| `OMLX_API_KEY` | from `~/.omlx/settings.json` | oMLX's API key. Stays on this machine. |
 
 Server-side, `VIRTUAI_ENV_WAIT_SECONDS` (default `45`) is how long the server
 holds a message waiting for a runner to reconnect before telling the user the
@@ -116,7 +151,7 @@ environment is not responding.
 | `virtuai opencode`      | Merge this workspace's agents into the local OpenCode TUI config |
 | `virtuai pair <code>`    | Pair this machine with a workspace |
 | `virtuai unpair`         | Revoke the current pairing |
-| `virtuai run`            | Headless WebSocket runner — let the web app/channels invoke local tools without opening the TUI |
+| `virtuai run`            | Headless runner — let the web app and channels use this machine's tools and local models without opening the TUI (`--serve-models own\|workspace\|none`) |
 | `virtuai status`         | Show pairing and connection status |
 | `virtuai logs`           | Show recent commands the agent executed locally |
 | `virtuai login`          | Authenticate with your VirtuAI account (browser flow) |

{virtuai_cli-1.0.1 → virtuai_cli-1.0.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "virtuai-cli"
-version = "1.0.1"
+version = "1.0.3"
 description = "Run VirtuAI deep agents on your local machine"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -35,7 +35,12 @@ dependencies = [
 # Ollama. Install with: pip install 'virtuai-cli[local]'
 [project.optional-dependencies]
 local = [
-    "langchain-ollama>=0.2",
+    "langchain-ollama>=0.3.4",  # `reasoning` (think on/off)
+    # Which pulled models fit this machine, before they are first loaded
+    # (virtuai_cli/ollama_state.py). Ships its own binary in the wheel.
+    "llmfit>=1.1.16",
+    # oMLX (MLX on Apple Silicon) speaks the OpenAI API (virtuai_cli/omlx_state.py).
+    "langchain-openai>=0.3",
 ]
 browser = [
     "browser-use>=0.1.40",

{virtuai_cli-1.0.1 → virtuai_cli-1.0.3}/src/virtuai_cli/__init__.py

@@ -1,2 +1,2 @@
 """VirtuAI local CLI."""
-__version__ = "1.0.1"
+__version__ = "1.0.3"

{virtuai_cli-1.0.1 → virtuai_cli-1.0.3}/src/virtuai_cli/main.py

@@ -242,6 +242,13 @@ def run(
              "per person. Each is a real Chrome (roughly half a gigabyte), so "
              "size this to the machine's memory. Default 5.",
     ),
+    serve_models: Optional[str] = typer.Option(
+        None,
+        "--serve-models",
+        help="Whose local-model turns this machine answers (Ollama or oMLX): "
+             "'own' (yours — the default), 'workspace' (anyone's: a shared "
+             "inference host such as a Mac mini or a GPU box) or 'none'.",
+    ),
 ):
     """Start the local runner. Keep this running while using VirtuAI."""
     lock = None
@@ -276,6 +283,11 @@ def run(
     # None means "not passed", so the env var a container already set survives.
     if max_namespaces is not None:
         os.environ["VIRTUAI_MAX_NAMESPACES"] = str(max_namespaces)
+    if serve_models is not None:
+        if serve_models not in ("own", "workspace", "none"):
+            console.print("[red]--serve-models must be own, workspace or none.[/red]")
+            raise typer.Exit(2)
+        os.environ["VIRTUAI_SERVE_MODELS"] = serve_models
     console.print(f"[bold]VirtuAI CLI[/bold] v{__version__} — Press Ctrl+C to stop.")
     # Re-read rather than testing the flags, so the line reports the effective
     # value whichever way it was set.
@@ -289,6 +301,19 @@ def run(
             )
         except ImportError:  # the browser extra is not installed
             pass
+    from virtuai_cli.runner import _serve_models
+    if _serve_models() != "none":
+        from virtuai_cli import ollama_state
+        _ollama = asyncio.run(ollama_state.probe())
+        if ollama_state.outdated(_ollama.get("version")):
+            console.print(
+                f"[yellow]Ollama {_ollama['version']} is old: before 0.40 it held the same "
+                "models in two to three times the memory. Update it for local models.[/yellow]"
+            )
+    if _serve_models() == "workspace":
+        console.print("[dim]Serving local-model turns for the whole workspace[/dim]")
+    elif _serve_models() == "none":
+        console.print("[dim]Not serving local-model turns[/dim]")
 
     from virtuai_cli.runner import run_forever
     try:

virtuai_cli-1.0.3/src/virtuai_cli/ollama_state.py

@@ -0,0 +1,245 @@
+"""What this machine's Ollama can serve, kept current for the server.
+
+The runner used to report Ollama once, at connect: a model pulled afterwards,
+or Ollama stopped, was invisible to the server until the runner reconnected.
+And what a model needs was assumed, not measured — on a 16 GB Mac, Ollama
+0.35 held qwen3:8b in 15 GB with a quarter on CPU (0.1 tokens/s), while 0.40
+holds it in 6.6 GB.
+
+- `probe()` — reachable, models, version, and what is loaded with its real
+  footprint (`/api/ps`).
+- `status_loop()` — re-probes every STATUS_SECONDS and sends what changed.
+- `fit_report()` — with llmfit installed (the `[local]` extra), an estimate
+  of which pulled models fit this hardware, before they are first loaded.
+- `after_call()` / `context_for()` — a model that spilled to CPU gets half
+  the context on its next call here, down to MIN_NUM_CTX.
+
+docs/workitems/local-models.md (F5).
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import logging
+import os
+import shutil
+import sys
+from pathlib import Path
+from typing import Any, Awaitable, Callable, Optional
+
+logger = logging.getLogger(__name__)
+
+# The same endpoint the runner answers model_generate frames from.
+OLLAMA_BASE_URL = os.environ.get("OLLAMA_BASE_URL", "http://localhost:11434")
+
+STATUS_SECONDS = 60.0
+#: Below this, Ollama held the same tags in two to three times the memory.
+MIN_GOOD_VERSION = (0, 40)
+MIN_NUM_CTX = 4_096
+_LLMFIT_TIMEOUT = 90.0
+
+#: model tag -> the context it is capped at here, after it spilled to CPU.
+_ctx_cap: dict[str, int] = {}
+#: The last fit report, recomputed only when the pulled models change.
+_fit: dict[str, Any] = {"models": None, "report": {}}
+
+
+def _url(path: str) -> str:
+    return f"{OLLAMA_BASE_URL.rstrip('/')}{path}"
+
+
+async def probe() -> dict:
+    """{"reachable", "models", "version", "loaded"}; never raises."""
+    try:
+        import httpx
+    except Exception:
+        return {"reachable": False, "models": []}
+    try:
+        async with httpx.AsyncClient(timeout=2.0) as client:
+            resp = await client.get(_url("/api/tags"))
+            resp.raise_for_status()
+            models = [m.get("name", "") for m in resp.json().get("models", []) if m.get("name")]
+            state: dict = {"reachable": True, "models": models}
+            # The version decides more than the model's size: 0.35 needed two
+            # to three times the memory 0.40 does for the same tag.
+            state["version"] = await _get_json(client, "/api/version", "version")
+            state["loaded"] = _loaded(await _get_json(client, "/api/ps", "models") or [])
+            if _ctx_cap:
+                state["context_cap"] = dict(_ctx_cap)
+            return state
+    except Exception:
+        return {"reachable": False, "models": []}
+
+
+async def _get_json(client: Any, path: str, key: str) -> Any:
+    try:
+        resp = await client.get(_url(path))
+        return resp.json().get(key) if resp.status_code == 200 else None
+    except Exception:
+        return None
+
+
+def _loaded(running: list) -> dict:
+    """{tag: {"size_gb", "on_gpu_pct", "num_ctx"}} for each loaded model."""
+    out = {}
+    for m in running:
+        size, vram = m.get("size") or 0, m.get("size_vram") or 0
+        if not m.get("name") or not size:
+            continue
+        out[m["name"]] = {"size_gb": round(size / 1e9, 2),
+                          "on_gpu_pct": round(100 * vram / size),
+                          "num_ctx": m.get("context_length")}
+    return out
+
+
+def outdated(version: Optional[str]) -> bool:
+    """True for an Ollama older than MIN_GOOD_VERSION; False when unknown."""
+    try:
+        parts = tuple(int(p) for p in (version or "").split(".")[:2])
+    except ValueError:
+        return False
+    return len(parts) == 2 and parts < MIN_GOOD_VERSION
+
+
+# ---------------------------------------------------------------------------
+# Context per model, from what it really took
+# ---------------------------------------------------------------------------
+
+def context_for(model: str, requested: Optional[int]) -> Optional[int]:
+    """The context to ask for: the server's, unless this model spilled here."""
+    cap = _ctx_cap.get(model)
+    if cap is None:
+        return requested
+    return min(requested, cap) if requested else cap
+
+
+async def after_call(model: str) -> None:
+    """Halve a model's context here if it did not fit on the GPU."""
+    state = await probe()
+    info = (state.get("loaded") or {}).get(model)
+    if not info or info.get("on_gpu_pct", 100) >= 100:
+        return
+    current = info.get("num_ctx") or _ctx_cap.get(model) or 8_192
+    lower = max(MIN_NUM_CTX, current // 2)
+    if lower < current:
+        _ctx_cap[model] = lower
+        logger.warning("%s ran %s%% on GPU at %s tokens of context; next calls use %s",
+                       model, info.get("on_gpu_pct"), current, lower)
+
+
+# ---------------------------------------------------------------------------
+# llmfit: an estimate before a model is first loaded
+# ---------------------------------------------------------------------------
+
+def _llmfit() -> Optional[str]:
+    """The llmfit binary: on PATH, or beside this interpreter (the pip wheel)."""
+    found = shutil.which("llmfit")
+    if found:
+        return found
+    beside = Path(sys.executable).parent / ("llmfit.exe" if os.name == "nt" else "llmfit")
+    return str(beside) if beside.exists() else None
+
+
+async def _run_json(*args: str) -> Optional[dict]:
+    try:
+        proc = await asyncio.create_subprocess_exec(
+            *args, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.DEVNULL)
+        out, _ = await asyncio.wait_for(proc.communicate(), timeout=_LLMFIT_TIMEOUT)
+        return json.loads(out) if proc.returncode == 0 else None
+    except Exception as exc:
+        logger.debug("llmfit %s failed: %s", args[1:3], exc)
+        return None
+
+
+async def _quantization(model: str) -> Optional[str]:
+    try:
+        import httpx
+
+        async with httpx.AsyncClient(timeout=5.0) as client:
+            resp = await client.post(_url("/api/show"), json={"model": model})
+            return (resp.json().get("details") or {}).get("quantization_level") if resp.status_code == 200 else None
+    except Exception:
+        return None
+
+
+async def fit_report(models: list[str], num_ctx: int) -> dict:
+    """{tag: {"fits", "memory_gb", "tps"}} for the pulled models llmfit knows.
+
+    Estimated at the quantization Ollama actually holds (llmfit's default is
+    Q8_0; Ollama's tags are mostly Q4_K_M). Measured on a 16 GB M3 against
+    Ollama 0.40: within ~0.5 GB. Empty without llmfit.
+    """
+    binary = _llmfit()
+    if not binary or not models:
+        return {}
+    catalog = await _run_json(binary, "--max-context", str(num_ctx), "recommend", "--no-dashboard",
+                              "-n", "100000", "--force-runtime", "llamacpp", "--json")
+    names = {}
+    for m in (catalog or {}).get("models") or []:
+        tag = m.get("ollama_name")
+        if tag and tag not in names:
+            names[tag] = m.get("name")
+    report = {}
+    for tag in models:
+        hf = names.get(tag) or names.get(tag.split(":")[0] + ":latest")
+        if not hf:
+            continue
+        args = [binary, "plan", hf, "--context", str(num_ctx), "--json", "--no-dashboard"]
+        quant = await _quantization(tag)
+        if quant:
+            args += ["--quant", quant]
+        plan = await _run_json(*args)
+        if not plan:
+            continue
+        current = plan.get("current") or {}
+        level = current.get("fit_level")
+        report[tag] = {"fits": level in ("Perfect", "Good", "Marginal"), "fit_level": level,
+                       "memory_gb": round((plan.get("minimum") or {}).get("vram_gb") or 0, 2),
+                       "tps": round(current.get("estimated_tps") or 0, 1)}
+    return report
+
+
+# ---------------------------------------------------------------------------
+# Keeping the server current
+# ---------------------------------------------------------------------------
+
+async def current(num_ctx: int) -> dict:
+    """The probe, with the fit report for the models pulled now."""
+    state = await probe()
+    models = state.get("models") or []
+    if state.get("reachable") and models != _fit["models"]:
+        _fit["models"], _fit["report"] = list(models), await fit_report(models, num_ctx)
+    if _fit["report"]:
+        state["fit"] = _fit["report"]
+    return state
+
+
+def _num_ctx() -> int:
+    try:
+        return int(os.environ.get("OLLAMA_NUM_CTX") or 8_192)
+    except ValueError:
+        return 8_192
+
+
+async def snapshot() -> dict:
+    """{"ollama": ..., "omlx": ...}: what a runner_status carries."""
+    from virtuai_cli import omlx_state
+
+    return {"ollama": await current(_num_ctx()), "omlx": await omlx_state.probe()}
+
+
+async def status_loop(send: Callable[[dict], Awaitable[None]], interval: float = STATUS_SECONDS) -> None:
+    """Send the local model servers' state whenever it changes; runs until cancelled."""
+    last = None
+    while True:
+        try:
+            state = await snapshot()
+            if state != last:
+                await send(state)
+                last = state
+        except asyncio.CancelledError:
+            raise
+        except Exception as exc:  # a failed report must not take the runner down
+            logger.debug("Ollama status not sent: %s", exc)
+        await asyncio.sleep(interval)

virtuai_cli-1.0.3/src/virtuai_cli/omlx_state.py

@@ -0,0 +1,83 @@
+"""oMLX on this machine: where it listens, what it serves (F6).
+
+oMLX (https://github.com/jundot/omlx) serves MLX models on Apple Silicon
+behind an OpenAI-compatible API. It used to be called from the *server*, at
+localhost — which is the server's own machine, so it only ever worked in
+development. Now it is a runner-side provider like Ollama: the runner calls
+it, and its API key stays on this machine.
+
+Where it listens and its key: VIRTUAI_OMLX_URL and OMLX_API_KEY, else what
+oMLX's own settings say (~/.omlx/settings.json), else localhost:8000.
+
+Thinking: a model the server says has a thinking mode to turn off (qwen3)
+gets `chat_template_kwargs: {"enable_thinking": false}` on every call — the
+same value every time, since an MLX server's prompt cache is keyed on the
+rendered template and a value that changed between calls would miss it.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+from typing import Optional
+
+_SETTINGS = Path.home() / ".omlx" / "settings.json"
+_DEFAULT_URL = "http://127.0.0.1:8000"
+
+
+def _settings() -> dict:
+    try:
+        return json.loads(_SETTINGS.read_text())
+    except Exception:
+        return {}
+
+
+def base_url() -> str:
+    """oMLX's root URL, without /v1."""
+    env = (os.environ.get("VIRTUAI_OMLX_URL") or "").strip()
+    if env:
+        return env.rstrip("/").removesuffix("/v1")
+    server = _settings().get("server") or {}
+    if server.get("port"):
+        return f"http://{server.get('host') or '127.0.0.1'}:{server['port']}"
+    return _DEFAULT_URL
+
+
+def api_key() -> Optional[str]:
+    key = (os.environ.get("OMLX_API_KEY") or "").strip()
+    return key or ((_settings().get("auth") or {}).get("api_key") or None)
+
+
+async def probe() -> dict:
+    """{"reachable", "models"}; never raises."""
+    try:
+        import httpx
+
+        headers = {"Authorization": f"Bearer {api_key()}"} if api_key() else {}
+        async with httpx.AsyncClient(timeout=2.0) as client:
+            resp = await client.get(f"{base_url()}/v1/models", headers=headers)
+            resp.raise_for_status()
+            data = resp.json()
+            items = data if isinstance(data, list) else data.get("data") or []
+            return {"reachable": True,
+                    "models": [m.get("id") for m in items if isinstance(m, dict) and m.get("id")]}
+    except Exception:
+        return {"reachable": False, "models": []}
+
+
+def chat_model(model: str, temperature: Optional[float], max_tokens: Optional[int],
+               think: Optional[bool] = None):
+    """The LangChain model the runner streams an oMLX call from."""
+    from langchain_openai import ChatOpenAI
+
+    kwargs = {"model": model, "base_url": f"{base_url()}/v1",
+              # oMLX may skip key checks; the client still wants one.
+              "api_key": api_key() or "omlx", "stream_usage": True}
+    if think is not None:
+        kwargs["extra_body"] = {"chat_template_kwargs": {"enable_thinking": bool(think)}}
+    if temperature is not None:
+        kwargs["temperature"] = temperature
+    if max_tokens is not None:
+        kwargs["max_tokens"] = max_tokens
+    return ChatOpenAI(**kwargs)