llm-rapid-mlx 0.1.0__tar.gz

llm_rapid_mlx-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Raullen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm_rapid_mlx-0.1.0/PKG-INFO

@@ -0,0 +1,152 @@
+Metadata-Version: 2.4
+Name: llm-rapid-mlx
+Version: 0.1.0
+Summary: Run LLM prompts against a local Rapid-MLX server
+Author-email: Raullen <989846+raullenchai@users.noreply.github.com>
+License-Expression: MIT
+Keywords: llm,rapid-mlx,mlx,local
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: llm<0.38,>=0.37.2
+Requires-Dist: openai<4,>=3
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == "test"
+Requires-Dist: pytest-asyncio>=0.24; extra == "test"
+Dynamic: license-file
+
+# llm-rapid-mlx
+
+A plugin for [LLM](https://llm.datasette.io/) to use models served by
+[Rapid-MLX](https://github.com/raullenchai/Rapid-MLX), a local inference server
+for Apple Silicon built on MLX. The plugin is a Python HTTP client; only the
+server requires Apple Silicon.
+
+## Installation
+
+Requires Python 3.11+ and LLM 0.37.2. From this checkout:
+
+```sh
+llm install -e .
+```
+
+Or install the locally built wheel with `llm install dist/llm_rapid_mlx-0.1.0-py3-none-any.whl`.
+This package has not yet been published to PyPI.
+
+Install and start Rapid-MLX separately (the server requires Python 3.12+ if
+installed through pip):
+
+```sh
+brew install rapid-mlx
+rapid-mlx serve qwen3.5-4b-4bit --port 8000
+```
+
+The plugin neither starts servers nor downloads model weights.
+
+## Usage
+
+```sh
+llm rapid-mlx models
+llm models -q rapid-mlx
+llm -m rapid-mlx/mlx-community/Qwen3.5-4B-MLX-4bit 'say hi'
+llm -m rapid-mlx/mlx-community/Qwen3.5-4B-MLX-4bit -s 'Answer briefly' 'Remember the word pelican'
+llm -c 'What word did I ask you to remember?'
+```
+
+Use the IDs shown by discovery. They are always `rapid-mlx/<server-id>`;
+the prefix is removed before sending the request. Streaming is on by default;
+`--no-stream` uses a non-streaming request. Standard LLM conversation logging,
+`llm chat`, options such as `-o temperature 0.5` and `-o max_tokens 128`,
+and Python sync/async models work through LLM's OpenAI adapter.
+
+```sh
+llm -m rapid-mlx/qwen3.5-4b-4bit --functions 'def add(a: int, b: int) -> int: return a + b' 'Use add to calculate 17 + 25'
+llm -m rapid-mlx/qwen3.5-4b-4bit --schema '{"type":"object","properties":{"greeting":{"type":"string"}},"required":["greeting"]}' 'Say hello in JSON'
+```
+
+Function tools are enabled when discovery advertises `tools`. JSON schemas are
+passed through using OpenAI's `response_format`; successful generation and
+schema enforcement depend on the model and server. Vision models advertising
+`vision` accept PNG, JPEG, WebP and GIF attachments through `-a image.png`.
+Audio, video and PDF attachments are outside this plugin's scope.
+
+## Configuration
+
+The default endpoint is `http://localhost:8000/v1`.
+
+```sh
+export RAPID_MLX_BASE_URL=http://127.0.0.1:8477/v1
+# Persistent alternative (stores a URL, not a secret):
+llm keys set rapid-mlx-base-url --value http://127.0.0.1:8477/v1
+```
+
+Endpoint precedence: `-o base_url URL` for an individual chat request, then
+`RAPID_MLX_BASE_URL`, then the `rapid-mlx-base-url` entry in LLM's key store,
+then the default. Discovery and embedding requests use the configured endpoint;
+a chat option alone cannot discover models on a different server. Set the
+environment variable or persistent URL before listing/selecting those models.
+A URL without `/v1` is normalized to include it, including reverse proxy paths.
+
+Authentication is optional. For an authenticated server:
+
+```sh
+llm keys set rapid-mlx
+# Or: export RAPID_MLX_API_KEY=your-secret
+llm -m rapid-mlx/qwen3.5-4b-4bit --key your-key 'Hello'
+```
+
+Keys follow LLM precedence: Python `model.key`, explicit `--key` (literal or key
+alias), stored `rapid-mlx` key, then `RAPID_MLX_API_KEY`. Discovery uses the stored
+key/environment variable, since model discovery runs before a prompt's `--key`.
+Without a key the client sends the harmless placeholder `not-needed`, accepted
+by Rapid-MLX's default unauthenticated mode. `OPENAI_API_KEY` is never used.
+
+## Discovery and embeddings
+
+Each registration reads `GET /v1/models` with a two-second timeout and no retries.
+There is no persistent catalogue cache. Restart an interactive LLM session after
+changing the server's model catalogue. Offline, unauthenticated or malformed
+catalogues register no Rapid-MLX models, leaving other plugins usable.
+`llm rapid-mlx models` reports discovery failures explicitly.
+
+Models tagged `embedding` are registered only as embedding models. Names are
+never used to guess embedding support. Entries with `text` are chat models;
+older entries without `capabilities` are treated as text-only. Other modalities
+are excluded from chat registration.
+
+If the server advertises an embedding model:
+
+```sh
+llm embed-models
+llm embed -m rapid-mlx/your-discovered-embedding-model -c 'A coastal bird'
+```
+
+Embeddings use `/v1/embeddings`, text input and batches of up to 32. No embedding
+model is registered when the server only serves chat. Embedding vectors are
+returned in input order, with indexes and finite numeric values checked.
+
+## Development
+
+```sh
+python3.12 -m venv .venv
+.venv/bin/python -m pip install llm '.[test]' build twine
+.venv/bin/llm install -e .
+.venv/bin/python -m pytest -q
+.venv/bin/python -m build
+.venv/bin/python -m twine check dist/*
+```
+
+Offline tests use a mocked HTTP server and exercise the real LLM/OpenAI adapters.
+Opt in to a small live smoke test against an existing server:
+
+```sh
+RAPID_MLX_LIVE_URL=http://127.0.0.1:8477/v1 .venv/bin/python -m pytest -m live -q
+```
+
+The optional test does not start or stop a server. Do not set that variable for
+offline tests. The dependency on LLM's bundled OpenAI adapter is constrained to
+0.37.x because that adapter is not a separately versioned public API.
+
+## License
+
+MIT. See [LICENSE](LICENSE).

llm_rapid_mlx-0.1.0/README.md

@@ -0,0 +1,135 @@
+# llm-rapid-mlx
+
+A plugin for [LLM](https://llm.datasette.io/) to use models served by
+[Rapid-MLX](https://github.com/raullenchai/Rapid-MLX), a local inference server
+for Apple Silicon built on MLX. The plugin is a Python HTTP client; only the
+server requires Apple Silicon.
+
+## Installation
+
+Requires Python 3.11+ and LLM 0.37.2. From this checkout:
+
+```sh
+llm install -e .
+```
+
+Or install the locally built wheel with `llm install dist/llm_rapid_mlx-0.1.0-py3-none-any.whl`.
+This package has not yet been published to PyPI.
+
+Install and start Rapid-MLX separately (the server requires Python 3.12+ if
+installed through pip):
+
+```sh
+brew install rapid-mlx
+rapid-mlx serve qwen3.5-4b-4bit --port 8000
+```
+
+The plugin neither starts servers nor downloads model weights.
+
+## Usage
+
+```sh
+llm rapid-mlx models
+llm models -q rapid-mlx
+llm -m rapid-mlx/mlx-community/Qwen3.5-4B-MLX-4bit 'say hi'
+llm -m rapid-mlx/mlx-community/Qwen3.5-4B-MLX-4bit -s 'Answer briefly' 'Remember the word pelican'
+llm -c 'What word did I ask you to remember?'
+```
+
+Use the IDs shown by discovery. They are always `rapid-mlx/<server-id>`;
+the prefix is removed before sending the request. Streaming is on by default;
+`--no-stream` uses a non-streaming request. Standard LLM conversation logging,
+`llm chat`, options such as `-o temperature 0.5` and `-o max_tokens 128`,
+and Python sync/async models work through LLM's OpenAI adapter.
+
+```sh
+llm -m rapid-mlx/qwen3.5-4b-4bit --functions 'def add(a: int, b: int) -> int: return a + b' 'Use add to calculate 17 + 25'
+llm -m rapid-mlx/qwen3.5-4b-4bit --schema '{"type":"object","properties":{"greeting":{"type":"string"}},"required":["greeting"]}' 'Say hello in JSON'
+```
+
+Function tools are enabled when discovery advertises `tools`. JSON schemas are
+passed through using OpenAI's `response_format`; successful generation and
+schema enforcement depend on the model and server. Vision models advertising
+`vision` accept PNG, JPEG, WebP and GIF attachments through `-a image.png`.
+Audio, video and PDF attachments are outside this plugin's scope.
+
+## Configuration
+
+The default endpoint is `http://localhost:8000/v1`.
+
+```sh
+export RAPID_MLX_BASE_URL=http://127.0.0.1:8477/v1
+# Persistent alternative (stores a URL, not a secret):
+llm keys set rapid-mlx-base-url --value http://127.0.0.1:8477/v1
+```
+
+Endpoint precedence: `-o base_url URL` for an individual chat request, then
+`RAPID_MLX_BASE_URL`, then the `rapid-mlx-base-url` entry in LLM's key store,
+then the default. Discovery and embedding requests use the configured endpoint;
+a chat option alone cannot discover models on a different server. Set the
+environment variable or persistent URL before listing/selecting those models.
+A URL without `/v1` is normalized to include it, including reverse proxy paths.
+
+Authentication is optional. For an authenticated server:
+
+```sh
+llm keys set rapid-mlx
+# Or: export RAPID_MLX_API_KEY=your-secret
+llm -m rapid-mlx/qwen3.5-4b-4bit --key your-key 'Hello'
+```
+
+Keys follow LLM precedence: Python `model.key`, explicit `--key` (literal or key
+alias), stored `rapid-mlx` key, then `RAPID_MLX_API_KEY`. Discovery uses the stored
+key/environment variable, since model discovery runs before a prompt's `--key`.
+Without a key the client sends the harmless placeholder `not-needed`, accepted
+by Rapid-MLX's default unauthenticated mode. `OPENAI_API_KEY` is never used.
+
+## Discovery and embeddings
+
+Each registration reads `GET /v1/models` with a two-second timeout and no retries.
+There is no persistent catalogue cache. Restart an interactive LLM session after
+changing the server's model catalogue. Offline, unauthenticated or malformed
+catalogues register no Rapid-MLX models, leaving other plugins usable.
+`llm rapid-mlx models` reports discovery failures explicitly.
+
+Models tagged `embedding` are registered only as embedding models. Names are
+never used to guess embedding support. Entries with `text` are chat models;
+older entries without `capabilities` are treated as text-only. Other modalities
+are excluded from chat registration.
+
+If the server advertises an embedding model:
+
+```sh
+llm embed-models
+llm embed -m rapid-mlx/your-discovered-embedding-model -c 'A coastal bird'
+```
+
+Embeddings use `/v1/embeddings`, text input and batches of up to 32. No embedding
+model is registered when the server only serves chat. Embedding vectors are
+returned in input order, with indexes and finite numeric values checked.
+
+## Development
+
+```sh
+python3.12 -m venv .venv
+.venv/bin/python -m pip install llm '.[test]' build twine
+.venv/bin/llm install -e .
+.venv/bin/python -m pytest -q
+.venv/bin/python -m build
+.venv/bin/python -m twine check dist/*
+```
+
+Offline tests use a mocked HTTP server and exercise the real LLM/OpenAI adapters.
+Opt in to a small live smoke test against an existing server:
+
+```sh
+RAPID_MLX_LIVE_URL=http://127.0.0.1:8477/v1 .venv/bin/python -m pytest -m live -q
+```
+
+The optional test does not start or stop a server. Do not set that variable for
+offline tests. The dependency on LLM's bundled OpenAI adapter is constrained to
+0.37.x because that adapter is not a separately versioned public API.
+
+## License
+
+MIT. See [LICENSE](LICENSE).

llm_rapid_mlx-0.1.0/llm_rapid_mlx.egg-info/PKG-INFO

@@ -0,0 +1,152 @@
+Metadata-Version: 2.4
+Name: llm-rapid-mlx
+Version: 0.1.0
+Summary: Run LLM prompts against a local Rapid-MLX server
+Author-email: Raullen <989846+raullenchai@users.noreply.github.com>
+License-Expression: MIT
+Keywords: llm,rapid-mlx,mlx,local
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: llm<0.38,>=0.37.2
+Requires-Dist: openai<4,>=3
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == "test"
+Requires-Dist: pytest-asyncio>=0.24; extra == "test"
+Dynamic: license-file
+
+# llm-rapid-mlx
+
+A plugin for [LLM](https://llm.datasette.io/) to use models served by
+[Rapid-MLX](https://github.com/raullenchai/Rapid-MLX), a local inference server
+for Apple Silicon built on MLX. The plugin is a Python HTTP client; only the
+server requires Apple Silicon.
+
+## Installation
+
+Requires Python 3.11+ and LLM 0.37.2. From this checkout:
+
+```sh
+llm install -e .
+```
+
+Or install the locally built wheel with `llm install dist/llm_rapid_mlx-0.1.0-py3-none-any.whl`.
+This package has not yet been published to PyPI.
+
+Install and start Rapid-MLX separately (the server requires Python 3.12+ if
+installed through pip):
+
+```sh
+brew install rapid-mlx
+rapid-mlx serve qwen3.5-4b-4bit --port 8000
+```
+
+The plugin neither starts servers nor downloads model weights.
+
+## Usage
+
+```sh
+llm rapid-mlx models
+llm models -q rapid-mlx
+llm -m rapid-mlx/mlx-community/Qwen3.5-4B-MLX-4bit 'say hi'
+llm -m rapid-mlx/mlx-community/Qwen3.5-4B-MLX-4bit -s 'Answer briefly' 'Remember the word pelican'
+llm -c 'What word did I ask you to remember?'
+```
+
+Use the IDs shown by discovery. They are always `rapid-mlx/<server-id>`;
+the prefix is removed before sending the request. Streaming is on by default;
+`--no-stream` uses a non-streaming request. Standard LLM conversation logging,
+`llm chat`, options such as `-o temperature 0.5` and `-o max_tokens 128`,
+and Python sync/async models work through LLM's OpenAI adapter.
+
+```sh
+llm -m rapid-mlx/qwen3.5-4b-4bit --functions 'def add(a: int, b: int) -> int: return a + b' 'Use add to calculate 17 + 25'
+llm -m rapid-mlx/qwen3.5-4b-4bit --schema '{"type":"object","properties":{"greeting":{"type":"string"}},"required":["greeting"]}' 'Say hello in JSON'
+```
+
+Function tools are enabled when discovery advertises `tools`. JSON schemas are
+passed through using OpenAI's `response_format`; successful generation and
+schema enforcement depend on the model and server. Vision models advertising
+`vision` accept PNG, JPEG, WebP and GIF attachments through `-a image.png`.
+Audio, video and PDF attachments are outside this plugin's scope.
+
+## Configuration
+
+The default endpoint is `http://localhost:8000/v1`.
+
+```sh
+export RAPID_MLX_BASE_URL=http://127.0.0.1:8477/v1
+# Persistent alternative (stores a URL, not a secret):
+llm keys set rapid-mlx-base-url --value http://127.0.0.1:8477/v1
+```
+
+Endpoint precedence: `-o base_url URL` for an individual chat request, then
+`RAPID_MLX_BASE_URL`, then the `rapid-mlx-base-url` entry in LLM's key store,
+then the default. Discovery and embedding requests use the configured endpoint;
+a chat option alone cannot discover models on a different server. Set the
+environment variable or persistent URL before listing/selecting those models.
+A URL without `/v1` is normalized to include it, including reverse proxy paths.
+
+Authentication is optional. For an authenticated server:
+
+```sh
+llm keys set rapid-mlx
+# Or: export RAPID_MLX_API_KEY=your-secret
+llm -m rapid-mlx/qwen3.5-4b-4bit --key your-key 'Hello'
+```
+
+Keys follow LLM precedence: Python `model.key`, explicit `--key` (literal or key
+alias), stored `rapid-mlx` key, then `RAPID_MLX_API_KEY`. Discovery uses the stored
+key/environment variable, since model discovery runs before a prompt's `--key`.
+Without a key the client sends the harmless placeholder `not-needed`, accepted
+by Rapid-MLX's default unauthenticated mode. `OPENAI_API_KEY` is never used.
+
+## Discovery and embeddings
+
+Each registration reads `GET /v1/models` with a two-second timeout and no retries.
+There is no persistent catalogue cache. Restart an interactive LLM session after
+changing the server's model catalogue. Offline, unauthenticated or malformed
+catalogues register no Rapid-MLX models, leaving other plugins usable.
+`llm rapid-mlx models` reports discovery failures explicitly.
+
+Models tagged `embedding` are registered only as embedding models. Names are
+never used to guess embedding support. Entries with `text` are chat models;
+older entries without `capabilities` are treated as text-only. Other modalities
+are excluded from chat registration.
+
+If the server advertises an embedding model:
+
+```sh
+llm embed-models
+llm embed -m rapid-mlx/your-discovered-embedding-model -c 'A coastal bird'
+```
+
+Embeddings use `/v1/embeddings`, text input and batches of up to 32. No embedding
+model is registered when the server only serves chat. Embedding vectors are
+returned in input order, with indexes and finite numeric values checked.
+
+## Development
+
+```sh
+python3.12 -m venv .venv
+.venv/bin/python -m pip install llm '.[test]' build twine
+.venv/bin/llm install -e .
+.venv/bin/python -m pytest -q
+.venv/bin/python -m build
+.venv/bin/python -m twine check dist/*
+```
+
+Offline tests use a mocked HTTP server and exercise the real LLM/OpenAI adapters.
+Opt in to a small live smoke test against an existing server:
+
+```sh
+RAPID_MLX_LIVE_URL=http://127.0.0.1:8477/v1 .venv/bin/python -m pytest -m live -q
+```
+
+The optional test does not start or stop a server. Do not set that variable for
+offline tests. The dependency on LLM's bundled OpenAI adapter is constrained to
+0.37.x because that adapter is not a separately versioned public API.
+
+## License
+
+MIT. See [LICENSE](LICENSE).

llm_rapid_mlx-0.1.0/llm_rapid_mlx.egg-info/SOURCES.txt

@@ -0,0 +1,12 @@
+LICENSE
+README.md
+llm_rapid_mlx.py
+pyproject.toml
+llm_rapid_mlx.egg-info/PKG-INFO
+llm_rapid_mlx.egg-info/SOURCES.txt
+llm_rapid_mlx.egg-info/dependency_links.txt
+llm_rapid_mlx.egg-info/entry_points.txt
+llm_rapid_mlx.egg-info/requires.txt
+llm_rapid_mlx.egg-info/top_level.txt
+tests/test_live.py
+tests/test_plugin.py

llm_rapid_mlx-0.1.0/llm_rapid_mlx.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

llm_rapid_mlx-0.1.0/llm_rapid_mlx.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[llm]
+rapid_mlx = llm_rapid_mlx

llm_rapid_mlx-0.1.0/llm_rapid_mlx.egg-info/requires.txt

@@ -0,0 +1,6 @@
+llm<0.38,>=0.37.2
+openai<4,>=3
+
+[test]
+pytest>=8
+pytest-asyncio>=0.24

llm_rapid_mlx-0.1.0/llm_rapid_mlx.egg-info/top_level.txt

@@ -0,0 +1 @@
+llm_rapid_mlx

llm_rapid_mlx-0.1.0/llm_rapid_mlx.py

@@ -0,0 +1,259 @@
+"""LLM integration for the Rapid-MLX OpenAI-compatible API."""
+
+from __future__ import annotations
+
+import copy
+import math
+import os
+from urllib.parse import urlsplit
+
+import click
+import llm
+from llm.default_plugins.openai_models import AsyncChat, Chat
+from openai import AsyncOpenAI, OpenAI, OpenAIError
+from pydantic import Field, field_validator
+
+DEFAULT_BASE_URL = "http://localhost:8000/v1"
+
+
+def normalize_base_url(value: str) -> str:
+    parsed = urlsplit(value)
+    try:
+        port = parsed.port
+    except ValueError:
+        raise ValueError("Rapid-MLX base URL has an invalid port") from None
+    if (
+        parsed.scheme not in {"http", "https"}
+        or not parsed.hostname
+        or parsed.username is not None
+        or parsed.password is not None
+        or parsed.query
+        or parsed.fragment
+        or port == 0
+    ):
+        raise ValueError(
+            "Rapid-MLX base URL must be HTTP(S), without credentials, query or fragment"
+        )
+    value = value.rstrip("/")
+    return value if parsed.path.rstrip("/").endswith("/v1") else value + "/v1"
+
+
+def configured_base_url():
+    # Key storage also gives CLI users a persistent endpoint setting.
+    return normalize_base_url(
+        os.environ.get("RAPID_MLX_BASE_URL")
+        or llm.get_key(key_alias="rapid-mlx-base-url")
+        or DEFAULT_BASE_URL
+    )
+
+
+def resolve_key(explicit_key=None):
+    return llm.get_key(
+        explicit_key=explicit_key, key_alias="rapid-mlx", env_var="RAPID_MLX_API_KEY"
+    )
+
+
+def client_kwargs(base_url, key=None, timeout=120):
+    # Always provide a key so OPENAI_API_KEY cannot leak to a local endpoint.
+    return {
+        "base_url": base_url,
+        "api_key": key or "not-needed",
+        "timeout": timeout,
+        "max_retries": 0,
+    }
+
+
+def model_error(error):
+    status = getattr(error, "status_code", None)
+    detail = f"HTTP {status}" if status else type(error).__name__
+    return llm.ModelError(
+        f"Rapid-MLX request failed ({detail}); check the server and configuration"
+    )
+
+
+def discover_models(base_url=None):
+    """Fetch a fresh catalogue; never infer embeddings from a model's name."""
+    base_url = normalize_base_url(base_url) if base_url else configured_base_url()
+    with OpenAI(**client_kwargs(base_url, resolve_key(), timeout=2)) as client:
+        records = client.models.list().data
+    seen = set()
+    result = []
+    for record in records:
+        if not isinstance(record.id, str) or not record.id or record.id in seen:
+            continue
+        caps = getattr(record, "capabilities", None)
+        if caps is not None and (
+            not isinstance(caps, list) or not all(isinstance(c, str) for c in caps)
+        ):
+            continue
+        seen.add(record.id)
+        # Older servers may omit capabilities: those entries are text only.
+        caps = ["text"] if caps is None else caps
+        result.append((record.id, caps))
+    return base_url, result
+
+
+def _catalogue():
+    try:
+        return discover_models()
+    except (OpenAIError, ValueError, TypeError, AttributeError, OSError):
+        return None, []
+
+
+class Options(Chat.Options):
+    base_url: str | None = Field(
+        default=None, description="Override the Rapid-MLX API base URL for this prompt"
+    )
+
+    @field_validator("base_url")
+    @classmethod
+    def validate_base_url(cls, value):
+        return normalize_base_url(value) if value is not None else None
+
+
+class _RapidMixin:
+    needs_key = "rapid-mlx"
+    key_env_var = "RAPID_MLX_API_KEY"
+    Options = Options
+
+    def __init__(self, server_id, base_url=None, capabilities=None):
+        caps = ["text"] if capabilities is None else capabilities
+        super().__init__(
+            model_id="rapid-mlx/" + server_id,
+            model_name=server_id,
+            api_base=base_url or configured_base_url(),
+            supports_schema=True,
+            supports_tools="tools" in caps,
+            vision="vision" in caps,
+        )
+        # The base also accepts PDFs for OpenAI; Rapid-MLX accepts images here.
+        self.attachment_types.discard("application/pdf")
+
+    def get_key(self, explicit_key=None):
+        return self.key if self.key is not None else resolve_key(explicit_key)
+
+    def get_client(self, key, *, async_=False):
+        factory = AsyncOpenAI if async_ else OpenAI
+        return factory(**client_kwargs(self.api_base, self.get_key(key)))
+
+    def build_kwargs(self, prompt, stream):
+        kwargs = super().build_kwargs(prompt, stream)
+        kwargs.pop("base_url", None)
+        return kwargs
+
+    def for_prompt(self, prompt):
+        # An override must not change a shared registered model/conversation.
+        model = copy.copy(self)
+        model.api_base = prompt.options.base_url or self.api_base
+        return model
+
+    def __str__(self):
+        return f"Rapid-MLX: {self.model_id}"
+
+
+class RapidChat(_RapidMixin, Chat):
+    def execute(self, prompt, stream, response, conversation=None, key=None):
+        model = self.for_prompt(prompt)
+        try:
+            with model.get_client(key) as client:
+                # Let the base adapter own message, stream, usage and tool handling.
+                model.get_client = lambda *args, **kwargs: client
+                yield from Chat.execute(
+                    model, prompt, stream, response, conversation, key
+                )
+        except OpenAIError as ex:
+            raise model_error(ex) from None
+
+
+class RapidAsyncChat(_RapidMixin, AsyncChat):
+    async def execute(self, prompt, stream, response, conversation=None, key=None):
+        model = self.for_prompt(prompt)
+        try:
+            async with model.get_client(key, async_=True) as client:
+                model.get_client = lambda *args, **kwargs: client
+                async for chunk in AsyncChat.execute(
+                    model, prompt, stream, response, conversation, key
+                ):
+                    yield chunk
+        except OpenAIError as ex:
+            raise model_error(ex) from None
+
+
+class RapidEmbedding(llm.EmbeddingModel):
+    needs_key = "rapid-mlx"
+    key_env_var = "RAPID_MLX_API_KEY"
+    supports_text = True
+    supports_binary = False
+    batch_size = 32
+
+    def __init__(self, server_id, base_url=None):
+        self.model_id = "rapid-mlx/" + server_id
+        self.server_id = server_id
+        self.base_url = base_url or configured_base_url()
+
+    def get_key(self, explicit_key=None):
+        return self.key if self.key is not None else resolve_key(explicit_key)
+
+    def embed_batch(self, items, *, key=None):
+        items = list(items)
+        if any(not isinstance(item, str) for item in items):
+            raise ValueError("Rapid-MLX embeddings accept text strings only")
+        if not items:
+            return
+        try:
+            with OpenAI(**client_kwargs(self.base_url, self.get_key(key))) as client:
+                data = client.embeddings.create(
+                    model=self.server_id, input=items, encoding_format="float"
+                ).data
+        except OpenAIError as ex:
+            raise model_error(ex) from None
+        ordered = sorted(data, key=lambda item: item.index)
+        if [item.index for item in ordered] != list(range(len(items))):
+            raise llm.ModelError("Rapid-MLX returned invalid embedding indexes")
+        vectors = [[float(value) for value in item.embedding] for item in ordered]
+        if any(
+            not vector or any(not math.isfinite(v) for v in vector)
+            for vector in vectors
+        ):
+            raise llm.ModelError("Rapid-MLX returned invalid embedding vectors")
+        yield from vectors
+
+
+@llm.hookimpl
+def register_models(register):
+    base_url, records = _catalogue()
+    for server_id, caps in records:
+        if "embedding" not in caps and "text" in caps:
+            register(
+                RapidChat(server_id, base_url, caps),
+                RapidAsyncChat(server_id, base_url, caps),
+            )
+
+
+@llm.hookimpl
+def register_embedding_models(register):
+    base_url, records = _catalogue()
+    for server_id, caps in records:
+        if "embedding" in caps:
+            register(RapidEmbedding(server_id, base_url))
+
+
+@llm.hookimpl
+def register_commands(cli):
+    @cli.group(name="rapid-mlx")
+    def rapid_mlx():
+        "Commands for Rapid-MLX."
+
+    @rapid_mlx.command(name="models")
+    def models():
+        "Discover server models, reporting connection or authentication errors."
+        try:
+            _, records = discover_models()
+        except (OpenAIError, ValueError, TypeError, AttributeError, OSError) as ex:
+            status = getattr(ex, "status_code", None)
+            detail = f"HTTP {status}" if status else type(ex).__name__
+            raise click.ClickException(
+                f"Rapid-MLX discovery failed ({detail}); check RAPID_MLX_BASE_URL and the rapid-mlx key"
+            ) from None
+        for server_id, caps in records:
+            click.echo(f"rapid-mlx/{server_id}\t{', '.join(caps)}")

llm_rapid_mlx-0.1.0/pyproject.toml

@@ -0,0 +1,30 @@
+[project]
+name = "llm-rapid-mlx"
+version = "0.1.0"
+description = "Run LLM prompts against a local Rapid-MLX server"
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{name = "Raullen", email = "989846+raullenchai@users.noreply.github.com"}]
+dependencies = ["llm>=0.37.2,<0.38", "openai>=3,<4"]
+keywords = ["llm", "rapid-mlx", "mlx", "local"]
+
+[project.entry-points.llm]
+rapid_mlx = "llm_rapid_mlx"
+
+[project.optional-dependencies]
+test = ["pytest>=8", "pytest-asyncio>=0.24"]
+
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+py-modules = ["llm_rapid_mlx"]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+asyncio_default_fixture_loop_scope = "function"
+markers = ["live: opt-in tests against an existing Rapid-MLX server"]
+testpaths = ["tests"]

llm_rapid_mlx-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

llm_rapid_mlx-0.1.0/tests/test_live.py

@@ -0,0 +1,43 @@
+"""Opt-in smoke test; never launches or stops a server."""
+
+import os
+
+import pytest
+
+import llm_rapid_mlx as plugin
+
+
+@pytest.mark.live
+def test_live_stream_and_conversation(monkeypatch):
+    url = os.environ.get("RAPID_MLX_LIVE_URL")
+    if not url:
+        pytest.skip("set RAPID_MLX_LIVE_URL to use an existing server")
+    monkeypatch.setenv("RAPID_MLX_BASE_URL", url)
+    base_url, records = plugin.discover_models()
+    server_id, caps = next((name, caps) for name, caps in records if "text" in caps)
+    conversation = plugin.RapidChat(server_id, base_url, caps).conversation()
+    response = conversation.prompt(
+        "Remember the word pelican. Say hi.", system="Answer briefly.", max_tokens=128
+    )
+    chunks = list(response)
+    assert chunks and response.text().strip()
+    assert (
+        "pelican"
+        in conversation.prompt("What word did I ask you to remember?", max_tokens=128)
+        .text()
+        .lower()
+    )
+
+
+@pytest.mark.live
+def test_live_embedding_if_advertised(monkeypatch):
+    url = os.environ.get("RAPID_MLX_LIVE_URL")
+    if not url:
+        pytest.skip("set RAPID_MLX_LIVE_URL to use an existing server")
+    monkeypatch.setenv("RAPID_MLX_BASE_URL", url)
+    base_url, records = plugin.discover_models()
+    embedding = next((name for name, caps in records if "embedding" in caps), None)
+    if embedding is None:
+        pytest.skip("server advertises no embedding model")
+    vector = plugin.RapidEmbedding(embedding, base_url).embed("A pelican")
+    assert vector and all(isinstance(value, float) for value in vector)

llm_rapid_mlx-0.1.0/tests/test_plugin.py

@@ -0,0 +1,338 @@
+import json
+
+import llm
+import pytest
+from click.testing import CliRunner
+from llm.cli import cli
+
+import llm_rapid_mlx as plugin
+
+
+def completion(text="Hi", tool=None):
+    message = {"role": "assistant", "content": text}
+    if tool:
+        message["tool_calls"] = [tool]
+    return {
+        "id": "reply",
+        "object": "chat.completion",
+        "model": "chat",
+        "created": 0,
+        "choices": [
+            {
+                "index": 0,
+                "message": message,
+                "finish_reason": "tool_calls" if tool else "stop",
+            }
+        ],
+        "usage": {"prompt_tokens": 3, "completion_tokens": 2, "total_tokens": 5},
+    }
+
+
+def stream(deltas, finish="stop"):
+    return [
+        {
+            "id": "reply",
+            "object": "chat.completion.chunk",
+            "model": "chat",
+            "created": 0,
+            "choices": [{"index": 0, "delta": delta, "finish_reason": None}],
+        }
+        for delta in deltas
+    ] + [
+        {
+            "id": "reply",
+            "object": "chat.completion.chunk",
+            "model": "chat",
+            "created": 0,
+            "choices": [{"index": 0, "delta": {}, "finish_reason": finish}],
+        },
+        {
+            "id": "reply",
+            "object": "chat.completion.chunk",
+            "model": "chat",
+            "created": 0,
+            "choices": [],
+            "usage": {"prompt_tokens": 3, "completion_tokens": 2, "total_tokens": 5},
+        },
+    ]
+
+
+def test_discovery_registration_and_cli(server):
+    server.catalogue += [
+        {"id": "sounds", "capabilities": ["audio.speech"]},
+        {"id": "legacy-embed-in-name"},
+        {"id": "vision", "capabilities": ["text", "vision"]},
+        {"id": "chat", "capabilities": ["text"]},
+        {"id": "invalid", "capabilities": "embedding"},
+    ]
+    registered = []
+    plugin.register_models(lambda *models: registered.append(models))
+    assert [pair[0].model_id for pair in registered] == [
+        "rapid-mlx/chat",
+        "rapid-mlx/legacy-embed-in-name",
+        "rapid-mlx/vision",
+    ]
+    assert isinstance(registered[0][1], plugin.RapidAsyncChat)
+    assert registered[0][0].supports_tools
+    assert not registered[1][0].supports_tools
+    assert "image/png" in registered[2][0].attachment_types
+    assert "application/pdf" not in registered[2][0].attachment_types
+    embeddings = []
+    plugin.register_embedding_models(embeddings.append)
+    assert [model.model_id for model in embeddings] == ["rapid-mlx/embed"]
+    result = CliRunner().invoke(cli, ["models", "-q", "rapid-mlx"])
+    assert result.exit_code == 0, result.output
+    assert "rapid-mlx/chat" in result.output
+    assert "rapid-mlx/embed" not in result.output
+    assert server.requests[0][0] == "/v1/models"
+    assert server.requests[0][1]["authorization"] == "Bearer not-needed"
+
+
+def test_no_embedding_without_advertisement(server):
+    server.catalogue = [{"id": "embedding-looking-name", "capabilities": ["text"]}]
+    registered = []
+    plugin.register_embedding_models(registered.append)
+    assert registered == []
+
+
+@pytest.mark.parametrize("status", [401, 500])
+def test_discovery_failure_is_nonfatal_and_diagnostic(server, status):
+    server.status = status
+    registered = []
+    plugin.register_models(lambda *models: registered.extend(models))
+    plugin.register_embedding_models(registered.append)
+    assert registered == []
+    result = CliRunner().invoke(cli, ["rapid-mlx", "models"])
+    assert result.exit_code == 1
+    assert f"HTTP {status}" in result.output
+    assert "never-send-this" not in result.output
+
+
+def test_unreachable_and_malformed_discovery(monkeypatch, server):
+    monkeypatch.setenv("RAPID_MLX_BASE_URL", "http://127.0.0.1:0")
+    assert plugin._catalogue() == (None, [])
+    monkeypatch.setenv("RAPID_MLX_BASE_URL", server.url)
+    server.catalogue = None
+    assert plugin._catalogue() == (None, [])
+
+
+@pytest.mark.parametrize(
+    "value,expected",
+    [
+        ("http://localhost:8000", "http://localhost:8000/v1"),
+        ("http://localhost:8000/v1/", "http://localhost:8000/v1"),
+        ("https://example.org/proxy", "https://example.org/proxy/v1"),
+    ],
+)
+def test_url_normalization(value, expected):
+    assert plugin.normalize_base_url(value) == expected
+
+
+@pytest.mark.parametrize(
+    "value",
+    [
+        "ftp://host",
+        "http://",
+        "http://u:p@host",
+        "http://host?key=x",
+        "http://host#x",
+        "http://host:no",
+    ],
+)
+def test_invalid_urls(value):
+    with pytest.raises(ValueError):
+        plugin.normalize_base_url(value)
+
+
+def test_key_and_endpoint_precedence(server, monkeypatch):
+    user_dir = llm.user_dir()
+    user_dir.mkdir(parents=True, exist_ok=True)
+    (user_dir / "keys.json").write_text(
+        json.dumps({"rapid-mlx": "stored", "rapid-mlx-base-url": "http://stored:8000"})
+    )
+    monkeypatch.setenv("RAPID_MLX_API_KEY", "environment")
+    assert plugin.resolve_key() == "stored"
+    assert plugin.resolve_key("explicit") == "explicit"
+    assert plugin.configured_base_url() == server.url
+    monkeypatch.delenv("RAPID_MLX_BASE_URL")
+    assert plugin.configured_base_url() == "http://stored:8000/v1"
+    (user_dir / "keys.json").unlink()
+    assert plugin.resolve_key() == "environment"
+    assert plugin.configured_base_url() == plugin.DEFAULT_BASE_URL
+    model = plugin.RapidChat("chat", server.url)
+    model.key = "python"
+    assert model.get_key("explicit") == "python"
+
+
+def test_stream_conversation_system_and_usage(server):
+    server.responses = [
+        stream([{"content": "H"}, {"content": "i"}]),
+        completion("pelican"),
+    ]
+    model = plugin.RapidChat("chat")
+    conversation = model.conversation()
+    first = conversation.prompt("Remember pelican", system="Be brief")
+    assert list(first) == ["H", "i"]
+    assert first.usage().input == 3
+    assert conversation.prompt("What word?", stream=False).text() == "pelican"
+    first_body = server.requests[0][2]
+    assert first_body["stream"] is True
+    assert first_body["model"] == "chat"
+    assert first_body["stream_options"] == {"include_usage": True}
+    assert server.requests[1][2]["messages"] == [
+        {"role": "system", "content": "Be brief"},
+        {"role": "user", "content": "Remember pelican"},
+        {"role": "assistant", "content": "Hi"},
+        {"role": "user", "content": "What word?"},
+    ]
+
+
+def test_schema_endpoint_option_and_key(server):
+    server.responses = [completion('{"greeting":"hi"}')]
+    model = plugin.RapidChat("chat", "http://127.0.0.1:1/v1")
+    schema = {"type": "object", "properties": {"greeting": {"type": "string"}}}
+    assert json.loads(
+        model.prompt(
+            "Hi",
+            schema=schema,
+            stream=False,
+            base_url=server.url,
+            temperature=0.2,
+            key="secret",
+        ).text()
+    ) == {"greeting": "hi"}
+    path, headers, body = server.requests[0]
+    assert path == "/v1/chat/completions"
+    assert headers["authorization"] == "Bearer secret"
+    assert body["response_format"]["json_schema"]["schema"] == schema
+    assert body["temperature"] == 0.2
+    assert "base_url" not in body
+    assert model.api_base == "http://127.0.0.1:1/v1"
+
+
+@pytest.mark.parametrize("streaming", [False, True])
+def test_tool_loop(server, streaming):
+    def add(a: int, b: int) -> int:
+        "Add two numbers."
+        return a + b
+
+    call = {
+        "id": "call1",
+        "type": "function",
+        "function": {"name": "add", "arguments": '{"a":17,"b":25}'},
+    }
+    if streaming:
+        server.responses = [
+            stream(
+                [
+                    {
+                        "tool_calls": [
+                            {
+                                "index": 0,
+                                "id": "call1",
+                                "type": "function",
+                                "function": {"name": "add", "arguments": '{"a":17,'},
+                            }
+                        ]
+                    },
+                    {
+                        "tool_calls": [
+                            {"index": 0, "function": {"arguments": '"b":25}'}}
+                        ]
+                    },
+                ],
+                "tool_calls",
+            ),
+            stream([{"content": "42"}]),
+        ]
+    else:
+        server.responses = [completion(None, call), completion("42")]
+    model = plugin.RapidChat("chat", capabilities=["text", "tools"])
+    chain = model.chain("Add 17 and 25", tools=[add], stream=streaming)
+    assert chain.text() == "42"
+    body = server.requests[1][2]
+    assert body["messages"][-1] == {
+        "role": "tool",
+        "tool_call_id": "call1",
+        "content": "42",
+    }
+    assert body["messages"][-2]["tool_calls"][0]["id"] == "call1"
+    assert server.requests[0][2]["tools"][0]["function"]["name"] == "add"
+
+
+@pytest.mark.parametrize("streaming", [False, True])
+async def test_async(server, streaming):
+    server.responses = [
+        stream([{"content": "Hello"}]) if streaming else completion("Hello")
+    ]
+    model = plugin.RapidAsyncChat("chat")
+    assert await model.prompt("Hi", stream=streaming).text() == "Hello"
+
+
+def test_embeddings_order_and_auth(server, monkeypatch):
+    monkeypatch.setenv("RAPID_MLX_API_KEY", "embed-key")
+    server.responses = [
+        {
+            "object": "list",
+            "model": "embed",
+            "data": [
+                {"object": "embedding", "index": 1, "embedding": [3, 4]},
+                {"object": "embedding", "index": 0, "embedding": [1, 2]},
+            ],
+            "usage": {"prompt_tokens": 2, "total_tokens": 2},
+        }
+    ]
+    model = plugin.RapidEmbedding("embed")
+    assert list(model.embed_multi(iter(["one", "two"]))) == [[1.0, 2.0], [3.0, 4.0]]
+    path, headers, body = server.requests[0]
+    assert path == "/v1/embeddings"
+    assert headers["authorization"] == "Bearer embed-key"
+    assert body == {
+        "model": "embed",
+        "input": ["one", "two"],
+        "encoding_format": "float",
+    }
+    with pytest.raises(ValueError, match="text strings"):
+        list(model.embed_batch([b"binary"]))
+
+
+def test_invalid_embedding_indexes(server):
+    server.responses = [
+        {
+            "object": "list",
+            "model": "embed",
+            "data": [
+                {"object": "embedding", "index": 1, "embedding": [1, 2]},
+            ],
+        }
+    ]
+    with pytest.raises(llm.ModelError, match="indexes"):
+        plugin.RapidEmbedding("embed").embed("one")
+
+
+@pytest.mark.parametrize("kind", ["chat", "async", "embedding"])
+async def test_api_errors_are_actionable_without_echoing_secrets(server, kind):
+    server.post_status = 401
+    with pytest.raises(llm.ModelError, match="HTTP 401") as error:
+        if kind == "chat":
+            plugin.RapidChat("chat").prompt(
+                "secret-prompt", key="secret-key", stream=False
+            ).text()
+        elif kind == "async":
+            await (
+                plugin.RapidAsyncChat("chat")
+                .prompt("secret-prompt", key="secret-key", stream=False)
+                .text()
+            )
+        else:
+            plugin.RapidEmbedding("embed").embed("secret-prompt", key="secret-key")
+    assert "secret" not in str(error.value)
+    assert len(server.requests) == 1  # no hidden SDK retries
+
+
+def test_discovery_uses_stored_authentication(server):
+    (llm.user_dir() / "keys.json").write_text(
+        json.dumps({"rapid-mlx": "stored-secret"})
+    )
+    plugin.discover_models()
+    assert server.requests[0][1]["authorization"] == "Bearer stored-secret"