mlxturbo 0.1.0__tar.gz

mlxturbo-0.1.0/.gitignore

@@ -0,0 +1,10 @@
+.venv/
+__pycache__/
+*.pyc
+.DS_Store
+tools/isa/applegpu/
+data/
+models/
+tools/bridge/*.dylib
+tools/isa/build/
+bench/results/

mlxturbo-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 akakeishin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mlxturbo-0.1.0/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.5
+Name: mlxturbo
+Version: 0.1.0
+Summary: A fast MLX inference engine and OpenAI/Anthropic-compatible server for Apple Silicon
+Project-URL: Homepage, https://github.com/akakeishin/mlxturbo
+Project-URL: Repository, https://github.com/akakeishin/mlxturbo
+Author-email: akakeishin <mizururi_22@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Environment :: GPU
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: MacOS :: MacOS X
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.13
+Requires-Dist: fastapi>=0.141.1
+Requires-Dist: huggingface-hub>=1.28.0
+Requires-Dist: mlx-lm<0.32,>=0.31.3
+Requires-Dist: mlx<0.33,>=0.32.2
+Requires-Dist: uvicorn>=0.52.4
+Description-Content-Type: text/markdown
+
+# mlxturbo
+
+English | [日本語](README.md)
+
+A local inference engine for Apple Silicon (MLX), plus an HTTP server compatible with the OpenAI, Anthropic, and Responses APIs.
+
+**Only two specific architectures are fast.** Every other model runs at the same speed as plain [mlx-lm](https://github.com/ml-explore/mlx-lm) (see the compatibility table below). This is not a "general-purpose acceleration runtime."
+
+If you're coming from Ollama or LM Studio, the first difference you'll hit: **one process holds exactly one model.** Switching models means restarting the process (hosting multiple models at once from a single server is not supported). See "Constraints" below for details.
+
+## What this is
+
+- An engine that loads a model into one long-lived process and speeds up decode with speculative decoding (self-speculation: the model's own MTP head plus context-based suffix lookup)
+- An HTTP server (`mlxturbo-serve`) that exposes that engine over both the OpenAI-compatible API (`/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/models`) and the Anthropic-compatible API (`/v1/messages`)
+- Unsupported models still work (fallback), but in that case you get plain, unspeculated mlx-lm speed
+
+## Compatibility table (the honest version)
+
+| Route | Target | Speculation | Measured speedup (vs. mlx-lm, see reproduction commands below) |
+|---|---|---|---|
+| `flash_spec` | Qwen3.8-Flash-Next (`qwen4_exp` architecture) + MTP sidecar | Yes (MTP depth 1 + fused hyper-connections kernel) | ~1.25x-1.39x (depends on task type, see below) |
+| `spec` | Models satisfying the `qwen3_5` contract (e.g. the Qwen3.8-27B family) | Yes (MTP chain + suffix-lookup hybrid) | 1.3x-2.2x (depends on prompt content, see below) |
+| `fallback` | Everything else | No | 1.0x (same speed as plain mlx-lm) |
+
+You can see which route was selected in the startup log and in `GET /health`'s `runner` / `fallback_reason` fields. If you want to guard against silently falling back when speculation should be working, start with `--require-runner flash_spec` (or `spec`): the server refuses to start at all if the conditions aren't met. Details in [`docs/SERVER.md`](docs/SERVER.md).
+
+## Conditions for speculation to kick in
+
+- `flash_spec` requires the model's `model_type` to be `qwen4_exp`, and MTP weights to be found (explicitly via `--mtp`, bundled in the main shards, or auto-discovered as a sidecar). If MTP isn't found, `flash_spec` does not degrade to a non-speculative mode — the route simply doesn't come up at all. Measurements are in [`docs/MTP-FLASH.md`](docs/MTP-FLASH.md)
+- `spec` requires the model to match the `qwen3_5` contract (the layer structure and attention shape `SpecEngine` expects). If it doesn't match, the server falls back to `fallback`
+- On both routes, `temp=0` (greedy) runs under exact-match verification against the true output distribution, and `temp>0` uses rejection sampling that keeps the distribution exactly identical too. However, sampling parameters that alter the distribution — `top_p<1.0` or `repetition_penalty!=1.0` — get a 400. The speculative block-verification assumes exact sampling from the target distribution, so distorting that distribution with sampling parameters breaks the verification's premise. This restriction does not apply to the `fallback` route
+
+## Installation
+
+Requires an Apple Silicon Mac (macOS, with MLX/Metal available).
+
+```
+git clone <this repository>
+cd mlxturbo
+uv sync
+```
+
+## Getting a model
+
+- `fallback` / `spec` routes: point directly at a normal mlx-lm-compatible checkpoint (a Hugging Face repo ID or a local path)
+- `flash_spec` route (Qwen3.8-Flash-Next): requires converting from the original checkpoint and extracting MTP. See [`docs/MTP-FLASH.md`](docs/MTP-FLASH.md) and `mlxturbo/convert_flash.py --help` (subcommands `estimate` / `extract-mtp` / `convert`). The `qwen4_exp` architecture isn't in mlx-lm upstream, but importing `mlxturbo` resolves it automatically — nothing is written to your mlx-lm package
+
+## Running it
+
+Interactive CLI (`spec` route, aimed at the 27B family):
+
+```
+uv run mlxturbo --model <path-or-repo-id> --prompt "hello"
+```
+
+HTTP server:
+
+```
+uv run mlxturbo-serve --model <path-or-repo-id> --served-model-name mymodel --port 8000
+```
+
+Connection methods, the full option list, API key auth, and connection examples for opencode / Codex CLI / Claude Code / Chatbox are all in [`docs/SERVER.md`](docs/SERVER.md).
+
+## Constraints
+
+- **One process, one model.** A model is loaded exactly once at startup and stays resident; switching models means starting a different process
+- **Requests are processed serially.** Continuous batching is not implemented; each request serializes to one generation at a time. If multiple clients connect at once, later requests wait for the earlier one to finish (once the queue exceeds `--max-queue`, new requests get a 503)
+- **`spec` / `flash_spec` routes return 400 for non-identity sampling parameters.** As noted above, this is because the speculative block-verification assumes exact distribution matching
+- **Token logprobs are not supported.** The response's `logprobs` field is always `null` or an empty array
+- For the full constraint list (the scope of determinism, context-length guards, etc.), see the "Constraints" section of [`docs/SERVER.md`](docs/SERVER.md)
+
+## Measured numbers and how to reproduce them
+
+Raw result JSON files and the `docs/` write-ups linked below live in the GitHub repository, not in the PyPI package (the wheel/sdist ship only the `mlxturbo` library) — follow the links from [github.com/akakeishin/mlxturbo](https://github.com/akakeishin/mlxturbo) if you installed via `pip`.
+
+All of the following were measured on an M3 Max 128GB / macOS 26.4 / mlx 0.32.2. Numbers shift across hardware generations (see "judgments that expire when the hardware generation changes" in [`docs/research/ROOFLINE-2026-08-26.md`](docs/research/ROOFLINE-2026-08-26.md)).
+
+### `spec` route (Qwen3.8-27B-4bit, greedy, 512 tok)
+
+| Condition | decode tok/s | vs. mlx-lm |
+|---|---|---|
+| Plain mlx-lm (fallback-equivalent) | 21-23 | 1.0x |
+| Self-speculation, sustained hard content (code) | 31.9 | 1.49x |
+| Self-speculation, sustained hard content (prose) | 28.3 | 1.32x |
+
+Reproduce: `uv run mlxturbo --model <model> --prompt "<prompt>"` (it prints `decode tok/s` automatically after generation). Comparison against plain mlx-lm: `uv run python bench/baseline.py <model-id>`.
+
+### `flash_spec` route (Qwen3.8-Flash-Next, v-l recipe + MTP 4bit, greedy, 48 tok)
+
+| Prompt | Acceptance rate | Greedy tok/s | Speculative tok/s | Speedup |
+|---|---|---|---|---|
+| Prose (English 0.720 ± 0.046 / Japanese 0.682 ± 0.047) | ~0.70 | ~1.39x |
+| Code | 0.564 ± 0.068 | ~1.25x |
+
+Acceptance rate also varies by task type (code / prose / step-by-step instructions, etc). Details and reproduction commands are in the "tools" section of [`docs/MTP-FLASH.md`](docs/MTP-FLASH.md) (`tools/spec_flash_bench.py` and others). See also [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) for exactly which result file each number in this README traces back to, including one row whose source file we could not track down and one table whose own source document later flags it as not statistically significant.
+
+## Other documentation
+
+- [`docs/README.md`](docs/README.md) — index of all documentation
+- [`docs/SERVER.md`](docs/SERVER.md) — server startup, options, connection methods, constraints
+- [`docs/OPERATIONS.md`](docs/OPERATIONS.md) — running a public instance: reverse proxy, `/health` monitoring, log reading, launchd
+- [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) — which reproduction command and result file backs each published number
+- [`docs/MTP-FLASH.md`](docs/MTP-FLASH.md) — design of Flash-Next's MTP speculative decoding
+- [`docs/BACKLOG.md`](docs/BACKLOG.md) — things worth doing that haven't been started yet
+- [`docs/RELEASE.md`](docs/RELEASE.md) — what to do before publishing
+
+Everything under `docs/` besides this README's translation is Japanese-only for now.

mlxturbo-0.1.0/README.en.md

@@ -0,0 +1,107 @@
+# mlxturbo
+
+English | [日本語](README.md)
+
+A local inference engine for Apple Silicon (MLX), plus an HTTP server compatible with the OpenAI, Anthropic, and Responses APIs.
+
+**Only two specific architectures are fast.** Every other model runs at the same speed as plain [mlx-lm](https://github.com/ml-explore/mlx-lm) (see the compatibility table below). This is not a "general-purpose acceleration runtime."
+
+If you're coming from Ollama or LM Studio, the first difference you'll hit: **one process holds exactly one model.** Switching models means restarting the process (hosting multiple models at once from a single server is not supported). See "Constraints" below for details.
+
+## What this is
+
+- An engine that loads a model into one long-lived process and speeds up decode with speculative decoding (self-speculation: the model's own MTP head plus context-based suffix lookup)
+- An HTTP server (`mlxturbo-serve`) that exposes that engine over both the OpenAI-compatible API (`/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/models`) and the Anthropic-compatible API (`/v1/messages`)
+- Unsupported models still work (fallback), but in that case you get plain, unspeculated mlx-lm speed
+
+## Compatibility table (the honest version)
+
+| Route | Target | Speculation | Measured speedup (vs. mlx-lm, see reproduction commands below) |
+|---|---|---|---|
+| `flash_spec` | Qwen3.8-Flash-Next (`qwen4_exp` architecture) + MTP sidecar | Yes (MTP depth 1 + fused hyper-connections kernel) | ~1.25x-1.39x (depends on task type, see below) |
+| `spec` | Models satisfying the `qwen3_5` contract (e.g. the Qwen3.8-27B family) | Yes (MTP chain + suffix-lookup hybrid) | 1.3x-2.2x (depends on prompt content, see below) |
+| `fallback` | Everything else | No | 1.0x (same speed as plain mlx-lm) |
+
+You can see which route was selected in the startup log and in `GET /health`'s `runner` / `fallback_reason` fields. If you want to guard against silently falling back when speculation should be working, start with `--require-runner flash_spec` (or `spec`): the server refuses to start at all if the conditions aren't met. Details in [`docs/SERVER.md`](docs/SERVER.md).
+
+## Conditions for speculation to kick in
+
+- `flash_spec` requires the model's `model_type` to be `qwen4_exp`, and MTP weights to be found (explicitly via `--mtp`, bundled in the main shards, or auto-discovered as a sidecar). If MTP isn't found, `flash_spec` does not degrade to a non-speculative mode — the route simply doesn't come up at all. Measurements are in [`docs/MTP-FLASH.md`](docs/MTP-FLASH.md)
+- `spec` requires the model to match the `qwen3_5` contract (the layer structure and attention shape `SpecEngine` expects). If it doesn't match, the server falls back to `fallback`
+- On both routes, `temp=0` (greedy) runs under exact-match verification against the true output distribution, and `temp>0` uses rejection sampling that keeps the distribution exactly identical too. However, sampling parameters that alter the distribution — `top_p<1.0` or `repetition_penalty!=1.0` — get a 400. The speculative block-verification assumes exact sampling from the target distribution, so distorting that distribution with sampling parameters breaks the verification's premise. This restriction does not apply to the `fallback` route
+
+## Installation
+
+Requires an Apple Silicon Mac (macOS, with MLX/Metal available).
+
+```
+git clone <this repository>
+cd mlxturbo
+uv sync
+```
+
+## Getting a model
+
+- `fallback` / `spec` routes: point directly at a normal mlx-lm-compatible checkpoint (a Hugging Face repo ID or a local path)
+- `flash_spec` route (Qwen3.8-Flash-Next): requires converting from the original checkpoint and extracting MTP. See [`docs/MTP-FLASH.md`](docs/MTP-FLASH.md) and `mlxturbo/convert_flash.py --help` (subcommands `estimate` / `extract-mtp` / `convert`). The `qwen4_exp` architecture isn't in mlx-lm upstream, but importing `mlxturbo` resolves it automatically — nothing is written to your mlx-lm package
+
+## Running it
+
+Interactive CLI (`spec` route, aimed at the 27B family):
+
+```
+uv run mlxturbo --model <path-or-repo-id> --prompt "hello"
+```
+
+HTTP server:
+
+```
+uv run mlxturbo-serve --model <path-or-repo-id> --served-model-name mymodel --port 8000
+```
+
+Connection methods, the full option list, API key auth, and connection examples for opencode / Codex CLI / Claude Code / Chatbox are all in [`docs/SERVER.md`](docs/SERVER.md).
+
+## Constraints
+
+- **One process, one model.** A model is loaded exactly once at startup and stays resident; switching models means starting a different process
+- **Requests are processed serially.** Continuous batching is not implemented; each request serializes to one generation at a time. If multiple clients connect at once, later requests wait for the earlier one to finish (once the queue exceeds `--max-queue`, new requests get a 503)
+- **`spec` / `flash_spec` routes return 400 for non-identity sampling parameters.** As noted above, this is because the speculative block-verification assumes exact distribution matching
+- **Token logprobs are not supported.** The response's `logprobs` field is always `null` or an empty array
+- For the full constraint list (the scope of determinism, context-length guards, etc.), see the "Constraints" section of [`docs/SERVER.md`](docs/SERVER.md)
+
+## Measured numbers and how to reproduce them
+
+Raw result JSON files and the `docs/` write-ups linked below live in the GitHub repository, not in the PyPI package (the wheel/sdist ship only the `mlxturbo` library) — follow the links from [github.com/akakeishin/mlxturbo](https://github.com/akakeishin/mlxturbo) if you installed via `pip`.
+
+All of the following were measured on an M3 Max 128GB / macOS 26.4 / mlx 0.32.2. Numbers shift across hardware generations (see "judgments that expire when the hardware generation changes" in [`docs/research/ROOFLINE-2026-08-26.md`](docs/research/ROOFLINE-2026-08-26.md)).
+
+### `spec` route (Qwen3.8-27B-4bit, greedy, 512 tok)
+
+| Condition | decode tok/s | vs. mlx-lm |
+|---|---|---|
+| Plain mlx-lm (fallback-equivalent) | 21-23 | 1.0x |
+| Self-speculation, sustained hard content (code) | 31.9 | 1.49x |
+| Self-speculation, sustained hard content (prose) | 28.3 | 1.32x |
+
+Reproduce: `uv run mlxturbo --model <model> --prompt "<prompt>"` (it prints `decode tok/s` automatically after generation). Comparison against plain mlx-lm: `uv run python bench/baseline.py <model-id>`.
+
+### `flash_spec` route (Qwen3.8-Flash-Next, v-l recipe + MTP 4bit, greedy, 48 tok)
+
+| Prompt | Acceptance rate | Greedy tok/s | Speculative tok/s | Speedup |
+|---|---|---|---|---|
+| Prose (English 0.720 ± 0.046 / Japanese 0.682 ± 0.047) | ~0.70 | ~1.39x |
+| Code | 0.564 ± 0.068 | ~1.25x |
+
+Acceptance rate also varies by task type (code / prose / step-by-step instructions, etc). Details and reproduction commands are in the "tools" section of [`docs/MTP-FLASH.md`](docs/MTP-FLASH.md) (`tools/spec_flash_bench.py` and others). See also [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) for exactly which result file each number in this README traces back to, including one row whose source file we could not track down and one table whose own source document later flags it as not statistically significant.
+
+## Other documentation
+
+- [`docs/README.md`](docs/README.md) — index of all documentation
+- [`docs/SERVER.md`](docs/SERVER.md) — server startup, options, connection methods, constraints
+- [`docs/OPERATIONS.md`](docs/OPERATIONS.md) — running a public instance: reverse proxy, `/health` monitoring, log reading, launchd
+- [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) — which reproduction command and result file backs each published number
+- [`docs/MTP-FLASH.md`](docs/MTP-FLASH.md) — design of Flash-Next's MTP speculative decoding
+- [`docs/BACKLOG.md`](docs/BACKLOG.md) — things worth doing that haven't been started yet
+- [`docs/RELEASE.md`](docs/RELEASE.md) — what to do before publishing
+
+Everything under `docs/` besides this README's translation is Japanese-only for now.

mlxturbo-0.1.0/README.md

@@ -0,0 +1,111 @@
+# mlxturbo
+
+[English](README.en.md) | 日本語
+
+Apple Silicon (MLX) 上のローカル推論エンジンと、OpenAI / Anthropic / Responses 互換の HTTP サーバー。
+
+**速いのは特定の2アーキテクチャだけ**で、それ以外のモデルは素の [mlx-lm](https://github.com/ml-explore/mlx-lm) と同速で動く汎用ランタイムです（後述の対応表を参照）。「汎用高速化ランタイム」ではありません。
+
+Ollama や LM Studio から来た人が最初に踏む差分: **1 プロセスにつき 1 モデルしかロードしない。** モデルの切り替えはプロセスの再起動が要る（複数モデルの同時ホストは非対応）。詳しくは後述の「制約」を参照。
+
+## これは何か
+
+- モデルを 1 プロセスにロードして常駐させ、投機デコード（自己投機: 本体のモデル自身が出す MTP ヘッド + 文脈 suffix-lookup）で decode を高速化するエンジン
+- 上記エンジンを OpenAI 互換 (`/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/models`) と Anthropic 互換 (`/v1/messages`) の両方で叩ける HTTP サーバー (`mlxturbo-serve`)
+- 対応していないモデルでも動く（フォールバック）が、その場合は投機なしの素の mlx-lm と同速
+
+## 対応表（正直に）
+
+| 経路 | 対象 | 投機 | 実測倍率（mlx-lm 比、後述の再現コマンド参照） |
+|---|---|---|---|
+| `flash_spec` | Qwen3.8-Flash-Next (`qwen4_exp` アーキテクチャ) + MTP サイドカー | あり（MTP 深さ1 + hyper-connections 融合カーネル） | 約 1.25x〜1.39x（課題の種類に依存、後述） |
+| `spec` | `qwen3_5` 契約を満たすモデル（例: Qwen3.8-27B 系） | あり（MTP チェイン + suffix-lookup 混成） | 1.3x〜2.2x（プロンプト内容依存、後述） |
+| `fallback` | 上記以外の全モデル | なし | 1.0x（素の mlx-lm と同速） |
+
+どの経路が選ばれたかは起動ログと `GET /health` の `runner` / `fallback_reason` で確認できる。「投機が効くはずなのに黙って fallback に落ちている」を防ぎたい場合は `--require-runner flash_spec`（または `spec`）を付けて起動すると、条件を満たさないときに起動自体が失敗する（詳細は [`docs/SERVER.md`](docs/SERVER.md)）。
+
+## 投機が効く条件
+
+- `flash_spec` が動くには、モデルの `model_type` が `qwen4_exp` で、かつ MTP 重みが見つかっている必要がある（`--mtp` で明示指定するか、本体シャードに同梱されているか、サイドカーとして自動発見される）。MTP が見つからないときは投機無しの `flash_spec` にはならず、経路自体が成立しない。実測は [`docs/MTP-FLASH.md`](docs/MTP-FLASH.md) にある
+- `spec` が動くには、モデルが `qwen3_5` の contract（`SpecEngine` が要求する層構成や attention の形）に合っている必要がある。合っていなければ `fallback` に落ちる
+- 両経路とも、`temp=0`（greedy）は出力分布と厳密に一致する完全一致検証で動き、`temp>0` も棄却サンプリングで分布は厳密に同一のまま。ただし `top_p<1.0` や `repetition_penalty≠1.0` のように分布を変えるサンプリングパラメータを渡すと 400 になる。投機側のブロック検証が「対象分布からの厳密なサンプリング一致」を前提にしているため、サンプリングパラメータで分布そのものを歪められると検証が成り立たなくなる。この制限は `fallback` 経路には無い
+
+## インストール
+
+Apple Silicon Mac（macOS、MLX/Metal が使える環境）が前提。
+
+```
+git clone <このリポジトリ>
+cd mlxturbo
+uv sync
+```
+
+## モデルの入手
+
+- `fallback` / `spec` 経路: 通常の mlx-lm 互換チェックポイント（Hugging Face repo ID かローカルパス）をそのまま指定できる
+- `flash_spec` 経路（Qwen3.8-Flash-Next）: 元 checkpoint からの変換とMTP抽出が要る。手順は [`docs/MTP-FLASH.md`](docs/MTP-FLASH.md) と `mlxturbo/convert_flash.py --help`（`estimate` / `extract-mtp` / `convert` の各サブコマンド）を参照。`qwen4_exp` アーキテクチャは mlx-lm 本体に無いが、`mlxturbo` を import した時点で自動的に解決される（利用者の mlx-lm パッケージへは何も書き込まない）
+
+## 起動
+
+対話 CLI（`spec` 経路、27B 系向け）:
+
+```
+uv run mlxturbo --model <path-or-repo-id> --prompt "こんにちは"
+```
+
+HTTP サーバー:
+
+```
+uv run mlxturbo-serve --model <path-or-repo-id> --served-model-name mymodel --port 8000
+```
+
+接続方法・オプション一覧・API キー認証・opencode / Codex CLI / Claude Code / Chatbox からの接続例は [`docs/SERVER.md`](docs/SERVER.md) に詳しくまとめてある。
+
+## 制約
+
+- **1 プロセス 1 モデル。** モデルは起動時に 1 回だけロードして常駐する設計で、複数モデルの切り替えは別プロセスを立てる必要がある
+- **リクエストは直列処理。** 継続バッチング（continuous batching）は未実装で、1 リクエスト = 1 生成に直列化される。複数クライアントを同時に繋ぐと、後着のリクエストは先着の生成が終わるまで待たされる（待ち行列上限 `--max-queue` を超えると 503）
+- **spec / flash_spec 経路は非恒等値のサンプリングパラメータを 400 にする。** 理由は上述のとおり、投機のブロック検証が厳密な分布一致を前提にしているため
+- **token logprobs は未対応。** レスポンスの `logprobs` は常に `null` または空配列
+- 詳しい制約一覧（決定性の範囲、文脈長ガードなど）は [`docs/SERVER.md`](docs/SERVER.md) の「制約」節を参照
+
+## 実測値と再現コマンド
+
+以下でリンクしている結果 JSON や `docs/` の文書は GitHub リポジトリ側にあり、PyPI パッケージ（wheel/sdist）には含まれない（配布物に入るのは `mlxturbo` ライブラリ本体のみ）。`pip` で入れた場合は [github.com/akakeishin/mlxturbo](https://github.com/akakeishin/mlxturbo) 側からリンクを辿ること。
+
+以下はすべて M3 Max 128GB / macOS 26.4 / mlx 0.32.2 での実測。ハードウェア世代が変わると数字は変わる（詳しくは [`docs/research/ROOFLINE-2026-08-26.md`](docs/research/ROOFLINE-2026-08-26.md) の「ハード世代が変わると失効する判定」を参照）。
+
+### `spec` 経路（Qwen3.8-27B-4bit、greedy、512 tok）
+
+| 条件 | decode tok/s | 対 mlx-lm |
+|---|---|---|
+| mlx-lm 素（フォールバック相当） | 21〜23 | 1.0x |
+| 自己投機・難しい内容持続（code） | 31.9 | 1.49x |
+| 自己投機・難しい内容持続（prose） | 28.3 | 1.32x |
+
+再現: `uv run mlxturbo --model <model> --prompt "<prompt>"`（生成の後に `decode tok/s` が自動で出力される）。mlx-lm 素との対照は `uv run python bench/baseline.py <model-id>`。
+
+### `flash_spec` 経路（Qwen3.8-Flash-Next、v-l レシピ + MTP 4bit、greedy）
+
+受理率は 160 トークン × 10 課題で測った（反復 200 以上）。
+
+| 課題 | 受理率 | 倍率 |
+|---|---|---|
+| 散文（英語 0.720 ± 0.046 / 日本語 0.682 ± 0.047） | 約 0.70 | 約 1.39x |
+| コード | 0.564 ± 0.068 | 約 1.25x |
+
+投機なしは 31.14 ms/token。**効くかどうかは言語ではなく課題の種類で決まり、コードが最も低い**（信頼区間が重ならない）。どの課題でも投機は効く。
+
+短い試行（48 トークン、反復 30 前後）では言語差があるように見えるが、標準誤差が ±0.09 あり有意ではない。**受理率は反復 300 以上で見ること。**経緯は [`docs/MTP-FLASH.md`](docs/MTP-FLASH.md) を参照。
+
+再現: `tools/spec_flash_accept.py`（受理率）と `tools/spec_flash_bench.py`（速度）。コマンドは [`docs/MTP-FLASH.md`](docs/MTP-FLASH.md) の「道具」節。
+
+## その他のドキュメント
+
+- [`docs/README.md`](docs/README.md) — ドキュメント全体の索引
+- [`docs/SERVER.md`](docs/SERVER.md) — サーバーの起動・オプション・接続方法・制約
+- [`docs/OPERATIONS.md`](docs/OPERATIONS.md) — 公開インスタンスの運用（リバースプロキシ・`/health` 監視・ログの読み方・launchd 常駐）
+- [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) — この README の実測値がどのコマンド・どの結果 JSON から来ているか
+- [`docs/MTP-FLASH.md`](docs/MTP-FLASH.md) — Flash-Next の MTP 投機デコード設計
+- [`docs/BACKLOG.md`](docs/BACKLOG.md) — やりたいが未着手のもの
+- [`docs/RELEASE.md`](docs/RELEASE.md) — 公開時にやること

mlxturbo-0.1.0/mlxturbo/__init__.py

@@ -0,0 +1,10 @@
+from . import _arch_registry
+from .kernels.dispatch import enable
+
+# Make qwen4_exp (Flash-Next) resolvable without writing into the user's mlx_lm.
+# Importing any mlxturbo submodule always goes through here (a package __init__
+# runs before its submodules). See the module docstring of _arch_registry.py for
+# the details.
+_arch_registry.install()
+
+__all__ = ["enable"]

mlxturbo-0.1.0/mlxturbo/_arch_registry.py

@@ -0,0 +1,99 @@
+"""Resolve qwen4_exp (Qwen3.8-Flash-Next) without writing into the user's mlx_lm.
+
+## Background
+
+The Flash-Next model class is not part of mlx-lm proper (mlx-lm PR #1788 is
+still unmerged). This package carries a vendored copy as
+`mlxturbo/_vendor/qwen4_exp.py`, so that it ships inside the wheel and
+resolves from an installed environment, not just a repo checkout.
+
+mlx-lm has exactly one path for resolving a model class,
+
+    importlib.import_module(f"mlx_lm.models.{model_type}")
+
+in `mlx_lm/utils.py:_get_classes()`, and mlx-lm itself has no plugin-style
+registration mechanism. Previously we resolved this by having
+`convert_flash.py install-arch` physically copy the vendored file into the
+user's site-packages (`<mlx_lm install dir>/models/qwen4_exp.py`), but that
+rewrites the user's mlx_lm package, with the side effects that it disappears
+on `uv sync` or on an mlx-lm update, and that any other project sharing the
+same mlx_lm gets dragged along with it (review comment: merely trying out
+mlxturbo breaks your environment).
+
+## What this does instead
+
+It inserts into `sys.meta_path` a minimal finder that redirects only the fully
+qualified name `mlx_lm.models.qwen4_exp` to the vendored file. It takes no part
+in any other import (find_spec returns None and defers to the default finders
+as-is), so it affects nothing else, mlx_lm included.
+
+Nothing is written into site-packages. The `mlx_lm.models` package itself stays
+as it is (the normal installation), and relative imports inside the vendored
+file such as `from .base import ...` ride naturally on the usual
+`mlx_lm.models.base` resolution, because the finder gives the spec the correct
+`__name__` / `__package__` (= "mlx_lm.models").
+
+## Why a meta_path finder rather than registering directly in `sys.modules`
+
+Simply placing `sys.modules["mlx_lm.models.qwen4_exp"] = <already-executed
+module>` ahead of time when mlxturbo is imported would also work for every
+resolution path — `import mlx_lm.models.qwen4_exp as Q` /
+`from mlx_lm.models import qwen4_exp` / `importlib.import_module(...)` — because
+CPython's import-as / from-import fall back to `sys.modules[fullname]` when the
+attribute lookup fails. But
+`NGRAM_ON_DISK = os.environ.get("FASTMLX_NGRAM_DISK") == "1"` at the top of the
+vendored file is evaluated exactly once when the module executes, and the
+convention on the calling side (cli.py / server.py / convert_flash.py / things
+under tools) is to set that environment variable immediately before qwen4_exp
+is actually loaded (it is not yet set at the time mlxturbo is imported).
+Registering into `sys.modules` up front would execute the module body before
+that convention gets its chance, freezing it in a state that ignored
+`FASTMLX_NGRAM_DISK`. A meta_path finder lets us defer execution until the
+moment the import of `mlx_lm.models.qwen4_exp` is actually requested (= after
+the caller has set the environment variable), so the convention stays intact.
+"""
+
+from __future__ import annotations
+
+import importlib.abc
+import importlib.util
+import sys
+from pathlib import Path
+
+MODULE_NAME = "mlx_lm.models.qwen4_exp"
+# Resolved relative to this module's own installed location (not the repo
+# root), so this works the same from a `pip install`-ed wheel as it does from
+# a repo checkout — nothing here depends on `tools/` existing on disk.
+VENDOR_PATH = Path(__file__).resolve().parent / "_vendor" / "qwen4_exp.py"
+
+
+class _FlashNextArchFinder(importlib.abc.MetaPathFinder):
+    """Redirect only `mlx_lm.models.qwen4_exp` to the vendored file."""
+
+    def find_spec(self, fullname, path, target=None):
+        if fullname != MODULE_NAME:
+            return None
+        if not VENDOR_PATH.exists():
+            # Should not happen for a normal install (the vendor file ships
+            # inside the mlxturbo package itself), but silently defer to the
+            # default finders rather than raise — that would also break
+            # imports of models that never use qwen4_exp.
+            return None
+        return importlib.util.spec_from_file_location(fullname, VENDOR_PATH)
+
+
+def install() -> None:
+    """Make qwen4_exp resolvable within the `mlx_lm.models` namespace.
+
+    Idempotent (calling it twice still installs only one finder). It is called
+    from `mlxturbo`'s `__init__.py`, so it is always in effect as soon as
+    `mlxturbo` is imported (regardless of whether qwen4_exp actually gets used).
+    """
+
+    if any(isinstance(f, _FlashNextArchFinder) for f in sys.meta_path):
+        return
+    # Even if an old install-arch copy is still sitting in site-packages, this
+    # one takes precedence (the vendored file is the source of truth). Since it
+    # takes no part in any other name, inserting it at the front has no effect
+    # on other imports.
+    sys.meta_path.insert(0, _FlashNextArchFinder())