dsh-plugin-local-infer 0.0.0-stage → 0.1.1

package/LICENSE

@@ -0,0 +1,7 @@
+Copyright Derek "cybtachyon" Reese
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

package/README.md

@@ -1,3 +1,97 @@
-# Temporary Holding Version
+# dsh-plugin-local-infer
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+[中文 (Chinese)](docs/zh/README.md)
+
+> For those who want more performance out of their specific hardware than what Unsloth gives out of the box.
+
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://github.com/cybtachyon/dsh-plugin-local-infer/blob/main/LICENSE)
+[![DSH 0.2.1-alpha.3](https://img.shields.io/badge/DSH-0.2.1--alpha.3-blue)](https://www.npmjs.com/package/@deepseek-ai/dsh)
+[![Type: DSH Plugin](https://img.shields.io/badge/Type-DSH%20Plugin-8A2BE2)](https://github.com/topics/dsh-plugin)
+
+A DeepSeek Harness plugin that manages local inference engines:
+
+- llama.cpp
+- NInfer
+- Hyper-Qwen (vLLM)
+- Strata
+- ...with more coming soon.
+
+![Plugin Screenshot](docs/screenshot-20261014.png)
+
+Pick a model and an engine, tune the parameters, watch a live **VRAM Est.** meter, see **hardware warnings** the moment a value (or a parameter itself) is unsafe for the detected hardware, then the plugin manages **build, deploy, and health-check** for each engine and **automatically register it as a DSH model** once it is confirmed working. No hand-editing of finicky yaml or launchers.
+
+## Install
+
+Command line:
+
+```bash
+dsh plugin --profile web add https://github.com/cybtachyon/dsh-plugin-local-infer
+```
+
+Web UI: click **Plugins** → **Add Plugin** → paste `https://github.com/cybtachyon/dsh-plugin-local-infer`.
+
+## Using the plugin
+
+The **Engine settings** card mounts on the Plugins page (older hosts: the Settings page).
+
+- **Pick an engine and model**, then tune the parameters. Each field renders by its type: checkboxes, dropdowns, sliders, text, directory pickers, and a chat-template combo (known templates plus free text).
+- **Watch the VRAM Est. meter.** It prices weights, KV cache, overhead, and workspace against your card as you adjust values, and shows the verdict (`fits` / `tight` / `over`) and the largest context that fits.
+- **Read the warnings.** Red banners and inline field warnings fire when a parameter value, or the presence of a parameter, is unsafe for your detected hardware.
+- **Satisfy the system dependencies.** The panel lists each missing host dependency (docker, CUDA toolkit, ffmpeg, hf CLI, ...) with its purpose, a copy/paste install string for your OS, and an optional one-click **Run** (sudo). The **Get** button stays disabled until every dependency reports satisfied.
+- **Launch / Stop.** `Launch` clones or pulls the engine, downloads the model, starts it, and polls its health endpoint. `Stop` shuts it down.
+- **Register / Remove.** `Register` publishes the confirmed-working engine to DSH as a model (optionally as the agent default model) through the settings service, so the change is revision-tracked and undoable. `Remove` pulls the entry back.
+
+All values persist to `~/.dsh/local-infer/config.json`; the browser never sees a value it did not store.
+
+## Supported Platforms
+
+### Operating systems
+
+| OS                           | State     | Notes                                                                                                               |
+|------------------------------|-----------|---------------------------------------------------------------------------------------------------------------------|
+| Linux (Debian/Ubuntu, `apt`) | Supported | Install blocks mirror the working NInfer launcher: the CUDA keyring sequence plus the compile toolchain.            |
+| Linux (Arch, `pacman`)       | Supported | `pacman` blocks plus the `/opt/cuda` export lines.                                                                  |
+| Linux (Fedora, `dnf`)        | Untested  | Command set present, flagged untested on a live host.                                                               |
+| macOS (`brew`)               | Partial   | CPU/docker path (llama.cpp). No NVIDIA Container Toolkit on macOS, so the CUDA engines' docker path is unavailable. |
+| Windows (WSL2)               | Untested  | WSL2 is detected and the dflash2 pin-memory rule fires; the command set is flagged untested.                        |
+| Windows                      | Partial   | Code paths are only partially written; further development needed.                                                  |  
+
+### Engines
+
+| Engine            | Docker        | Local build        | State                                                                        |
+|-------------------|---------------|--------------------|------------------------------------------------------------------------------|
+| NInfer            | Yes           | Yes (cmake + CUDA) | The reference engine. Measured VRAM models for the Qwen 27B/35B artifacts.   |
+| Hyper-Qwen (vLLM) | Yes (compose) | No                 | `.env`-driven docker-compose profiles.                                       |
+| vLLM              | No            | No                 | Work-in-progress.                                                            |
+| llama.cpp         | Yes           | Yes                | Specified, not verified until the binary is available.                       |
+| Strata            | Yes           | No                 | Placeholder engine; its parameter surface is empty until the flag set lands. |
+
+Runtime requirements: Node `^22.19.0 || >=24.0.0`, pnpm 12.11.2 (pinned by `mise.toml`), and the `@deepseek-ai/cordis >= 4.0.1` peer dependency.
+
+## Hardware guide
+
+What each common machine gets you in practice. Capability impressions are sourced from [r/LocalLLaMA](https://www.reddit.com/r/LocalLLaMA/) user threads. "Shared" means unified memory (the model and the OS share the pool).
+
+| Hardware                        | Memory          | What you get                                                                                                                                                                  |
+| ------------------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| NVIDIA RTX 5090                 | 32 GB VRAM      | Reliably fits model weights up to 23 GB with 240k+ context and vision. Handles complex coding and vision tasks, but needs special tooling for very long or complex tasks.     |
+| NVIDIA RTX 4090                 | 24 GB VRAM      | The community workhorse. 30B-class models fit at Q4 and drive coding agents; 70B needs slow CPU offload. Best speed-per-dollar single card.                                   |
+| NVIDIA RTX 3090                 | 24 GB VRAM      | The used-card value king. Same 24 GB as a 4090 but an older generation: 32B at Q4 with modest context, no FP8. A single card is limiting; the common advice is to pair two.   |
+| NVIDIA RTX PRO 6000             | 96 GB VRAM      | Near-frontier dense models (70B–80B) fit fully in memory with long context, and 32B models can even be fine-tuned (LoRA, full FT). The prosumer ceiling.                      |
+| NVIDIA RTX PRO 5500             | 84 GB VRAM      | The rack-oriented 84 GB tier: 70B-class models with headroom, close to the PRO 6000 at a lower price point.                                                                   |
+| NVIDIA Tesla V100               | 32 GB VRAM      | Cheap used HBM2 server card. 30B-class at Q4/Q8 at respectable speed (bandwidth is what matters for inference); no FP8, power-hungry. A budget homelab pick.                  |
+| NVIDIA Tesla V100               | 16 GB VRAM      | The budget entry. 13B–14B at Q8, or 30B with offload. NVLink pairs double the capacity. Slow by modern standards, surprisingly solid for inference.                           |
+| NVIDIA RTX 5090 Laptop          | 24 GB VRAM      | The mobile 4090. 32B at Q4 just fits; realistically ~17B at Q4 once you subtract context.                                                                                     |
+| NVIDIA RTX 4090 Laptop          | 16 GB VRAM      | The GPU is overkill for LLMs and the VRAM is underkill: 8B–14B at Q4/Q5 with small context.                                                                                   |
+| NVIDIA GB10 (DGX Spark)         | 128 GB shared   | The desktop AI dev box. A 35B MoE runs at ~100 tok/s; 120B+ MoE fits but is slow. Capacity over speed.                                                                        |
+| NVIDIA GB10 (DGX Spark)         | 64 GB shared    | Half the pool: 32B–35B MoE with context; 70B is tight.                                                                                                                        |
+| NVIDIA RTX N1X                  | 128 GB shared   | The next-generation GB10-class laptop: DGX Spark-class capacity in a laptop. Very new, little real-world data.                                                                |
+| NVIDIA RTX N1X                  | 64 GB shared    | The 64 GB tier: 32B-class with context.                                                                                                                                       |
+| Apple M4 Ultra                  | 256 GB shared   | The capacity king. 200B+ MoE and 70B at 128K context fit; expect ~5 tok/s on 70B. Slow per token, unmatched capacity.                                                         |
+| Apple M4 Ultra                  | 128 GB shared   | 70B–120B-class fits with room; ~5 tok/s. Great for capacity, slow for speed.                                                                                                  |
+| AMD Radeon AI PRO R9700         | 32 GB VRAM      | AMD's 32 GB Vulkan card. 32B at Q8 around $1,300, good value; the `radv` driver is not a conformant Vulkan implementation.                                                    |
+| AMD Ryzen AI Max+ PRO 495       | 160 GB shared   | The UMA monster. 120B+ MoE fits and coding agents run well (Qwen3-Coder-class models are tuned for UMA); $7k+ systems, slower than a discrete GPU.                            |
+| 2x NVIDIA RTX 3090              | 48 GB VRAM      | The community's "minimum for a usable local LLM": 70B at Q4/Q5.                                                                                                               |
+| 2x NVIDIA RTX 4090              | 48 GB VRAM      | 70B at Q4 with context; exllama2 quantizations run fast.                                                                                                                      |
+| 2x NVIDIA RTX 5090              | 64 GB VRAM      | 120B MoE (GPT-OSS-120B) with 128k context at ~10 tok/s with offload.                                                                                                          |
+| 2x NVIDIA Tesla V100 (NVLink)   | 64 GB VRAM      | The budget 64 GB: 70B at lower quant with NVLink bandwidth; older arch, no FP8.                                                                                               |

package/cordis.patch.yml

@@ -0,0 +1,42 @@
+# The dsh-plugin-local-infer bundle patch: mounts the whole local-inference
+# capability as ONE layer over an existing dsh profile.
+#
+# Row order carries no load semantics — activation is service-availability
+# driven — so the seam may appear before or after the providers that fill it.
+# The user's own profile patch layer can address any row below by `id` to
+# retune it (last write per row wins).
+
+- insert:
+    # Client carrier: mounts the bare package name so the client-module scan
+    # keys the `dsh.client` declaration on it and loads dsh/client.js (the
+    # browser "Engine settings" card) into the boot graph. The scan only maps
+    # a bare package-root entry to its `dsh.client` field, never a subpath, so
+    # this row is what makes the client half reachable. The host module is a
+    # no-op; the capability lives in the four subpath rows below.
+    - id: local-infer
+      name: 'dsh-plugin-local-infer'
+
+    # Service Definition: publishes ctx.localInfer, the engine/preset/guideline
+    # registries, and the orchestration entry points (estimate, guidelines,
+    # deps, installString, runInstall, launch, registerModel). Carries no store
+    # of its own.
+    - id: local-infer-service
+      name: 'dsh-plugin-local-infer/service'
+
+    # Service Providers: the concrete engine adapters (ninfer, hyperqwen,
+    # llamacpp, strata), the NInfer VramModel, the built-in presets, the
+    # built-in guidelines, and the per-OS SystemDep install command data. Each
+    # registers itself with ctx.localInfer.
+    - id: local-infer-providers
+      name: 'dsh-plugin-local-infer/providers'
+
+    # Consumer: the model-facing local_infer_* tools (estimate, launch, status,
+    # register), their system-prompt section, and their per-tool timeout budgets.
+    - id: local-infer-tool
+      name: 'dsh-plugin-local-infer/tool'
+
+    # Web UI: the parameter form (form-type aware), the VRAM meter, the
+    # red-warning banner, the engine lifecycle controls, and the register-as-
+    # model confirmation.
+    - id: local-infer-ui
+      name: 'dsh-plugin-local-infer/ui'

package/docs/zh/README.md

@@ -0,0 +1,97 @@
+# dsh-plugin-local-infer
+
+> 本中文翻译由 Qwen3.8-27B 生成。欢迎提交 Pull Request 改进翻译。
+
+> 面向那些希望从特定硬件中获得比 Unsloth 开箱即用更高性能的用户。
+
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://github.com/cybtachyon/dsh-plugin-local-infer/blob/main/LICENSE)
+[![DSH 0.2.1-alpha.2](https://img.shields.io/badge/DSH-0.2.1--alpha.2-blue)](https://www.npmjs.com/package/@deepseek-ai/dsh)
+[![Type: DSH Plugin](https://img.shields.io/badge/Type-DSH%20Plugin-8A2BE2)](https://github.com/topics/dsh-plugin)
+
+一个 DeepSeek Harness 插件，用于管理本地推理引擎：
+
+- llama.cpp
+- NInfer
+- Hyper-Qwen (vLLM)
+- Strata
+- ...更多即将推出。
+
+![插件截图](../screenshot-20261014.png)
+
+选择一个模型和引擎，调节参数，实时观察 **VRAM 估算** 仪表，当某个值（或参数本身）对检测到的硬件不安全时立即看到 **硬件警告**，随后该插件会管理每个引擎的 **构建、部署和健康检查**，并在确认其正常工作后 **自动将其注册为 DSH 模型**。无需手动编辑易出错的 yaml 或启动器。
+
+## 安装
+
+命令行：
+
+```bash
+dsh plugin --profile web add https://github.com/cybtachyon/dsh-plugin-local-infer
+```
+
+Web UI：点击 **插件** → **添加插件** → 粘贴 `https://github.com/cybtachyon/dsh-plugin-local-infer`。
+
+## 使用插件
+
+**引擎设置** 卡片挂载在插件页面（旧版本主机：设置页面）。
+
+- **选择引擎和模型**，然后调节参数。每个字段根据其类型渲染：复选框、下拉框、滑块、文本、目录选择器，以及一个聊天模板组合框（已知模板加上自由文本）。
+- **观察 VRAM 估算仪表。** 当你调整值时，它根据你的显卡对权重、KV 缓存、开销和工作区进行估算，并显示判定结果（`fits` / `tight` / `over`）以及能容纳的最大上下文。
+- **阅读警告。** 当某个参数值，或某个参数的存在，对你的检测到的硬件不安全时，红色横幅和内联字段警告会触发。
+- **满足系统依赖项。** 面板列出每个缺失的主机依赖项（docker、CUDA toolkit、ffmpeg、hf CLI、...），并附有其用途、针对你操作系统的复制粘贴安装字符串，以及可选的一键 **运行**（sudo）。**获取** 按钮在所有依赖项报告满足之前保持禁用状态。
+- **启动 / 停止。** `启动` 克隆或拉取引擎、下载模型、启动它，并轮询其健康端点。`停止` 将其关闭。
+- **注册 / 移除。** `注册` 通过设置服务将确认工作正常的引擎作为模型发布到 DSH（可选地作为代理默认模型），因此该变更可被修订跟踪并可撤销。`移除` 将该条目拉回。
+
+所有值都持久化到 `~/.dsh/local-infer/config.json`；浏览器永远不会看到它未存储的值。
+
+## 支持的平台
+
+### 操作系统
+
+| 操作系统                     | 状态     | 说明                                                                                                      |
+| ---------------------------- | -------- | --------------------------------------------------------------------------------------------------------- |
+| Linux (Debian/Ubuntu, `apt`) | 已支持   | 安装块镜像可用的 NInfer 启动器：CUDA keyring 序列加上编译工具链。                                         |
+| Linux (Arch, `pacman`)       | 已支持   | `pacman` 块加上 `/opt/cuda` 导出行。                                                                      |
+| Linux (Fedora, `dnf`)        | 未测试   | 命令集已存在，在实时主机上标记为未测试。                                                                  |
+| macOS (`brew`)               | 部分支持 | CPU/docker 路径（llama.cpp）。macOS 上没有 NVIDIA Container Toolkit，因此 CUDA 引擎的 docker 路径不可用。 |
+| Windows (WSL2)               | 未测试   | WSL2 被检测，dflash2 固定内存规则触发；命令集标记为未测试。                                               |
+| Windows                      | 部分支持 | 代码路径仅部分编写；需要进一步开发。                                                                      |
+
+### 引擎
+
+| 引擎              | Docker       | 本地构建          | 状态                                               |
+| ----------------- | ------------ | ----------------- | -------------------------------------------------- |
+| NInfer            | 是           | 是 (cmake + CUDA) | 参考引擎。针对 Qwen 27B/35B 制品的实测 VRAM 模型。 |
+| Hyper-Qwen (vLLM) | 是 (compose) | 否                | 由 `.env` 驱动的 docker-compose 配置。             |
+| vLLM              | 否           | 否                | 进行中。                                           |
+| llama.cpp         | 是           | 是                | 已指定，在二进制可用之前未验证。                   |
+| Strata            | 是           | 否                | 占位符引擎；在其标志集落地之前，其参数表面为空。   |
+
+运行时要求：Node `^22.19.0 || >=24.0.0`、pnpm 12.11.2（由 `mise.toml` 固定），以及 `@deepseek-ai/cordis >= 4.0.1` 同级依赖。
+
+## 硬件指南
+
+每台常见机器在实际中能为你带来什么。能力印象来源于 [r/LocalLLaMA](https://www.reddit.com/r/LocalLLaMA/) 用户帖子。"共享" 指统一内存（模型和操作系统共享内存池）。
+
+| 硬件                          | 内存        | 你能得到的                                                                                                                           |
+| ----------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------ |
+| NVIDIA RTX 5090               | 32 GB VRAM  | 可靠地容纳最多 23 GB 的模型权重，支持 240k+ 上下文和视觉。能处理复杂的编码和视觉任务，但需要特殊工具来处理非常长或复杂的任务。       |
+| NVIDIA RTX 4090               | 24 GB VRAM  | 社区主力。Q4 量化下可容纳 30B 级模型并驱动编码代理；70B 需要缓慢的 CPU 卸载。性价比最高的单卡。                                      |
+| NVIDIA RTX 3090               | 24 GB VRAM  | 二手卡的性价比之王。与 4090 相同的 24 GB，但属于较旧的世代：Q4 量化下 32B 及适度的上下文，不支持 FP8。单卡有限制；常见建议是配两张。 |
+| NVIDIA RTX PRO 6000           | 96 GB VRAM  | 接近前沿的稠密模型（70B–80B）可完全装入内存并支持长上下文，32B 模型甚至可以进行微调（LoRA、全量 FT）。消费级上限。                   |
+| NVIDIA RTX PRO 5500           | 84 GB VRAM  | 面向机架的 84 GB 级别：70B 级模型有余量，价格更低且接近 PRO 6000。                                                                   |
+| NVIDIA Tesla V100             | 32 GB VRAM  | 便宜的二手 HBM2 服务器卡。Q4/Q8 量化下 30B 级模型，速度可观（带宽对推理而言才是关键）；不支持 FP8，功耗高。预算型家庭实验室之选。    |
+| NVIDIA Tesla V100             | 16 GB VRAM  | 预算入门。Q8 量化下 13B–14B，或带卸载的 30B。NVLink 双卡翻倍容量。以现代标准来看较慢，但对推理而言出乎意料地扎实。                   |
+| NVIDIA RTX 5090 Laptop        | 24 GB VRAM  | 移动端 4090。Q4 量化下 32B 勉强容纳；扣除上下文后现实上约为 Q4 的 ~17B。                                                             |
+| NVIDIA RTX 4090 Laptop        | 16 GB VRAM  | GPU 对 LLM 而言过剩，而 VRAM 却不足：Q4/Q5 量化下 8B–14B，上下文较小。                                                               |
+| NVIDIA GB10 (DGX Spark)       | 128 GB 共享 | 桌面 AI 开发机。35B MoE 以 ~100 tok/s 运行；120B+ MoE 可容纳但较慢。容量优先于速度。                                                 |
+| NVIDIA GB10 (DGX Spark)       | 64 GB 共享  | 一半内存池：带上下文的 32B–35B MoE；70B 较为紧张。                                                                                   |
+| NVIDIA RTX N1X                | 128 GB 共享 | 下一代 GB10 级笔记本：笔记本中的 DGX Spark 级容量。非常新，几乎没有真实世界数据。                                                    |
+| NVIDIA RTX N1X                | 64 GB 共享  | 64 GB 级别：带上下文的 32B 级。                                                                                                      |
+| Apple M4 Ultra                | 256 GB 共享 | 容量之王。200B+ MoE 和 128K 上下文的 70B 均可容纳；70B 预期 ~5 tok/s。单 token 较慢，容量无出其右。                                  |
+| Apple M4 Ultra                | 128 GB 共享 | 70B–120B 级可容纳且有余量；~5 tok/s。容量出色，速度较慢。                                                                            |
+| AMD Radeon AI PRO R9700       | 32 GB VRAM  | AMD 的 32 GB Vulkan 卡。Q8 量化下 32B 约 $1,300，性价比较高；`radv` 驱动并非符合规范的 Vulkan 实现。                                 |
+| AMD Ryzen AI Max+ PRO 495     | 160 GB 共享 | UMA 怪兽。120B+ MoE 可容纳，编码代理运行良好（Qwen3-Coder 级模型针对 UMA 调优）；$7k+ 系统，速度慢于独立 GPU。                       |
+| 2x NVIDIA RTX 3090            | 48 GB VRAM  | 社区的"可用作本地 LLM 的最低配置"：Q4/Q5 量化下的 70B。                                                                              |
+| 2x NVIDIA RTX 4090            | 48 GB VRAM  | 带上下文的 Q4 量化 70B；exllama2 量化运行快速。                                                                                      |
+| 2x NVIDIA RTX 5090            | 64 GB VRAM  | 带卸载的 ~10 tok/s，支持 128k 上下文的 120B MoE（GPT-OSS-120B）。                                                                    |
+| 2x NVIDIA Tesla V100 (NVLink) | 64 GB VRAM  | 预算型 64 GB：NVLink 带宽下低量化的 70B；较旧的架构，不支持 FP8。                                                                    |