@adelie/gpu-model-serving 0.0.0-stage → 0.1.0

package/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

package/README.md

@@ -1,3 +1,31 @@
-# Temporary Holding Version
+# GPU Model Serving
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+An Adelie plugin that teaches an agent to **stand up an OpenAI-compatible LLM endpoint on a GPU box** — a rented instance or your own machine — and to survive the parts that normally cost a day.
+
+## What it installs
+
+One skill, `gpu-model-serving`, plus its references:
+
+| File | Contents |
+| --- | --- |
+| `skills/gpu-model-serving/SKILL.md` | The job in order: instance and disk layout, weights, environment, `serve.sh`, detached start, verification, hand-over — and the toolchain rules that actually bite. |
+| `reference/sglang-serve.md` | Environment setup, a complete working `serve.sh` for a 27B 4-bit model + draft model on one 32GB card, flag-by-flag notes, memory arithmetic, startup and restart behaviour. |
+| `reference/pitfalls.md` | The failure list: symptom → cause → fix, 14 entries, from NVFP4 checkpoint loading and CCCL header checks to JIT build OOM. |
+| `reference/verify.md` | The five checks that decide whether the service is done — health, model list, tool call, stream, prefix cache — plus client wiring and the restart checklist. |
+| `reference/gpu-cloud-console.md` | Operating the rented instance: console actions, the single exposed port, the public port that changes per boot, driving the web terminal, and prepaid billing that stops the instance at zero. |
+
+## When it fires
+
+The skill is loaded when a task involves self-hosting a model: deploying or moving an inference service, choosing quantization or serving flags, diagnosing a server that will not start, or wiring an agent framework to a local endpoint. It is written for the single-GPU, 24–48GB case that most agent workloads land on, and scales by arithmetic rather than by hope.
+
+## Install
+
+From the Adelie **Plugins** page, import from npm: `@adelie/gpu-model-serving`. The same string works from a URL — `https://www.npmjs.com/package/@adelie/gpu-model-serving` — or point the importer at the registry tarball. The plugin installs under the name `gpu-model-serving`.
+
+## Provenance
+
+Distilled from a real deployment: **算家云 RTX 5090 ×1 (32GB, sm_120), SGLang 0.5.21, a 27B NVFP4 model with a DFlash2 draft model**, serving tool-calling agent traffic with prefix caching on, verified end to end (health, `/v1/models`, `tool_calls`, streaming SSE, `#cached-token`, speculative accept length). Every entry in `pitfalls.md` is a failure that happened on the way there.
+
+## License
+
+Apache-2.0.

package/icon.svg

@@ -0,0 +1,7 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round">
+  <rect x="3" y="4" width="18" height="7" rx="2" />
+  <rect x="3" y="13" width="18" height="7" rx="2" />
+  <path d="M7 7.5h.01M7 16.5h.01" />
+  <path d="M11 7.5h5M11 16.5h3" />
+  <path d="M17.5 16.5l1.2-1.6 1.3 3 1.2-1.4" />
+</svg>

package/package.json

@@ -1,6 +1,32 @@
 {
   "name": "@adelie/gpu-model-serving",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.1.0",
+  "description": "Stand up an OpenAI-compatible LLM endpoint on a rented or owned GPU box: pick the instance, fetch and quantize weights, serve with SGLang — speculative decoding, prefix caching and tool calling for agent workloads — then verify it and expose the port.",
+  "license": "Apache-2.0",
+  "keywords": [
+    "gpu",
+    "llm",
+    "sglang",
+    "inference",
+    "serving",
+    "openai-compatible",
+    "quantization",
+    "speculative-decoding"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/lmliheng/Adelie.git",
+    "directory": "plugins/gpu-model-serving"
+  },
+  "files": [
+    "plugin.json",
+    "icon.svg",
+    "skills",
+    "hooks",
+    "LICENSE",
+    "README.md"
+  ],
+  "publishConfig": {
+    "access": "public"
+  }
+}

package/plugin.json

@@ -0,0 +1,8 @@
+{
+  "description": "Stand up an OpenAI-compatible LLM endpoint on a rented or owned GPU box: pick the instance, fetch and quantize weights, serve with SGLang — speculative decoding, prefix caching and tool calling for agent workloads — then verify it and expose the port. Carries the CUDA/JIT traps that cost hours the first time (CCCL header checks, MAX_JOBS OOM, flashinfer cubin version skew, NVFP4 checkpoints, mamba state cache).",
+  "description_zh": "在租用或自有的 GPU 机器上部署 OpenAI 兼容的模型服务：选实例、拉权重与量化、用 SGLang 开投机解码/前缀缓存/tool calling 服务 agent 负载，再验证并开放端口。附带第一次部署会耗掉数小时的 CUDA/JIT 坑（CCCL 头文件校验、MAX_JOBS 内存打爆、flashinfer cubin 版本错位、NVFP4 checkpoint、mamba 状态缓存）。",
+  "short_description": "Self-host an LLM endpoint on a GPU box with SGLang.",
+  "short_description_zh": "在 GPU 机器上用 SGLang 自部署模型接口。",
+  "version": "2026.10.30.1",
+  "category": "software-development"
+}

package/skills/gpu-model-serving/SKILL.md

@@ -0,0 +1,52 @@
+---
+name: gpu-model-serving
+description: Stand up an OpenAI-compatible LLM endpoint on a rented or owned GPU box — choose the instance and disk layout, fetch and quantize the weights, serve with SGLang (speculative decoding, prefix caching, tool calling), then verify it and expose the port. Use when a task needs a self-hosted model service, especially one GPU of 24–48GB serving an agent workload.
+---
+
+# GPU Model Serving
+
+You are standing up a **self-hosted, OpenAI-compatible LLM endpoint** on a GPU machine — rented by the hour from a cloud, or a box you already have. The deliverable is a URL like `http://<host>:<port>/v1` plus a model name, good enough that an agent framework can point at it and call tools.
+
+Two things make this harder than it looks, and both are scheduling, not modelling: the **build toolchain** (CUDA headers, nvcc, JIT kernel compilation) and the **memory budget** of a single card. Read [`reference/pitfalls.md`](reference/pitfalls.md) before you start typing; every entry there is a real failure that costs 10–60 minutes when it first appears.
+
+## The shape of the job
+
+1. **Pick the machine and where the bytes live.** A single 24–48GB card runs a 27B-class model at 4-bit with room for a draft model and KV cache. Put weights, virtualenv and scripts on a **persistent data disk**, never on the small system disk — a stopped/restarted instance keeps the data disk and loses nothing else, and re-downloading 20–30GB is the slowest thing in the whole job.
+2. **Get the weights.** Prefer an already-quantized checkpoint (NVFP4 / AWQ / FP8) over quantizing yourself. Check the file set: an index plus its shards, the tokenizer, and a chat template that advertises `tools`. **Delete or move aside zero-byte leftovers** — a stale empty `*.index.json` or shard makes the loader crash on read.
+3. **Build the environment.** One virtualenv with the torch build matching the driver, `sglang[all]`, and a CUDA toolkit from pip. Pin nvcc deliberately (see *Toolchain* below).
+4. **Write a `serve.sh`** that exports the environment and ends in `exec sglang serve …`. Launch it detached: `setsid nohup ./serve.sh > serve.log 2>&1 < /dev/null &`. The first start compiles kernels and can take 20–60 minutes; every later start reuses the cache and takes minutes.
+5. **Verify before wiring clients** — `/health`, `/v1/models`, a tool call, a streamed response, a repeated prefix. [`reference/verify.md`](reference/verify.md) has the exact commands and what each answer should look like.
+6. **Expose the port and hand over** the base URL, the model name, the concurrency limit, and the cost per hour.
+
+## Toolchain — the part that actually bites
+
+- The pip-installed CUDA packages must be on the environment's path: `CUDA_HOME`/`CUDA_PATH` at `<venv>/…/nvidia/cu13` (with its `bin` on `PATH`), `LD_LIBRARY_PATH` and `LIBRARY_PATH` at `<venv>/…/nvidia/cu13/lib`. Without this, `deep_gemm` asserts `cuda_home is not None` and sgl_kernel fails to load `libnvrtc`.
+- pip ships `libcudart.so.13` but no `libcudart.so`; linkers ask for the latter. Symlink `libcudart.so`, `libcublas.so`, `libcublasLt.so` in that lib directory.
+- **CCCL refuses a nvcc that does not exactly match the CUDA headers** (`#error "CUDA compiler and CUDA toolkit headers are incompatible"`). If you cannot make them match, set `#define CCCL_DISABLE_CTK_COMPATIBILITY_CHECK 1` at the top of every `cuda_toolkit.h` in the venv — there is more than one copy, and the second one only shows up when SGLang's own JIT kernels compile. Find them all first: `grep -rln "headers are incompatible" <venv>/lib/python3.*/site-packages`.
+- **Cap the compile parallelism.** `ninja` defaults to one compiler per core plus two; a 27B CUTLASS/FP4 template can take gigabytes each, and a 60GB container gets OOM-killed mid-build (`ninja` exits 137, the log only says `Received sigquit from a child process`). `export MAX_JOBS=3` in `serve.sh` fixes it; the JIT cache keeps whatever already compiled.
+- flashinfer's cubin package lags the runtime, and the version check is fatal. `FLASHINFER_DISABLE_VERSION_CHECK=1`; the venv's flashinfer cache (`~/.cache/sglang/.cache/flashinfer`) is worth keeping between restarts.
+
+## Serving choices for an agent workload
+
+- **Speculative decoding** is a large win on a single card: a small draft model plus `--speculative-algorithm` (e.g. `DFLASH`, `EAGLE`) and 4–8 draft tokens. Expect an accept length of ~3–4; watch `accept len` in the decode log line.
+- **Prefix caching** matters more than raw throughput when your client sends the same system prompt and tool schemas every turn. A repeat request should show `#cached-token:` close to the prompt length in the log.
+- **Tool calling and reasoning** are parser settings, not model settings: `--tool-call-parser` (e.g. `qwen3_coder`), `--reasoning-parser` (e.g. `qwen3`). With a reasoning parser the chain of thought arrives in `reasoning_content` and the answer in `content`; without it your client sees raw think-tags.
+- **Concurrency is usually not what you set.** Hybrid (mamba/GDN) models cap `max_running_requests` by the state cache and say so in the log: `max_running_requests is capped to 1 by the mamba state cache (max_mamba_cache_size=8, 5 state slots per request)`. Either raise `--max-mamba-cache-size` and accept a smaller KV pool, or tell the user it serialises.
+- Budget the card explicitly: `--mem-fraction-static` sizes the KV pool off what is left; `--kv-cache-dtype fp8_e4m3`, `--context-length` and `--cuda-graph-max-bs-decode` trade context, concurrency and speed against each other. Read the two lines `KV Cache is allocated … #tokens:` and `available_gpu_mem` from the log and report them as the real limit.
+
+The complete working example — every flag, every export, and the memory arithmetic — is in [`reference/sglang-serve.md`](reference/sglang-serve.md).
+
+## Rented GPU clouds
+
+The console, the port mapping and the billing are where a rented instance differs from your own box, and where an agent usually needs help: the container listens on one fixed port, the public port is assigned per boot, the port switch turns itself off when the instance stops, and a depleted balance stops the instance without warning. [`reference/gpu-cloud-console.md`](reference/gpu-cloud-console.md) walks through it with the 算家云 (suanjiayun) console as the concrete case, including how to drive its web terminal when SSH is password-only.
+
+## Reporting
+
+Give the user: the base URL and model name; what you verified (and what you could not); the measured concurrency and context limits with the reason; the hourly cost and the remaining balance; and the exact commands to restart the service after a reboot. If a client needs configuration, hand them the `base_url` / `model` pair and note that the placeholder API key can be any string when the server has no `--api-key`.
+
+## Reference
+
+- [`reference/pitfalls.md`](reference/pitfalls.md) — the failure list: symptom → cause → fix.
+- [`reference/sglang-serve.md`](reference/sglang-serve.md) — environment setup, the full `serve.sh`, flag-by-flag notes, memory math, restart behaviour.
+- [`reference/verify.md`](reference/verify.md) — verification commands and expected answers, prefix-cache evidence, client wiring.
+- [`reference/gpu-cloud-console.md`](reference/gpu-cloud-console.md) — renting and operating the instance: console, ports, web terminal, billing.

package/skills/gpu-model-serving/reference/gpu-cloud-console.md

@@ -0,0 +1,42 @@
+# Rented GPU clouds — console, ports, terminal, billing
+
+The concrete case is **算家云 (suanjiayun.com)**, but the shape repeats on most Chinese GPU-rental consoles: a web console for the instance, a container you reach through a browser terminal, one exposed port, a persistent data disk, and prepaid billing.
+
+## The instance
+
+- Console: `https://suanjiayun.com/container/#/instanceMine` — the instance list; each row carries 立即关机/开机, WebShell, SSH/SFTP, 文件管理, 开放端口, 更多.
+- Storage layout inside the container:
+  - `/` — small system disk (~30GB), wiped on a system reset. Never put weights here.
+  - `/root/sj-tmp` — the data disk (~50GB), **survives stop/restart and reset**. Keep models, the venv and `serve.sh` here.
+- Memory is a cgroup cap of the rented size (`cat /sys/fs/cgroup/memory.max`, e.g. 60GiB) while `free` reports the **host's** memory. Never size `MAX_JOBS` or a cache from `free`.
+- SSH: `ssh <account>@<region>.suanjiayun.com -p 2020` — **password-only**; a pubkey is refused, so a scripted shell must go through the web terminal (below) or an interactive login.
+
+## Ports — read this twice
+
+- The container must listen on **8080**; the platform forwards that one port.
+- The forwarding is configured per instance under **开放端口 → 对外开放端口** (a switch, saved immediately), and the public address is revealed by **获取地址 / 新页面访问**, e.g. `http://<region>.suanjiayun.com:51197` → your OpenAI base URL is `http://<region>.suanjiayun.com:51197/v1`.
+- **The public port is randomly assigned and can change on every boot** (observed: 51197 → 51476 across one stop/start), and **stopping the instance turns the port switch off**. So after every power-on: reopen the switch, read the new address, and update anything that hard-codes it (client configs, model registries, bookmarks).
+
+## The web terminal (WebShell)
+
+The WebShell tab is an xterm in an iframe. To drive it from a script:
+
+1. `document.querySelector('iframe').contentDocument`, then the terminal's hidden input is `.xterm-helper-textarea`.
+2. `focus()` it and dispatch a synthetic `ClipboardEvent('paste')` carrying the command as `text/plain` — synthetic `keydown` alone does not insert text.
+3. To run it, dispatch synthetic `keydown`/`keyup` for `Enter` (keyCode 13).
+4. Read output from `.xterm-rows` `innerText` after setting `.xterm-viewport.scrollTop = scrollHeight` — only the visible ~60 lines exist, so `grep`/`tail` inside the container first.
+5. If the terminal freezes, reload the iframe (`iframe.contentWindow.location.reload()`); anything started with `setsid nohup` keeps running.
+
+Long commands can lose characters or fail to submit on the first Enter — send a second bare Enter script rather than retyping.
+
+## Billing
+
+- Hourly, prepaid: an RTX 5090 ×1 instance billed **¥2.92/hour**.
+- When the balance reaches zero the platform **stops the instance automatically** ("已关机 / 已暂停计费") and the public address dies with it. A negative balance blocks starting again until it is topped up.
+- Consequences for an agent: check the balance before promising an always-on service, tell the user the hourly rate and roughly how long the remaining balance lasts, and remember that a stop also disables the port switch.
+- Because a stopped instance loses nothing on the data disk, the restart ritual is short:
+  1. top up and power on;
+  2. **开放端口 → 对外开放端口**: switch on, read the new address;
+  3. in WebShell: `cd /root/sj-tmp; setsid nohup ./serve.sh > serve.log 2>&1 < /dev/null &`;
+  4. wait ~4 minutes (warm JIT cache) and re-verify;
+  5. update every client that held the old address.

package/skills/gpu-model-serving/reference/pitfalls.md

@@ -0,0 +1,85 @@
+# Failure list — symptom → cause → fix
+
+In the order they show up. Search this file by the error string before doing anything clever.
+
+## 1. Checkpoint will not load: `assert param_data.shape == loaded_weight.shape`
+
+**Cause:** the serving build does not know this quantization layout (here: an NVFP4/ModelOpt checkpoint on sglang 0.5.15).
+**Fix:** upgrade (`sglang[all]==0.5.21` loaded it unchanged). If no version knows it, re-quantize or pick another checkpoint — do not patch the loader.
+
+## 2. `AssertionError: extra_buffer_lazy unsupported with spec`
+
+**Cause:** a lazy mamba radix-cache strategy under speculative decoding.
+**Fix:** `--mamba-radix-cache-strategy extra_buffer`.
+
+## 3. `unrecognized arguments: --mm-feature-transport`
+
+**Cause:** the flag does not exist in the installed version.
+**Fix:** upgrade, or drop it. Do not leave an unknown flag in `serve.sh` — the process exits before loading anything.
+
+## 4. `ImportError: libnvrtc.so.13: cannot open shared object file`
+
+**Cause:** the CUDA runtime inside the venv is not on the loader path.
+**Fix:** `export LD_LIBRARY_PATH=<venv>/lib/python3.*/site-packages/nvidia/cu13/lib:$LD_LIBRARY_PATH`.
+
+## 5. `deep_gemm: AssertionError: assert cuda_home is not None`
+
+**Cause:** no toolkit for the JIT to call — a container has no system CUDA.
+**Fix:** `export CUDA_HOME=$CS/nvidia/cu13`, `CUDA_PATH=$CUDA_HOME`, `PATH=$CUDA_HOME/bin:$PATH` **and** `LIBRARY_PATH=$CUDA_HOME/lib`.
+
+## 6. `flashinfer-cubin 0.6.12 != flashinfer 0.6.18` (fatal version check)
+
+**Cause:** the runtime and its prebuilt cubins are versioned separately, and the matching cubin often is not on PyPI yet.
+**Fix:** `FLASHINFER_DISABLE_VERSION_CHECK=1`.
+
+## 7. `#error "CUDA compiler and CUDA toolkit headers are incompatible, please check your include paths"`
+
+**Cause:** CCCL inside the venv requires nvcc and `cuda_runtime.h` to be the *same* minor version (here nvcc 13.2 vs headers 13.0).
+**Fix:** prepend `#define CCCL_DISABLE_CTK_COMPATIBILITY_CHECK 1` to every `cuda/std/__cccl/cuda_toolkit.h` in the venv. Verify with a full-line match — the file already contains `#ifndef CCCL_DISABLE_CTK_COMPATIBILITY_CHECK`, so a substring test always says "already fixed":
+
+```bash
+f=<venv>/…/nvidia/cu13/include/cccl/cuda/std/__cccl/cuda_toolkit.h
+grep -q '^#define CCCL_DISABLE_CTK_COMPATIBILITY_CHECK 1' "$f" || sed -i '1i #define CCCL_DISABLE_CTK_COMPATIBILITY_CHECK 1' "$f"
+```
+
+## 8. `ptxas ... Unsupported .version 9.2` / `Unsupported .target sm_120`
+
+**Cause:** you "fixed" #7 by downgrading nvcc to match the headers. The older toolchain cannot emit code for a newer card.
+**Fix:** keep nvcc at the newest version and disable the check instead (#7). Never downgrade below the card's architecture support.
+
+## 9. `ld: cannot find -lcudart`
+
+**Cause:** pip ships versioned library names only (`libcudart.so.13`, never `libcudart.so`).
+**Fix:** symlink `libcudart.so` (and `libcublas.so`, `libcublasLt.so` if the same error repeats) in the venv's `nvidia/cu13/lib`, and export `LIBRARY_PATH` to that directory.
+
+## 10. Draft/auxiliary model directory crashes the loader
+
+**Cause:** zero-byte leftovers — truncated shards, an empty `model.safetensors.index.json`, empty tokenizer files — a half-finished download that the loader trusts and then fails on.
+**Fix:** move every zero-byte file out of the directory; keep only the files the manifest actually names (e.g. `config.json` + a single `model.safetensors` for a draft model). Check with `find <dir> -size 0`.
+
+## 11. `Received sigquit from a child process. It usually means the child failed.`
+
+**Cause:** this line is a symptom, never the reason. The scheduler dies because a compile or a child process died.
+**Fix:** read upward: `grep -inE 'Killed|error:|Traceback|ninja:|CalledProcessError' serve.log | tail -20`. Expect #12 or #13.
+
+## 12. The same CCCL error, after you already fixed it — from `sgl_kernel_jit_speculative_sampling_*`
+
+**Cause:** there is more than one vendored CCCL in a venv. The flashinfer copy is patched by #7; SGLang's own JIT (speculative sampling, attention) compiles against `nvidia/cu13/include/cccl/…` and stops on the same check.
+**Fix:** apply #7 to *every* copy found by `grep -rln "headers are incompatible" <venv>/lib/python3.*/site-packages`.
+
+## 13. JIT build OOMs: `RuntimeError: Ninja build failed` with several `Killed` and exit status 137
+
+**Cause:** ninja parallelises to `nproc+2` compilers; each CUTLASS/FP4 template can take gigabytes. A rented instance's memory is a cgroup cap (check `/sys/fs/cgroup/memory.max`), even though `free` inside the container prints the *host's* total and looks enormous.
+**Fix:** `export MAX_JOBS=3` (flashinfer's `jit/cpp_ext.py` honours it) — or whatever keeps peak usage under the cap. Already-compiled kernels stay cached, so a restart continues from where it died.
+
+## 14. Only one port is reachable from outside
+
+**Cause:** the platform exposes a fixed container port (算家云: **8080**) and forwards it to a random public port; nothing else is reachable.
+**Fix:** serve on the exposed port, and do not run anything else on it. A temporary `python3 -m http.server 8080` to read logs is a useful trick **only** while the model server is down — kill it before starting SGLang, and remember the public port can change on the next boot (`gpu-cloud-console.md`).
+
+## Working habits that saved the most time
+
+- **Read the log, not the terminal.** Long output through a web terminal is painful; `grep`/`tail` on the server first, and only then look at the screen. The reason for a crash is always above the last line.
+- **Start detached** (`setsid nohup … &`) so a browser reload or a dropped SSH session cannot kill a 20-minute build.
+- **Cache is an asset.** `~/.cache/sglang` holds every JIT artifact and autotune result; a warm restart is ~4 minutes instead of ~40.
+- **Verify with the API, not the log.** "ready to roll" plus a green `/health` still does not prove tool calls, streaming or prefix caching work — see `verify.md`.

package/skills/gpu-model-serving/reference/sglang-serve.md

@@ -0,0 +1,124 @@
+# Deploying with SGLang — the working setup
+
+Everything here was run on **one RTX 5090 (32GB, sm_120), 15 vCPU, 60GB RAM**, serving a 27B 4-bit model with a draft model for speculative decoding, Python 3.10. Scale the numbers, keep the shape.
+
+## 1. Environment
+
+```bash
+apt-get install -y python3-venv || true
+cd /root/sj-tmp                       # the persistent data disk
+python3 -m venv venv && . venv/bin/activate
+pip install -U pip
+pip install "sglang[all]==0.5.21"            # 0.5.15 could not load the NVFP4 checkpoint
+pip install nvidia-cuda-nvcc==13.2.86        # pin the toolchain; see pitfalls 7 & 8
+```
+
+`torch 2.13.0+cu130` came in with `sglang[all]`. The pip CUDA packages land in the venv at `lib/python3.<x>/site-packages/nvidia/cu13/`, which is the toolkit you point everything at.
+
+Three fixes to apply once, after installing:
+
+```bash
+CS=venv/lib/python3.10/site-packages
+# (a) linker names pip does not ship
+ln -sf libcudart.so.13  $CS/nvidia/cu13/lib/libcudart.so
+ln -sf libcublas.so.13  $CS/nvidia/cu13/lib/libcublas.so 2>/dev/null || true
+ln -sf libcublasLt.so.13 $CS/nvidia/cu13/lib/libcublasLt.so 2>/dev/null || true
+# (b) find every CCCL copy that guards the toolkit-header check, then disable the check
+grep -rln "headers are incompatible" $CS | tee /tmp/cccl.txt
+while read -r f; do
+  grep -q '^#define CCCL_DISABLE_CTK_COMPATIBILITY_CHECK 1' "$f" || \
+    sed -i '1i #define CCCL_DISABLE_CTK_COMPATIBILITY_CHECK 1' "$f"
+done < /tmp/cccl.txt
+```
+
+## 2. `serve.sh`
+
+```bash
+#!/bin/bash
+cd /root/sj-tmp
+. venv/bin/activate
+
+export MAX_JOBS=3                                  # ninja parallelism -> container RAM
+export FLASHINFER_DISABLE_VERSION_CHECK=1          # vendored cubin lags the runtime
+CS=/root/sj-tmp/venv/lib/python3.10/site-packages
+export LD_LIBRARY_PATH=$CS/nvidia/cu13/lib:$LD_LIBRARY_PATH
+export LIBRARY_PATH=$CS/nvidia/cu13/lib:$LIBRARY_PATH
+export CUDA_HOME=$CS/nvidia/cu13
+export CUDA_PATH=$CUDA_HOME
+export PATH=$CUDA_HOME/bin:$PATH
+
+M=/root/sj-tmp/models/<target-model>
+D=/root/sj-tmp/models/<draft-model>
+
+exec sglang serve \
+  --model-path "$M" \
+  --served-model-name qwen3.8-27b-uncensored \
+  --quantization modelopt_fp4 \
+  --trust-remote-code \
+  --speculative-algorithm DFLASH \
+  --speculative-draft-model-path "$D" \
+  --speculative-num-draft-tokens 8 \
+  --context-length 122880 \
+  --max-mamba-cache-size 8 \
+  --mamba-radix-cache-strategy extra_buffer \
+  --mamba-ssm-dtype bfloat16 \
+  --kv-cache-dtype fp8_e4m3 \
+  --attention-backend triton \
+  --chunked-prefill-size 1024 \
+  --mm-feature-transport cpu \
+  --cuda-graph-max-bs-decode 1 \
+  --mem-fraction-static 0.86 \
+  --max-running-requests 4 \
+  --reasoning-parser qwen3 \
+  --tool-call-parser qwen3_coder \
+  --host 0.0.0.0 --port 8080
+```
+
+Start it detached, and keep it that way — the service must survive the terminal that launched it:
+
+```bash
+cd /root/sj-tmp
+setsid nohup ./serve.sh > serve.log 2>&1 < /dev/null &
+```
+
+## 3. Flags, and why each one is there
+
+| Flag | Why |
+| --- | --- |
+| `--quantization modelopt_fp4` | The checkpoint is NVFP4 (ModelOpt). Wrong value → shape assertion while loading. |
+| `--speculative-algorithm DFLASH` + `--speculative-draft-model-path` | A small draft model proposes tokens; the target verifies them. Cheapest latency win on one card. |
+| `--speculative-num-draft-tokens 8` | 4–8 is the useful range; watch `accept len` in the decode log. |
+| `--mamba-radix-cache-strategy extra_buffer` | Hybrid GDN models need this variant for prefix caching; the `*_lazy` strategies assert under speculative decoding. |
+| `--mamba-ssm-dtype bfloat16` | Halves the mamba state size — the thing that caps concurrency. |
+| `--kv-cache-dtype fp8_e4m3` | Doubles the tokens the KV pool holds at a small quality cost. |
+| `--attention-backend triton` | Avoids a flashinfer attention path on sm_120; the JIT then compiles less. |
+| `--chunked-prefill-size 1024` | Bounds the prefill burst so long prompts do not spike memory. |
+| `--mm-feature-transport cpu` | Multimodal feature transport off the GPU; only exists from 0.5.21. |
+| `--cuda-graph-max-bs-decode 1` | Capture CUDA graphs only up to this batch; more concurrency needs a bigger value and more memory. |
+| `--mem-fraction-static 0.86` | Fraction of the card the server may use. Everything else (KV pool) is sized from what remains. |
+| `--reasoning-parser qwen3` | Splits `…` into `reasoning_content`. |
+| `--tool-call-parser qwen3_coder` | Turns the model's tool-call syntax into OpenAI `tool_calls`. |
+| `--host 0.0.0.0 --port 8080` | The port the platform exposes; binding to `127.0.0.1` makes it unreachable from outside. |
+
+## 4. Memory arithmetic (read these four log lines)
+
+```
+Load weight end. elapsed=32.91 s, type=… quant=modelopt_fp4, quant_algo=NVFP4, avail mem=11.89 GB
+Mamba Cache is allocated. max_mamba_cache_size: 8, conv_state 0.02GB, ssm_state 0.63GB …
+KV Cache is allocated. dtype torch.float8_e4m3fn, #tokens: 45259, K size: 0.22 GB, V size: 0.22 GB
+max_running_requests is capped to 1 by the mamba state cache …
+max_total_num_tokens=45259, chunked_prefill_size=1024, max_running_requests=1, context_len=122880, available_gpu_mem=3.18 GB
+```
+
+Derived rules of thumb for a 32GB card serving 27B@4bit + draft:
+
+- mamba (state) cache: `ssm_state` ≈ **79MB per slot**, and each request takes ~5 slots → 8 slots = 1 request, 16 ≈ 2, 24 ≈ 3. Raising concurrency is a direct trade against KV tokens.
+- KV pool ≈ 45k tokens here; a 122880 context window means one long conversation does not fit. If you need more effective context, raise `--mem-fraction-static`, drop `--kv-cache-dtype` to the default, or add SGLang's hierarchical cache (HiCache) so host RAM becomes a second-level prefix cache.
+- `--cuda-graph-max-bs-decode` should be at least the concurrency you intend, or larger batches run without a captured graph.
+
+## 5. Startup timeline and restart behaviour
+
+- Cold start: weights ~30–40s, then **FlashInfer/`sgl_kernel` JIT compilation — the long pole**. With `MAX_JOBS=3` and a 60GB container a first start ran well past 20 minutes; a rebuild after changing compiler flags is faster but still minutes.
+- Artifacts are cached in `~/.cache/sglang` (`jit/` and `flashinfer/`), plus flashinfer's own `~/.cache/sglang/.cache/flashinfer/<version>/…`. **Do not wipe them** — a restart with a warm cache reaches “The server is fired up and ready to roll!” in about 4 minutes.
+- If you suspect a hung build, check progress cheaply: `ps -eo comm | grep -cE 'nvcc|cicc'` and `find ~/.cache/sglang -name '*.so' | wc -l`.
+- The service writes only to `serve.log`; auditing a failed start is `grep -inE 'Killed|error:|Traceback|ninja:' serve.log | tail -20` — the reason is always *above* the last line.

package/skills/gpu-model-serving/reference/verify.md

@@ -0,0 +1,100 @@
+# Verification
+
+A model server is done when five things answer correctly. Run them in this order; stop at the first failure and read `serve.log`.
+
+Set `BASE=http://<host>:<public-port>`.
+
+## 1. The process is alive and listening
+
+```bash
+curl -s -o /dev/null -w '%{http_code}\n' $BASE/health          # 200
+# from inside the container, if the public address looks dead:
+curl -s -o /dev/null -w 'local=%{http_code}\n' http://127.0.0.1:8080/health
+```
+
+Local 200 but a dead public address means the **port mapping**, not the server.
+
+## 2. The model is registered
+
+```bash
+curl -s $BASE/v1/models
+# {"object":"list","data":[{"id":"<served-model-name>",…,"max_model_len":122880}]}
+```
+
+## 3. Tool calling works
+
+```bash
+cat > req.json <<'JSON'
+{"model":"<served-model-name>",
+ "messages":[{"role":"system","content":"Use the provided tools when needed."},
+             {"role":"user","content":"北京今天天气怎么样？请调用工具查询。"}],
+ "tools":[{"type":"function","function":{"name":"get_weather","description":"查询指定城市的当前天气",
+           "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
+ "tool_choice":"auto","temperature":0,"max_tokens":256}
+JSON
+curl -s -H 'Content-Type: application/json' -d @req.json $BASE/v1/chat/completions
+```
+
+Expect `"finish_reason":"tool_calls"` and a parsed call:
+
+```json
+"tool_calls":[{"id":"call_…","type":"function",
+  "function":{"name":"get_weather","arguments":"{\"city\": \"北京\"}"}}]
+```
+
+With a reasoning parser the chain of thought sits in `reasoning_content` and the visible answer in `content`; a model that only emits think-tags means `--reasoning-parser` is missing or wrong for this family.
+
+## 4. Streaming works
+
+```bash
+curl -sN -H 'Content-Type: application/json' -d @req.json $BASE/v1/chat/completions | head -c 800
+```
+
+Expect `data: {...}` chunks whose deltas accumulate, then a final chunk with `usage`, then `data: [DONE]`.
+
+## 5. Prefix caching works
+
+Send the same request a second time (any long, identical prefix) and read the server log — no client change is needed:
+
+```bash
+grep -oE '#cached-token: [0-9]+' serve.log | tail -4
+```
+
+A cached-token count close to the prompt length (`#cached-token: 320` out of 347) proves the radix cache is hitting. Zero on a genuine repeat means the strategy or the prompt identity is wrong.
+
+## 6. Know your real limits
+
+```bash
+grep -nE 'capped|KV Cache is allocated|max_total_num_tokens' serve.log | tail -6
+```
+
+Report these to the user verbatim: the effective `max_running_requests`, the KV token pool, and the reason for any cap. A single-GPU hybrid model commonly caps concurrency at 1 because of the state cache — say so instead of letting requests pile up silently.
+
+## Wiring a client
+
+Any OpenAI-compatible client works:
+
+```python
+from openai import OpenAI
+client = OpenAI(base_url=f"{BASE}/v1", api_key="sk-local")   # placeholder: the server has no --api-key
+client.chat.completions.create(model="<served-model-name>",
+                               messages=[{"role": "user", "content": "hi"}])
+```
+
+If the runtime is Adelie itself, register the endpoint so it appears in the model picker — the placeholder key is fine:
+
+```bash
+penguin config model add --root <data root> \
+  --provider custom --model-id <served-model-name> \
+  --base-url $BASE/v1 --client-type openai \
+  --context-window <context-length> --no-vision --api-key sk-local
+```
+
+Re-run the same command with a new `--base-url` to move the entry when the public port changes.
+
+## Restart checklist
+
+1. Power the instance on; reopen the exposed port and read the (possibly new) public address.
+2. `cd /root/sj-tmp; setsid nohup ./serve.sh > serve.log 2>&1 < /dev/null &`
+3. Wait until the log says `The server is fired up and ready to roll!` (minutes with a warm JIT cache).
+4. Repeat steps 1–5 above, then update every client with the new address.