lobes-cli 0.78.3__tar.gz → 0.78.4__tar.gz

lobes_cli-0.78.4/.devague/current

@@ -0,0 +1 @@
+qwen3-8-flash-next-on-thor

lobes_cli-0.78.4/.devague/frames/qwen3-8-flash-next-on-thor.json

@@ -0,0 +1,301 @@
+{
+  "slug": "qwen3-8-flash-next-on-thor",
+  "title": "qwen3.8 flash next on thor",
+  "schema_version": 6,
+  "status": "drafting",
+  "created": "2026-09-13T12:31:42Z",
+  "updated": "2026-09-13T12:42:53Z",
+  "claims": [
+    {
+      "id": "c1",
+      "kind": "announcement",
+      "text": "The NemoClaw-Thor recipe for Qwen3.8-Flash-Next (RadixArk NVFP4, 125B MoE) on a Jetson AGX Thor is recorded as a reference doc in this repo, with every measurement attributed to the recipe author \u2014 not built, booted, measured, or reproduced on any fleet box",
+      "origin": "user",
+      "status": "confirmed",
+      "honesty_conditions": [],
+      "hard_questions": [],
+      "links": [],
+      "instruction": "",
+      "revisions": [
+        {
+          "text": "The Jetson AGX Thor serves Qwen3.8-Flash-Next (RadixArk NVFP4, 125B MoE) on vLLM, reproduced from the NemoClaw-Thor fast recipe and measured in this fleet's own evidence",
+          "kind": "announcement",
+          "reason": "PR #267 Qodo 3999631524: the original announcement claimed a fleet reproduction that the operator decided against (2026-09-13, reference only)"
+        }
+      ]
+    },
+    {
+      "id": "c2",
+      "kind": "decision",
+      "text": "operator first authorized dropping the Thor's lobes lanes for this work, then (2026-09-13) chose reference-only: the Thor keeps serving worker and nothing is booted",
+      "origin": "user",
+      "status": "confirmed",
+      "honesty_conditions": [],
+      "hard_questions": [],
+      "links": [],
+      "instruction": "",
+      "revisions": [
+        {
+          "text": "operator authorizes dropping the Thor's lobes lanes for this work and raising them again as needed (user, this session: 'We can drop lobes and raise as needed on thor. You have ssh thor control')",
+          "kind": "decision",
+          "reason": "superseded by the operator's doc-only decision in the same session"
+        }
+      ]
+    },
+    {
+      "id": "c3",
+      "kind": "assumption",
+      "text": "the NemoClaw-Thor fast recipe (serving/docs/QWEN38-FLASH-NEXT-FAST-THOR.md, commit 9c9c0873, 2026-09-06) is the starting recipe: RadixArk/Qwen3.8-Flash-Next-NVFP4 @7b719225, gpu_mem_util 0.90, BF16 KV, max_model_len 262144, max_num_seqs 4, embedded MTP n=3, fused SM110a GDN CUDA decode, flashinfer_cutlass MoE, PLE mmap MADV_RANDOM + parallel reads; its reported numbers (34.98 tok/s single-stream mean, 677,323 KV tokens / 2.58x, 68.10 tok/s 4-way short) are the author's, NOT yet reproduced here",
+      "origin": "llm",
+      "status": "proposed",
+      "honesty_conditions": [],
+      "hard_questions": [],
+      "links": [],
+      "instruction": "",
+      "revisions": []
+    },
+    {
+      "id": "c4",
+      "kind": "requirement",
+      "text": "the served image is a 3-layer LOCAL build pinned by sha: blazux/qwen3.8-Flash-DGX @4b723de2 (DET_ARCH=110a) -> Dockerfile.qwen38-flash-next-sm110 (SM110 flashinfer_cutlass MoE gate patch) -> Dockerfile.qwen38-flash-next-gdn-sm110 (vLLM @082cf021 fused GDN decode kernel, PR #53835 merged 2026-09-05, + PLE I/O patch); every sha and the resulting image ID are recorded in docs/image-ledger.md, following the lobes/vllm-gemma4:local precedent",
+      "origin": "llm",
+      "status": "proposed",
+      "honesty_conditions": [],
+      "hard_questions": [],
+      "links": [],
+      "instruction": "",
+      "revisions": []
+    },
+    {
+      "id": "c5",
+      "kind": "boundary",
+      "text": "pin, don't follow upstream: blazux/qwen3.8-Flash-DGX main is already 32 commits past the recipe's 4b723de2 pin (pushed 2026-09-13, incl. a 'temporary' block-fp8 MTP shim pending vllm#55513); the reproduction uses the recipe's pins, and a re-pin is a separate measured change",
+      "origin": "llm",
+      "status": "proposed",
+      "honesty_conditions": [],
+      "hard_questions": [],
+      "links": [],
+      "instruction": "",
+      "revisions": []
+    },
+    {
+      "id": "c6",
+      "kind": "assumption",
+      "text": "the 2026-08-27 lobes conclusion that RadixArk/Qwen3.8-Flash-Next-NVFP4 cannot fit the Thor ('135 GB, SGLang-only, misses by ~13 GB'; VLLM_PLE_CPU_OFFLOAD a no-op on unified memory) is SUPERSEDED as a premise: the preview vLLM serves it with VLLM_PLE_MMAP=1 mapping the FP8 PLE table from NVMe, and HF reports the pinned revision at 126.0 GiB over 419 files",
+      "origin": "llm",
+      "status": "proposed",
+      "honesty_conditions": [],
+      "hard_questions": [],
+      "links": [],
+      "instruction": "",
+      "revisions": []
+    },
+    {
+      "id": "c7",
+      "kind": "requirement",
+      "text": "disk must be reclaimed before the checkpoint download: the Thor's single NVMe (WD SN5000S 1 TB) has 146 GB free against a 126 GiB checkpoint plus three new ~30 GB image layers; docker reports 105.9 GB reclaimable images and 52.7 GB reclaimable build cache",
+      "origin": "llm",
+      "status": "proposed",
+      "honesty_conditions": [],
+      "hard_questions": [],
+      "links": [],
+      "instruction": "",
+      "revisions": []
+    },
+    {
+      "id": "c8",
+      "kind": "boundary",
+      "text": "the non-lobes stack on the Thor (culture-nodes prod-api/worker/scheduler/notifier, postgres:17, minio, backup) is not part of this work and is not stopped; it holds ~0.6 GiB resident, which counts against the recipe's 0.90 util and its ~7.2 GiB post-run available-memory margin",
+      "origin": "llm",
+      "status": "proposed",
+      "honesty_conditions": [],
+      "hard_questions": [],
+      "links": [],
+      "instruction": "",
+      "revisions": []
+    },
+    {
+      "id": "c9",
+      "kind": "requirement",
+      "text": "dropping lobes on the Thor removes the mesh's only worker host: the Spark reaches model=worker by proxy to the Thor (validated 2026-09-11), so for the window model=worker is unavailable mesh-wide, and restoring it means re-raising the captured thor-worker deployment",
+      "origin": "llm",
+      "status": "proposed",
+      "honesty_conditions": [],
+      "hard_questions": [],
+      "links": [],
+      "instruction": "",
+      "revisions": []
+    },
+    {
+      "id": "c10",
+      "kind": "non_goal",
+      "text": "no new Colleague role and no role re-point in this scope: Flash-Next lands, if at all, as a catalog candidate; docs/colleague-stack.md says a different checkpoint is a catalog change and adding a role is effectively irreversible, and both 2026-08-27 specs recorded the same non-goal",
+      "origin": "llm",
+      "status": "proposed",
+      "honesty_conditions": [],
+      "hard_questions": [],
+      "links": [],
+      "instruction": "",
+      "revisions": []
+    },
+    {
+      "id": "c11",
+      "kind": "assumption",
+      "text": "integrating it as a lobes lane would touch: lobes/catalog.py (new engine=vllm candidate with moe_backend flashinfer_cutlass + MTP speculative_config; SupportedModel has no image field), lobes/templates/fleet/docker-compose.yml (vllm-primary has NO per-lane image override \u2014 only VLLM_NIGHTLY_IMAGE; vllm-worker has WORKER_IMAGE \u2014 and none passes the VLLM_PLE_*/VLLM_QSA_*/compile-cache env or -cc.splitting_ops), and a new solo Thor shape, since 0.90 util can't co-reside with thor-worker's lanes",
+      "origin": "llm",
+      "status": "proposed",
+      "honesty_conditions": [],
+      "hard_questions": [],
+      "links": [],
+      "instruction": "",
+      "revisions": []
+    }
+  ],
+  "open_vagueness": [
+    {
+      "id": "v1",
+      "text": "vision: the checkpoint declares vision_config / image-text-to-text, but the NemoClaw recipe never tested image input; whether the preview image serves it on sm_110 is unmeasured",
+      "kind": "unknown_nonblocking",
+      "claim_id": null,
+      "resolved": false,
+      "resolution": "",
+      "resolution_claim_id": null
+    },
+    {
+      "id": "v2",
+      "text": "restart stability: the recipe recorded one CUDA Xid 13 (illegal instruction) on restart with the serialized compile cache, worked around with VLLM_DISABLE_COMPILE_CACHE=1, explicitly not a restart soak \u2014 root cause unknown",
+      "kind": "unknown_nonblocking",
+      "claim_id": null,
+      "resolved": false,
+      "resolution": "",
+      "resolution_claim_id": null
+    },
+    {
+      "id": "v3",
+      "text": "mixed-length concurrent prefill OOM: upstream vLLM issue allocates a dense num_prefills x max_chunk PLE short-conv buffer; the recipe's 4-way mixed smoke passed with ~7.2 GiB available but is 'not a guarantee'",
+      "kind": "unknown_nonblocking",
+      "claim_id": null,
+      "resolved": false,
+      "resolution": "",
+      "resolution_claim_id": null
+    },
+    {
+      "id": "v4",
+      "text": "PLE read performance depends on NVMe: the recipe's Thor storage model is unstated; this Thor's WD SN5000S may read the PLE table slower or faster, which feeds decode tok/s directly",
+      "kind": "unknown_nonblocking",
+      "claim_id": null,
+      "resolved": false,
+      "resolution": "",
+      "resolution_claim_id": null
+    },
+    {
+      "id": "v5",
+      "text": "whether dropping every lobes lane frees enough for 0.90 on this Thor given the recipe's own ~7-10 GiB available-memory margin and this box's 7.3 GiB already-swapped pages; boot log KV capacity is authoritative",
+      "kind": "unknown_blocking",
+      "claim_id": null,
+      "resolved": true,
+      "resolution": "moot: operator chose not to run the recipe; the Thor stays the worker host (2026-09-13)",
+      "resolution_claim_id": null
+    }
+  ],
+  "scope_entries": [
+    {
+      "id": "s1",
+      "surface": "operator instruction (this session)",
+      "finding": "Thor lobes lanes may be stopped for the Flash-Next window and re-raised as needed; ssh thor@thor control granted",
+      "seeds": [
+        "c2"
+      ]
+    },
+    {
+      "id": "s2",
+      "surface": "NemoClaw-Thor serving/docs/QWEN38-FLASH-NEXT-FAST-THOR.md + serving/start-qwen38-flash-next-fast.sh",
+      "finding": "measured on a Thor 2026-09-05: 3-prompt coding probe, temp 0, thinking off, 512-token cap -> 34.98 tok/s mean (vs 24.97 Triton baseline); 19.58 GiB KV = 677,323 tokens; tool-call and 25,640-token retrieval smoke only; no full-256K request, no quality eval, no vision test",
+      "seeds": [
+        "c3"
+      ]
+    },
+    {
+      "id": "s3",
+      "surface": "NemoClaw-Thor serving/docker/Dockerfile.qwen38-flash-next{,-gdn}-sm110 + patches/",
+      "finding": "overlay 1 lifts vLLM's SM110 exclusion in flashinfer_cutlass_moe.py (refuses if gate text not found exactly once); overlay 2 ADDs 4 files from vllm@082cf021 and builds thor_gdn ops (build.py/install.py/patch_ple_io.py), sets VLLM_GDN_DECODE_KERNEL=cuda",
+      "seeds": [
+        "c4"
+      ]
+    },
+    {
+      "id": "s4",
+      "surface": "docs/image-ledger.md",
+      "finding": "no rule forbids an unpublished local image (lobes/vllm-gemma4:local precedent); registry pulls must be digest-pinned and source commits recorded",
+      "seeds": [
+        "c4"
+      ]
+    },
+    {
+      "id": "s5",
+      "surface": "github blazux/qwen3.8-Flash-DGX (gh api compare 4b723de2...main)",
+      "finding": "ahead_by 32; latest commit 2c5701d0 2026-09-13 notes patch 11 is temporary pending vllm#55513",
+      "seeds": [
+        "c5"
+      ]
+    },
+    {
+      "id": "s6",
+      "surface": "docs/specs/2026-08-27-flash-next-on-vllm.md + docs/experiments/qwen3.8-flash-next-gguf-llamacpp-vllm.md",
+      "finding": "both specs exported, execution deferred 2026-08-27 before stage 1; nothing ever booted; the NVFP4 was ruled out on footprint and the GGUF-via-vllm-gguf-plugin route chosen on paper; success bar there: >=25 tok/s single-stream at MAXN (~2x cortex's 12.1)",
+      "seeds": [
+        "c6"
+      ]
+    },
+    {
+      "id": "s7",
+      "surface": "HF API RadixArk/Qwen3.8-Flash-Next-NVFP4 @7b719225",
+      "finding": "126.0 GiB, 419 files, many model-plefp8-*.safetensors shards (4.84 GiB each); arch Qwen4ExpForConditionalGeneration, vision_config present, 48 layers, 512 experts, mtp_num_hidden_layers 1, max_position_embeddings 262144; ungated",
+      "seeds": [
+        "c6"
+      ]
+    },
+    {
+      "id": "s8",
+      "surface": "Thor host (ssh thor@thor: df, docker system df, lsblk)",
+      "finding": "/ 937G, 744G used, 146G free, 84%; images 212.5GB (105.9GB reclaimable); build cache 63.96GB (52.74GB reclaimable); lobes HF cache ~/.cache/huggingface = 73G, has no Flash-Next snapshot; recipe expects ~/thor-hf-cache",
+      "seeds": [
+        "c7"
+      ]
+    },
+    {
+      "id": "s9",
+      "surface": "Thor host (docker ps / docker stats)",
+      "finding": "running: model-gear-gateway, vllm-worker (8.29 GiB rss, WORKER util 0.45), vllm-embed, vllm-rerank (lobes 0.77.0.dev544, COMPOSE_PROFILES=worker, LOBES_MESH_NAME=thor) plus 7 prod-* culture-nodes/postgres/minio containers; MAXN; R38.2.2; 122 GiB RAM, 59.6 GiB swapfile 7.3 GiB used; passwordless sudo OK",
+      "seeds": [
+        "c8"
+      ]
+    },
+    {
+      "id": "s10",
+      "surface": "CLAUDE.md worker paragraph + deployments/jetson-agx-thor__thor-worker/deployment.lock.toml",
+      "finding": "Thor hosts worker via thor-worker shape (util 0.45, 262144, WORKER_MAX_NUM_SEQS=1); Spark proxies worker to it (docs/evidence/2026-09-11-accept-worker-proxy-spark-thor.txt); lock captured 2026-09-11 = rollback reference",
+      "seeds": [
+        "c9"
+      ]
+    },
+    {
+      "id": "s11",
+      "surface": "docs/colleague-stack.md ('adding a role is effectively irreversible') + lobes/roles.py",
+      "finding": "a new role costs a public /capabilities key, an alias, six per-role tables, every profile/shape and ~28 golden .env files; checkpoint swaps are catalog changes",
+      "seeds": [
+        "c10"
+      ]
+    },
+    {
+      "id": "s12",
+      "surface": "lobes/catalog.py:70-130 + lobes/templates/fleet/docker-compose.yml:53,1370-1401 + builtin_shapes/thor-worker.toml + profiles/builtin/thor.toml",
+      "finding": "per-lane *_IMAGE override pattern exists for worker/hand/multimodal/associate, not primary; serve args are one shell-lexed string; thor.toml already carries VLLM_GDN_DECODE_KERNEL=triton as a required cortex env (sm_110 MTP precedent)",
+      "seeds": [
+        "c11"
+      ]
+    }
+  ],
+  "lapses": [],
+  "obligations": []
+}

{lobes_cli-0.78.3 → lobes_cli-0.78.4}/CHANGELOG.md

@@ -4,6 +4,17 @@ All notable changes to this project are documented here. The format is based on
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
 adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.78.4] - 2026-09-13
+
+### Added
+
+- docs/experiments/qwen3.8-flash-next-nvfp4-thor-reference.md — a reference record (not reproduced here) of the external NemoClaw-Thor recipe that serves RadixArk/Qwen3.8-Flash-Next-NVFP4 on a Jetson AGX Thor: PLE table mmapped from NVMe, fused SM110a GDN decode kernel, SM110 FlashInfer CUTLASS MoE gate, embedded MTP n=3, util 0.90 at 262144; author-reported ~35 tok/s single-stream. Decision: the Thor keeps serving worker; nothing is booted.
+
+### Changed
+
+- docs/experiments/qwen3.8-flash-next-gguf-llamacpp-vllm.md — correction note: the 2026-08-27 NVFP4 "misses by ~13 GB" premise assumed a resident PLE table; the table itself is kept as the evaluation record.
+- docs/experiments/README.md — index row for the new reference doc.
+
 ## [0.78.3] - 2026-09-13
 
 ### Changed

{lobes_cli-0.78.3 → lobes_cli-0.78.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: lobes-cli
-Version: 0.78.3
+Version: 0.78.4
 Summary: lobes — run, assess, and switch the local vLLM model.
 Project-URL: Homepage, https://github.com/agentculture/lobes-cli
 Project-URL: Issues, https://github.com/agentculture/lobes-cli/issues

{lobes_cli-0.78.3 → lobes_cli-0.78.4}/docs/experiments/README.md

@@ -40,3 +40,4 @@ was made, and deleting it loses that.
 | doc | checkpoint | status |
 |---|---|---|
 | [`qwen3.8-flash-next-gguf-llamacpp-vllm.md`](qwen3.8-flash-next-gguf-llamacpp-vllm.md) | `unsloth/Qwen3.8-Flash-Next-GGUF` — 125B MoE + 51B n-gram, 6B active | **NOT SERVED** — deferred 2026-08-27; nothing native fits 122 GiB, llama.cpp costs ~25x on prefill, vLLM route has four open unknowns |
+| [`qwen3.8-flash-next-nvfp4-thor-reference.md`](qwen3.8-flash-next-nvfp4-thor-reference.md) | `RadixArk/Qwen3.8-Flash-Next-NVFP4` @ `7b719225` — 126.0 GiB | **REFERENCE ONLY** — 2026-09-13; an external Thor recipe (NemoClaw-Thor) serves it on a preview vLLM by mmapping the PLE table from NVMe, author-reported ~35 tok/s; not reproduced here — the Thor stays the `worker` host |

{lobes_cli-0.78.3 → lobes_cli-0.78.4}/docs/experiments/qwen3.8-flash-next-gguf-llamacpp-vllm.md

@@ -57,8 +57,19 @@ table is the floor no expert-only quantization can go below — which is also wh
 `UD-IQ1_S` at 72.5 GB is not nearly as small as a 1-bit quant of 176B
 parameters should be.
 
-**So GGUF is currently the only format that fits.** Any engine choice here is a
-choice about *which engine reads a GGUF*, not about serving the native weights.
+**As of 2026-08-27, GGUF was the only route identified that fits.** Any engine
+choice here was a choice about *which engine reads a GGUF*, not about serving
+the native weights. That is no longer the only known route — see the
+correction below. The NVFP4 route it names is unverified on this fleet.
+
+> **Correction (2026-09-13):** the NVFP4 row above assumes the PLE table must be
+> resident. It does not have to be: an external Thor recipe serves
+> `RadixArk/Qwen3.8-Flash-Next-NVFP4` (126.0 GiB at the pinned revision) on a
+> preview vLLM by memory-mapping the table from NVMe, at util 0.90. That is
+> author-reported and not reproduced on a fleet box. See
+> [`qwen3.8-flash-next-nvfp4-thor-reference.md`](qwen3.8-flash-next-nvfp4-thor-reference.md#why-the-2026-08-27-doesnt-fit-was-wrong).
+> The table above is kept as written, because it is the record of that
+> evaluation.
 
 ## Route 1 — llama.cpp
 

lobes_cli-0.78.4/docs/experiments/qwen3.8-flash-next-nvfp4-thor-reference.md

@@ -0,0 +1,285 @@
+# Qwen3.8-Flash-Next NVFP4 on a Thor — the NemoClaw-Thor reference recipe
+
+**Status: REFERENCE ONLY. Not served, not reproduced here.** This fleet has not
+built these images or booted this checkpoint. Every number below is **the
+recipe author's own measurement on their own Jetson AGX Thor**, quoted from
+[pastoriomarco/NemoClaw-Thor](https://github.com/pastoriomarco/NemoClaw-Thor)
+at commit `9c9c0873` (2026-09-06). Per #108 none of it may be cited as
+validated on a fleet box until a transcript lands under `docs/evidence/`.
+
+**Decision (2026-09-13): record, don't run.** The Thor keeps serving `worker`
+through the `thor-worker` shape. Running this recipe would mean stopping every
+lobes lane there, which takes `model=worker` offline for the whole mesh, since
+the Spark forwards `worker` to the Thor. The recipe is kept here so that
+resuming it later needs no re-derivation.
+
+Sibling doc:
+[`qwen3.8-flash-next-gguf-llamacpp-vllm.md`](qwen3.8-flash-next-gguf-llamacpp-vllm.md),
+the 2026-08-27 evaluation that ruled this checkpoint out. This doc **corrects
+that doc's footprint premise** — see
+[Why the 2026-08-27 "doesn't fit" was wrong](#why-the-2026-08-27-doesnt-fit-was-wrong).
+
+## Sources
+
+| source | what it carries |
+|---|---|
+| `serving/docs/QWEN38-FLASH-NEXT-FAST-THOR.md` | the recommended recipe, measurements, caveats |
+| `serving/docs/QWEN38-FLASH-NEXT-NVFP4-THOR.md` | the earlier Triton-GDN baseline the fast recipe supersedes |
+| `serving/start-qwen38-flash-next-fast.sh` | the exact `docker run` / `vllm serve` invocation |
+| `serving/docker/Dockerfile.qwen38-flash-next-sm110` | overlay 1: the SM110 MoE-backend gate |
+| `serving/docker/Dockerfile.qwen38-flash-next-gdn-sm110` | overlay 2: the fused GDN decode kernel and PLE I/O |
+| `serving/docker/patches/qwen38-flash-next-sm110-moe.py`, `patches/thor-gdn/` | the patches both overlays apply |
+| [blazux/qwen3.8-Flash-DGX](https://github.com/blazux/qwen3.8-Flash-DGX) | the upstream DGX Spark preview vLLM build the recipe starts from |
+
+All paths are in NemoClaw-Thor unless a link says otherwise.
+
+## The checkpoint
+
+| fact | value | source |
+|---|---|---|
+| repo / revision | `RadixArk/Qwen3.8-Flash-Next-NVFP4` @ `7b719225242aacd3dbd3f9407468c2ee9a9d2594` | recipe pin; also HF HEAD as of 2026-09-13 |
+| size | **126.0 GiB** over 419 files; the FP8 PLE table ships as `model-plefp8-*.safetensors` shards of 4.84 GiB | HF API, read 2026-09-13 |
+| architecture | `Qwen4ExpForConditionalGeneration` — 48 layers, 512 experts, `mtp_num_hidden_layers: 1` | `config.json` at that revision |
+| modality | `vision_config` present, pipeline `image-text-to-text` | `config.json`, HF API |
+| context | `max_position_embeddings: 262144` | `config.json` |
+| quantization | native NVFP4 routed experts, BF16 protected layers, FP8 PLE table | recipe |
+| drafter | none needed — MTP is embedded in the checkpoint | recipe |
+| access | ungated | HF API |
+
+The recipe never tested vision. The checkpoint declares it; whether the preview
+image serves it on sm_110 is unknown.
+
+## How it fits and runs: six mechanisms
+
+### 1. The PLE table is memory-mapped from NVMe, not held in memory
+
+The checkpoint's n-gram PLE table is the part no expert-only quantization can
+shrink. The preview vLLM maps it from disk (`VLLM_PLE_MMAP=1`, 14 reader
+workers, no prewarm) and gathers only the rows a step actually needs. This is
+why a 126 GiB checkpoint serves on a 122 GiB unified-memory board at
+`gpu_memory_utilization=0.90`. The recipe accordingly requires **fast local
+NVMe** for the model cache: the table is read during inference, not just at
+load.
+
+### 2. PLE reads are tuned for random access
+
+Overlay 2's `patch_ple_io.py` maps the table with `MADV_RANDOM`, which stops the
+kernel's useless sequential read-ahead. `VLLM_PLE_MMAP_FAST_ROWS=0` routes decode
+through the existing parallel-read path as well. The author's read-only
+microprobe — 100 random 64-row gathers and one 32768-row gather, page cache
+cleared between variants:
+
+| mapping / gather | mean 64-row lookup | 32768-row lookup |
+|---|---:|---:|
+| original / serial small batches | 36.22 ms | 3561 ms |
+| `MADV_RANDOM` / serial | 10.17 ms | 422 ms |
+| `MADV_RANDOM` / parallel | 2.06 ms | 414 ms |
+
+All variants returned byte-identical rows (same SHA256). This is an I/O probe,
+not a tok/s result — and it depends on the author's NVMe, whose model the recipe
+does not state.
+
+### 3. A fused GDN decode kernel compiled for SM110a
+
+Three of every four layers are Gated DeltaNet. The preview image's fused CUDA
+GDN MTP decode kernel ships **no SM110 code object**, so the first recipe fell
+back to Triton (`VLLM_GDN_DECODE_KERNEL=triton`). This fleet hit the same gap
+on the 27B cortex; `lobes/profiles/builtin/thor.toml` carries the same
+`triton` conjunct.
+
+Overlay 2 closes the gap. It takes four files from vLLM at `082cf021`, the
+source associated with the merged
+[vLLM PR #53835](https://github.com/vllm-project/vllm/pull/53835) ("Build fused
+GDN MTP decode for SM110", merged 2026-09-05), and builds them with
+`TORCH_CUDA_ARCH_LIST=11.0a`. The build registers them in a separate
+`thor_gdn` op namespace and redirects only the Python dispatch. It then sets
+`VLLM_GDN_DECODE_KERNEL=cuda`. The upstream numerical tests
+(`test_fused_gdn_post_conv.py`, 22 of them, including recurrent-state rollback
+and ragged batches) passed on the author's Thor. Those tests use numerical
+tolerances; they are not a quality eval. GDN **prefill** and QSA stay on their
+Triton implementations.
+
+### 4. The FlashInfer CUTLASS MoE backend is unblocked on SM110
+
+The preview vLLM excludes compute capability 11.x from
+`flashinfer_cutlass_moe.py` (citing flashinfer#3134), and its automatic backend
+picked an SM100 vLLM CUTLASS kernel that failed on the Thor. Overlay 1's patch
+adds `is_device_capability_family(110)` to that gate. It refuses to run unless
+the expected gate text appears exactly once. The launcher then selects the
+backend explicitly with `--moe-backend flashinfer_cutlass`. The patch's
+rationale: bundled FlashInfer 0.6.17 already maps SM110 to its SM100 CUTLASS
+path.
+
+### 5. Embedded MTP at n=3
+
+`--speculative-config '{"method":"mtp","num_speculative_tokens":3}'` drafts from
+the checkpoint's own head. MTP=3 is the default because it measured faster **per
+request**. MTP=2 measured higher **aggregate throughput** (see below). The QSA
+backend still rebuilds metadata between draft steps, so the preview does not
+support fully fused multi-step drafting.
+
+### 6. Serialized compile cache bypassed
+
+One restart failed with CUDA Xid 13 (illegal instruction) during sampler warmup
+after loading an AOT artifact. It was not an OOM. The launcher now sets
+`VLLM_DISABLE_COMPILE_CACHE=1` and keeps `VLLM_USE_AOT_COMPILE=1`. That bypasses
+the serialized cache without going eager or losing CUDA graphs, at the cost of
+extra compile work on every boot. The author calls it a verified workaround, not
+a root cause, and has not soak-tested restarts.
+
+## The image: three local layers, every revision pinned
+
+None of these is a published image. Each is built locally, in order:
+
+| tag | built from | pin |
+|---|---|---|
+| `nemoclaw-thor/qwen38-flash-next-vllm:sm110` | `blazux/qwen3.8-Flash-DGX`, `--build-arg DET_ARCH=110a` | commit `4b723de2e2c465d866738b57ae64bde6e8c07744`; its Dockerfile pins its base by digest |
+| `…:sm110-flashinfer-moe` | `Dockerfile.qwen38-flash-next-sm110` (mechanism 4) | NemoClaw-Thor `9c9c0873` |
+| `…:sm110-gdn-cuda` | `Dockerfile.qwen38-flash-next-gdn-sm110` (mechanisms 2, 3) | vLLM `082cf021b7ef96e4819e386846ea34e5ef21c655` |
+
+The author's final running image ID:
+`sha256:c4800004609fffc26f9c5f6ed90671bee76ae918261e3b93824672d85303b462`.
+
+**Upstream has moved past the pin.** As of 2026-09-13, `blazux/qwen3.8-Flash-DGX`
+main is 32 commits ahead of `4b723de2`, including a block-fp8 MTP shim its own
+docs call temporary (pending vllm#55513). A reproduction should use the recipe's
+pins. Moving to a newer upstream commit is a separate, measured change.
+
+If this fleet ever builds these images, `docs/image-ledger.md` is where the
+shas and the resulting image ID go — `lobes/vllm-gemma4:local` is the precedent
+for an unpublished local build.
+
+## The serve invocation
+
+From `serving/start-qwen38-flash-next-fast.sh`, defaults shown:
+
+```bash
+docker run -it --pull never --name qwen38-flash-next-fast \
+  --runtime nvidia --gpus all --ipc host --network host --shm-size 16g \
+  -v "$HF_CACHE:/hf" -v "$VLLM_CACHE:/root/.cache/vllm" \
+  -v "$TORCH_CACHE:/root/.cache/torch" -v "$FLASHINFER_CACHE:/root/.cache/flashinfer" \
+  -e HF_HOME=/hf -e HF_HUB_OFFLINE=1 \
+  -e VLLM_DISABLE_COMPILE_CACHE=1 -e VLLM_USE_AOT_COMPILE=1 \
+  -e CUTE_DSL_ARCH=sm_110a -e TORCH_CUDA_ARCH_LIST=11.0a \
+  -e VLLM_PLE_MMAP=1 -e VLLM_PLE_MMAP_WORKERS=14 -e VLLM_PLE_MMAP_PREWARM=0 \
+  -e VLLM_PLE_MMAP_MADV_RANDOM=1 -e VLLM_PLE_MMAP_FAST_ROWS=0 \
+  -e VLLM_QSA_DET_TOPK=1 -e VLLM_QSA_DET_LIB=/opt/llm/kernel-det/_C_det.so \
+  -e VLLM_QSA_EXACT_TOPK=0 -e VLLM_USE_FLASHINFER_SAMPLER=1 \
+  -e VLLM_GDN_DECODE_KERNEL=cuda \
+  --entrypoint vllm nemoclaw-thor/qwen38-flash-next-vllm:sm110-gdn-cuda \
+  serve "/hf/hub/models--RadixArk--Qwen3.8-Flash-Next-NVFP4/snapshots/7b719225242aacd3dbd3f9407468c2ee9a9d2594" \
+  --served-model-name qwen3.8-flash-next --host 0.0.0.0 --port 8050 \
+  --load-format safetensors --max-model-len 262144 \
+  --max-num-seqs 4 --gpu-memory-utilization 0.90 \
+  --enable-prefix-caching --enable-chunked-prefill --max-num-batched-tokens 8192 \
+  -cc.cudagraph_mode=PIECEWISE "-cc.splitting_ops=$split_ops" \
+  --no-enable-flashinfer-autotune --moe-backend flashinfer_cutlass \
+  --kv-cache-dtype auto --enable-auto-tool-choice \
+  --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
+  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
+```
+
+`$split_ops` is the launcher's JSON list of piecewise-CUDA-graph split points.
+Beyond the usual attention/mamba ops, it names three Flash-Next-specific ops:
+`vllm::qwen3_8_flash_next_ple_short_conv`, `vllm::qwen3_8_flash_next_qsa_with_output`
+and `vllm::ple_mmap_lookup`. `VLLM_QSA_DET_TOPK=1` selects deterministic QSA
+top-k, and the recipe also keeps its prefix-cache fixes. The recipe's own
+prerequisites: JetPack 7.1, and stopping every other GPU model server first.
+
+## The author's measurements
+
+Three fixed coding prompts, one request at a time, temperature 0, thinking off,
+512-token output cap, warmup excluded:
+
+| configuration | per-request decode tok/s | mean |
+|---|---|---:|
+| Triton GDN, MTP=2, util 0.85, original PLE I/O | 23.96 / 22.99 / 27.96 | 24.97 |
+| CUDA GDN, MTP=2, 0.85, original PLE I/O | 25.10 / 24.80 / 28.52 | 26.14 |
+| CUDA GDN, MTP=2, 0.90, optimized PLE I/O | 33.84 / 31.31 / 33.81 | 32.99 |
+| CUDA GDN, MTP=3, 0.90, optimized PLE I/O | 35.92 / 32.57 / 37.82 | 35.43 |
+| **final: MTP=3, compile cache bypassed** | 35.77 / 32.20 / 36.96 | **34.98** |
+
+The fused kernel alone was worth ~4.7%; most of the gain came from the PLE I/O
+work plus util 0.90. The probe (`serving/benchmarks/flash-next-gdn-ab.py`)
+excludes TTFT and approximates multi-token speculative SSE chunks.
+
+| other figure | value |
+|---|---|
+| KV at the final boot (0.90, MTP=3, BF16 KV) | 19.58 GiB = **677,323 tokens**; KV-pool concurrency ceiling 2.58x at 262144 |
+| KV at an MTP=2 boot | 711,119 tokens; KV-pool ceiling 2.71x |
+| Triton baseline KV (0.85, MTP=2) | 12.45 GiB = 440,286 tokens; KV-pool ceiling 1.68x |
+| 4 concurrent short prompts, 256 tokens each, MTP=3 | **68.10 tok/s** aggregate; 22.52 tok/s mean per request |
+| same, MTP=2 | **73.44 tok/s** aggregate (+7.8%); 22.37 tok/s per request |
+| 4 concurrent unequal prompts (68 / 2968 / 11595 / 27068 input tokens), incl. cold prefill + JIT | 28.41 tok/s aggregate, no failure |
+| host memory available after the 4-way runs | ~10 GiB (short), ~7.2 GiB (unequal); swap unchanged |
+| earlier MTP=2 draft acceptance (Triton baseline, live windows) | 49.52% weighted, mean length 2.03 |
+| smoke | a parsed `read_file` tool call; a 25,640-token retrieval returned the right answer twice (13.28 s cold, 1.38 s prefix-cached) |
+
+The `x` figures are **KV-pool concurrency ceilings**: KV tokens divided by
+`max_model_len` (262144). They are arithmetic from the boot log, not measured
+throughput, and must not be multiplied by a single-stream tok/s.
+
+**Not tested by the author:** any full 256K request, four resident full-length
+contexts, a sustained or multi-user load, a quality eval, or vision.
+
+**For scale against this fleet's own measured numbers:** on the same board model,
+the 27B cortex measured 12.1 tok/s at 1M
+(`docs/evidence/2026-08-20-accept-cortex-local-thor.txt`), and the `worker` lane
+(`nvidia/Qwen3.6-35B-A3B-NVFP4`) measured 196.6 tok/s
+(`docs/nvidia-qwen3.6-35b-a3b-nvfp4.md`). The author's ~35 tok/s would clear
+the ≥25 tok/s bar the 2026-08-27 spec set — but it is a different box, a
+different workload, and not our measurement.
+
+## Why the 2026-08-27 "doesn't fit" was wrong
+
+The sibling doc lists `RadixArk/Qwen3.8-Flash-Next-NVFP4` as "135 GB — **no,
+misses by ~13 GB**", published for SGLang. It also ruled `VLLM_PLE_CPU_OFFLOAD`
+a no-op on a unified-memory board, since host RAM and GPU RAM are one pool.
+Both observations are correct as far as they go. The footprint conclusion
+drawn from them is not:
+
+- the footprint assumed the whole PLE table must be **resident** — mmapping it
+  from NVMe (mechanism 1) makes it disk-backed and paged in on demand, which is
+  a different axis from CPU offload;
+- a preview vLLM (`blazux/qwen3.8-Flash-DGX`) serves this checkpoint; the
+  checkpoint is not SGLang-only in practice;
+- the pinned revision measures 126.0 GiB on HF, not 135 GB.
+
+So the GGUF-through-`vllm-gguf-plugin` route that spec chose on paper was not
+the only route that fits. That spec's other open items — the vLLM version and
+sm_110 kernel coverage — are exactly what this recipe's two overlays answer,
+using a preview build instead of a release.
+
+## What running it on a fleet box would take
+
+This was scoped read-only on 2026-09-13 and not pursued:
+
+- **The Thor stops being the `worker` host** for the window. The recipe needs
+  0.90 util with no other model servers running, and the captured deployment
+  `deployments/jetson-agx-thor__thor-worker/` is the rollback. This is the
+  reason for the decision above.
+- **Disk**: the Thor's single 1 TB NVMe had 146 GB free, against 126 GiB of
+  weights plus three ~30 GB image layers. It would need a cleanup first.
+- **Memory headroom** is the author's ~7–10 GiB available after load, before
+  counting anything else resident on the box.
+- **As a lobes lane**, not a hand launcher, it would touch:
+  - `lobes/catalog.py` — a candidate entry; `SupportedModel` has no image field.
+  - The fleet compose template — `vllm-primary` has no per-lane image override,
+    and no lane passes through the `VLLM_PLE_*` / `VLLM_QSA_*` / compile-cache
+    env or `-cc.splitting_ops`.
+  - A solo Thor shape.
+- It would be a **catalog change, not a new role**, per
+  `docs/colleague-stack.md`.
+
+## Open unknowns carried by the recipe itself
+
+- **Mixed-length concurrent prefill OOM.** An upstream vLLM issue allocates a
+  dense `num_prefills x max_chunk_length` PLE short-conv buffer; this has OOMed
+  at util 0.85 even with low KV use. The 4-way unequal smoke passed, which the
+  author says is not a guarantee.
+- **Restart stability** after the Xid 13 workaround (mechanism 6).
+- **FP8 KV** for four resident full contexts is proposed in the baseline doc and
+  unvalidated. There is no FP4 KV in this stack.
+- **Storage sensitivity.** Decode speed rides PLE read latency, which depends on
+  the NVMe.
+- **Vision** is declared by the checkpoint and untested.

{lobes_cli-0.78.3 → lobes_cli-0.78.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "lobes-cli"
-version = "0.78.3"
+version = "0.78.4"
 description = "lobes — run, assess, and switch the local vLLM model."
 readme = "README.md"
 license = "Apache-2.0"