vllm-quantpack 0.1.0__tar.gz

vllm_quantpack-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

vllm_quantpack-0.1.0/NOTICE

@@ -0,0 +1,25 @@
+vllm-quantpack
+Copyright 2026 Jorge Ramírez Espejo
+
+Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+This product includes code from OrcaSAQ2 (https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel, commit 7ddffb1),
+licensed under the Apache License, Version 2.0: the EXL3 quantization config, linear and embedding methods, the shard
+GEMM custom op and the embedding-layer constructor wrapper (vllm_quantpack/exl3/). Modified: prefix normalisation, empty
+draft heads, padding, dtype conversion, host-memory release, tensor parallelism, MoE dispatch.
+
+The Flint model (vllm_quantpack/families/flint/model.py) is adapted from vLLM's AfMoE model
+(https://github.com/vllm-project/vllm, Apache-2.0); its configuration class (config.py) comes from the SGLang serving kit
+published with RecursiveSI Flint (https://huggingface.co/RecursiveSI, Apache-2.0).
+
+The quantized-embedding support for Qwen3.5-family GPTQ checkpoints (vllm_quantpack/families/qwen.py and
+vllm_quantpack/exl3/layers.py) replaces, without modifying vLLM, qwen3_5-embed-quant.patch from
+syv-ai/qwen38-27b-rtx3090 (https://github.com/syv-ai/qwen38-27b-rtx3090, Apache-2.0).
+
+The EXL3 format and its CUDA kernels come from exllamav3 (https://github.com/turboderp-org/exllamav3, MIT), which is a
+runtime dependency and is not included here. EXL3 is a streamlined variant of QTIP (https://github.com/Cornell-RelaxML/qtip).
+
+Developed with the support of the Department of Computer Science (Departament de Ciències de la Computació),
+Universitat Autònoma de Barcelona, which provided the hardware used to build and measure it.
+
+This repository contains no model weights.

vllm_quantpack-0.1.0/PKG-INFO

@@ -0,0 +1,73 @@
+Metadata-Version: 2.4
+Name: vllm-quantpack
+Version: 0.1.0
+Summary: vLLM plugin: serve EXL3 (exllamav3 trellis) checkpoints and the model-family fixes they need, with no patch to vLLM
+Author: Jorge Ramírez Espejo
+License-Expression: Apache-2.0
+Project-URL: Repository, https://github.com/jramirez-uab/vllm-quantpack
+Classifier: Development Status :: 3 - Alpha
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Dynamic: license-file
+
+# vllm-quantpack
+
+**Experimental, early stage.** A vLLM general plugin that serves EXL3 (exllamav3 trellis) checkpoints, and the
+model-family fixes they need, in stock vLLM: no patch to vLLM, no per-model plugin.
+
+## What it does
+
+- Registers `quant_method: "exl3"`, so `vllm serve <repo>` loads EXL3 packs with no extra flag:
+  - linear layers at any per-tensor bitrate, including fused layers whose shards have different bitrates;
+  - `lm_head` as an EXL3 tensor (`head_bits`) or dense;
+  - `embed_tokens` as int8 rows with a per-row scale, dequantized on the gathered rows;
+  - routed MoE experts on exllamav3's fused kernel, capturable in CUDA graphs;
+  - tensor parallelism for dense layers (`-tp N`) and expert parallelism for MoE (`-tp N --enable-expert-parallel`).
+- Model-family fixes:
+  - **Qwen 3.5 / 3.6 / 3.8**: the vision tower's duplicated attention tensors;
+  - **Qwen3.8-Flash-Next**: the FP8 PLE lookup on GPUs below sm_89.
+
+## Requirements
+
+- vLLM 0.31 (0.30 also works).
+- The exllamav3 wheel that matches your torch and CUDA. It is not on PyPI:
+  ```
+  python -c "import torch; print(torch.__version__, torch.version.cuda)"
+  # then pick the matching asset from https://github.com/turboderp-org/exllamav3/releases
+  ```
+- An NVIDIA GPU of compute capability 8.0 or newer (Ampere, Ada, Hopper, Blackwell).
+
+## Install
+
+```
+pip install vllm-quantpack
+```
+
+It replaces OrcaSAQ2, `qwen38_exl3_vision`, `flint_vllm` and `spark_vllm`. If any of them is installed in the same
+environment, uninstall it, or load only this plugin with `VLLM_PLUGINS=quantpack`.
+
+## Status
+
+| Part | State |
+|---|---|
+| EXL3 linear, int8 embedding, lm_head | validated: same scores as the previous stack (dense Qwen 27B) |
+| EXL3 MoE | validated on one GPU (Groot 19B-A3B): same as the previous stack within run-to-run noise |
+| Tensor / expert parallelism | validated with MoE TP2 + EP (Groot 19B-A3B); Flash-Next TP not tested yet |
+| Qwen vision, Flash-Next PLE | validated (Kestrel, Groot, Flash-Next 2.25) |
+| Flint architecture | validated (Flint 3.88): same as flint_vllm within run-to-run noise |
+| GPTQ with int4/int8 embeddings (Qwen3.5 family, vLLM <= 0.31) | validated: loads on unpatched vLLM 0.31, bit-identical to the patched vLLM in eager mode |
+| Spark architecture | served (Spark-X2.5 4B) |
+| Nemotron-3.5-Super-VL (gateless ReLU² MoE, MTP, vision) | served (2.25 bpw, one 96 GB GPU) |
+| Every published JRamirez-UAB checkpoint | each served with its card's Quick start command on a clean vLLM 0.31 install: text, image and tool call answered |
+
+## Credits
+
+Includes code from [OrcaSAQ2](https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel) (Apache-2.0). The EXL3 format and
+kernels are [exllamav3](https://github.com/turboderp-org/exllamav3) by turboderp (MIT), a streamlined variant of
+[QTIP](https://github.com/Cornell-RelaxML/qtip). See NOTICE.
+
+## License
+
+Apache-2.0.

vllm_quantpack-0.1.0/README.md

@@ -0,0 +1,59 @@
+# vllm-quantpack
+
+**Experimental, early stage.** A vLLM general plugin that serves EXL3 (exllamav3 trellis) checkpoints, and the
+model-family fixes they need, in stock vLLM: no patch to vLLM, no per-model plugin.
+
+## What it does
+
+- Registers `quant_method: "exl3"`, so `vllm serve <repo>` loads EXL3 packs with no extra flag:
+  - linear layers at any per-tensor bitrate, including fused layers whose shards have different bitrates;
+  - `lm_head` as an EXL3 tensor (`head_bits`) or dense;
+  - `embed_tokens` as int8 rows with a per-row scale, dequantized on the gathered rows;
+  - routed MoE experts on exllamav3's fused kernel, capturable in CUDA graphs;
+  - tensor parallelism for dense layers (`-tp N`) and expert parallelism for MoE (`-tp N --enable-expert-parallel`).
+- Model-family fixes:
+  - **Qwen 3.5 / 3.6 / 3.8**: the vision tower's duplicated attention tensors;
+  - **Qwen3.8-Flash-Next**: the FP8 PLE lookup on GPUs below sm_89.
+
+## Requirements
+
+- vLLM 0.31 (0.30 also works).
+- The exllamav3 wheel that matches your torch and CUDA. It is not on PyPI:
+  ```
+  python -c "import torch; print(torch.__version__, torch.version.cuda)"
+  # then pick the matching asset from https://github.com/turboderp-org/exllamav3/releases
+  ```
+- An NVIDIA GPU of compute capability 8.0 or newer (Ampere, Ada, Hopper, Blackwell).
+
+## Install
+
+```
+pip install vllm-quantpack
+```
+
+It replaces OrcaSAQ2, `qwen38_exl3_vision`, `flint_vllm` and `spark_vllm`. If any of them is installed in the same
+environment, uninstall it, or load only this plugin with `VLLM_PLUGINS=quantpack`.
+
+## Status
+
+| Part | State |
+|---|---|
+| EXL3 linear, int8 embedding, lm_head | validated: same scores as the previous stack (dense Qwen 27B) |
+| EXL3 MoE | validated on one GPU (Groot 19B-A3B): same as the previous stack within run-to-run noise |
+| Tensor / expert parallelism | validated with MoE TP2 + EP (Groot 19B-A3B); Flash-Next TP not tested yet |
+| Qwen vision, Flash-Next PLE | validated (Kestrel, Groot, Flash-Next 2.25) |
+| Flint architecture | validated (Flint 3.88): same as flint_vllm within run-to-run noise |
+| GPTQ with int4/int8 embeddings (Qwen3.5 family, vLLM <= 0.31) | validated: loads on unpatched vLLM 0.31, bit-identical to the patched vLLM in eager mode |
+| Spark architecture | served (Spark-X2.5 4B) |
+| Nemotron-3.5-Super-VL (gateless ReLU² MoE, MTP, vision) | served (2.25 bpw, one 96 GB GPU) |
+| Every published JRamirez-UAB checkpoint | each served with its card's Quick start command on a clean vLLM 0.31 install: text, image and tool call answered |
+
+## Credits
+
+Includes code from [OrcaSAQ2](https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel) (Apache-2.0). The EXL3 format and
+kernels are [exllamav3](https://github.com/turboderp-org/exllamav3) by turboderp (MIT), a streamlined variant of
+[QTIP](https://github.com/Cornell-RelaxML/qtip). See NOTICE.
+
+## License
+
+Apache-2.0.

vllm_quantpack-0.1.0/pyproject.toml

@@ -0,0 +1,25 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "vllm-quantpack"
+version = "0.1.0"
+description = "vLLM plugin: serve EXL3 (exllamav3 trellis) checkpoints and the model-family fixes they need, with no patch to vLLM"
+requires-python = ">=3.10"
+license = "Apache-2.0"
+readme = "README.md"
+authors = [{ name = "Jorge Ramírez Espejo" }]
+classifiers = ["Development Status :: 3 - Alpha"]
+# vLLM, torch and exllamav3 are deliberately not pinned: exllamav3's wheel is ABI-linked to torch and the CUDA major
+# version and is published as a GitHub release asset, not on PyPI. See README.
+dependencies = []
+
+[project.urls]
+Repository = "https://github.com/jramirez-uab/vllm-quantpack"
+
+[project.entry-points."vllm.general_plugins"]
+quantpack = "vllm_quantpack:register"
+
+[tool.setuptools.packages.find]
+include = ["vllm_quantpack*"]

vllm_quantpack-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

vllm_quantpack-0.1.0/tests/test_pure.py

@@ -0,0 +1,125 @@
+"""Pure helpers: shard slots, prefix handling, K, tensor-parallel ranges. No GPU, no vLLM."""
+import pytest
+
+from vllm_quantpack.exl3.names import expand, manifest_prefixes, norm_prefix, slot, trellis_k
+from vllm_quantpack.exl3.tp_ranges import col_ranges, logical_parts, on_block, row_range
+
+
+def test_slot_maps_every_loader_shard_id_form():
+    assert slot(None) == 0
+    assert slot(1) == 1
+    assert slot((0, 1, 2)) == 0
+    assert [slot(s) for s in ("q", "k", "v")] == [0, 1, 2]
+    with pytest.raises(ValueError):
+        slot("x")
+
+
+def test_expand_returns_hf_names_of_fused_layers():
+    assert expand("model.layers.3.self_attn.qkv_proj") == [
+        "model.layers.3.self_attn.q_proj", "model.layers.3.self_attn.k_proj", "model.layers.3.self_attn.v_proj"]
+    assert expand("model.layers.3.linear_attn.in_proj_qkvz") == [
+        "model.layers.3.linear_attn.in_proj_qkv", "model.layers.3.linear_attn.in_proj_z"]
+    assert expand("model.layers.3.mlp.down_proj") == ["model.layers.3.mlp.down_proj"]
+
+
+def test_norm_prefix_strips_one_tree_segment():
+    assert norm_prefix("language_model.model.layers.0.mlp.down_proj") == "layers.0.mlp.down_proj"
+    assert norm_prefix("model.layers.0.mlp.down_proj") == "layers.0.mlp.down_proj"
+    assert norm_prefix("backbone.layers.0.mixer.in_proj") == "layers.0.mixer.in_proj"
+    assert norm_prefix("visual.blocks.0.mlp.fc1") == "visual.blocks.0.mlp.fc1"
+
+
+def test_manifest_keeps_only_modules_with_a_trellis():
+    ts = {
+        "model.layers.0.mlp.down_proj": {"stored_tensors": {"model.layers.0.mlp.down_proj.trellis": {}}},
+        "model.layers.0.linear_attn.in_proj_a": {"stored_tensors": {"model.layers.0.linear_attn.in_proj_a.weight": {}}},
+        "backbone.layers.1.mixer.in_proj": {"stored_tensors": {"backbone.layers.1.mixer.in_proj.trellis": {}}},
+    }
+    assert manifest_prefixes(ts) == {"layers.0.mlp.down_proj", "layers.1.mixer.in_proj"}
+    assert manifest_prefixes({}) == set()
+
+
+def test_trellis_k_keeps_mul1_half_rates():
+    assert trellis_k(64) == 4 and isinstance(trellis_k(64), int)
+    assert trellis_k(56) == 3.5
+
+
+def test_logical_parts():
+    assert logical_parts(None, 3) == [0, 1, 2]
+    assert logical_parts(1, 2) == [1]
+    assert logical_parts((0, 1, 2), 4) == [0, 1, 2]
+    assert logical_parts("k", 3) == [1]
+
+
+def test_col_ranges_single_partition():
+    # gate_proj of 17408 outputs, tp=2: each rank keeps half
+    assert col_ranges(17408, [0], [8704, 8704], 0, 2) == [(0, 8704)]
+    assert col_ranges(17408, [0], [8704, 8704], 1, 2) == [(8704, 17408)]
+
+
+def test_col_ranges_tensor_holding_several_partitions():
+    # in_proj_qkv = q|k|v of 2048 each, tp=2: rank 1 keeps [q_1, k_1, v_1]
+    assert col_ranges(6144, [0, 1, 2], [1024, 1024, 1024, 4096], 1, 2) == [(1024, 2048), (3072, 4096), (5120, 6144)]
+
+
+def test_col_ranges_replicated_kv_heads():
+    # k_proj with 2 KV heads of 256 on tp=4: ranks 0,1 share head 0 and ranks 2,3 share head 1
+    local = [1536, 256, 256]
+    assert [col_ranges(512, [1], local, r, 4) for r in range(4)] == [[(0, 256)], [(0, 256)], [(256, 512)], [(256, 512)]]
+
+
+def test_col_ranges_padded_single_layer():
+    # Groot vision mlp.linear_fc1: 4304 outputs stored padded to 4352, tp=2; the padding columns belong to no rank
+    assert [col_ranges(4352, [0], [2152], r, 2) for r in range(2)] == [[(0, 2152)], [(2152, 4304)]]
+    with pytest.raises(NotImplementedError):     # 128 extra columns or more is not padding
+        col_ranges(4432, [0], [2152], 0, 2)
+
+
+def test_col_ranges_rejects_a_split_that_does_not_add_up():
+    with pytest.raises(NotImplementedError):
+        col_ranges(6000, [0, 1, 2], [1024, 1024, 1024], 0, 2)
+
+
+def test_row_range_and_block_alignment():
+    assert row_range(1, 2560) == (2560, 5120)
+    assert on_block([(0, 2560), (2560, 5120)])
+    assert not on_block([(320, 640)])
+
+
+def test_flint_interm_div_targets_only_late_layers_of_flint():
+    from vllm_quantpack.families.flint.interm_div import kind
+    assert kind("model.layers.46.mlp.shared_expert.gate_up_proj", "flint") == "shared_expert.gate_up_proj"
+    assert kind("model.layers.48.mlp.shared_expert.down_proj", "flint") == "shared_expert.down_proj"
+    assert kind("model.layers.47.mlp.experts", "flint") == "experts"
+    assert kind("model.layers.47.mlp.experts.routed", "flint") == "experts"     # FusedMoEFactory child
+    assert kind("model.layers.45.mlp.experts", "flint") is None
+    assert kind("model.layers.46.self_attn.qkv_proj", "flint") is None
+    assert kind("model.layers.46.mlp.experts", "qwen3_5_moe") is None
+
+
+def test_embedding_prefix_is_claimed_once_inside_its_context():
+    from vllm_quantpack.exl3.layers import embedding_prefix, take_embedding_prefix
+    assert take_embedding_prefix() is None
+    with embedding_prefix("model.language_model.embed_tokens"):
+        assert take_embedding_prefix() == "model.language_model.embed_tokens"
+        assert take_embedding_prefix() is None          # only the model's own embed_tokens, not a later embedding
+    assert take_embedding_prefix() is None
+
+
+def test_gateless_moe_is_detected_from_the_stored_projections():
+    from vllm_quantpack.exl3.names import gateless_up_key
+    gated = {(e, p, "trellis"): 0 for e in range(2) for p in ("gate_proj", "up_proj", "down_proj")}
+    relu2 = {(e, p, "trellis"): 0 for e in range(2) for p in ("up_proj", "down_proj")}
+    drafter = {(e, p, "trellis"): 0 for e in range(2) for p in ("gate_proj", "down_proj")}   # up loaded into w1
+    assert gateless_up_key(gated, range(2)) is None
+    assert gateless_up_key(relu2, range(2)) == "up_proj"
+    assert gateless_up_key(drafter, range(2)) == "gate_proj"
+
+
+def test_expert_store_key_maps_fused_parameter_names():
+    from vllm_quantpack.exl3.names import expert_store_key
+    assert expert_store_key("mtp.layers.0.mixer.experts.w13_trellis", "w1", 5) == (5, "gate_proj", "trellis")
+    assert expert_store_key("experts.w13_suh", "w3", "7") == (7, "up_proj", "suh")
+    assert expert_store_key("experts.w2_svh", "w2", 0) == (0, "down_proj", "svh")
+    assert expert_store_key("experts.w2_weight", "w2", 0) is None
+    assert expert_store_key("experts.e_score_correction_bias", None, 0) is None

vllm_quantpack-0.1.0/vllm_quantpack/__init__.py

@@ -0,0 +1,61 @@
+"""vllm-quantpack: serve EXL3 (exllamav3 trellis) checkpoints, and the model-family fixes they need, in stock vLLM.
+
+Installed as a vLLM general plugin (entry point `vllm.general_plugins`), so `vllm serve <repo>` picks up any checkpoint
+whose config says `quant_method: "exl3"` with no flag and no patch to vLLM.
+
+Layout:
+  exl3/      quantization config, linear / embedding / MoE methods, the custom ops, tensor-parallel cuts
+  families/  fixes for specific model families (Qwen vision and Flash-Next PLE) and models vLLM lacks or
+             maps incompletely (Flint, Spark, Nemotron-3.5-Super-VL)
+
+It replaces OrcaSAQ2 (whose code it includes, Apache-2.0), qwen38_exl3_vision, flint_vllm and spark_vllm, which must
+not be installed alongside.
+"""
+__version__ = "0.1.0"
+
+# Entry-point names of the plugins this package replaces.
+_CONFLICTS = ("orcasaq2", "qwen38_exl3_vision", "tarninfer_exl3", "flint_vllm", "spark_vllm")
+
+
+def _conflicts():
+    """Replaced plugins that vLLM will also load: installed, and not left out by VLLM_PLUGINS."""
+    import os
+    from importlib.metadata import entry_points
+
+    allowed = os.environ.get("VLLM_PLUGINS")
+    allowed = None if allowed is None else {p.strip() for p in allowed.split(",")}
+    return sorted({ep.name for ep in entry_points(group="vllm.general_plugins")
+                   if ep.name in _CONFLICTS and (allowed is None or ep.name in allowed)})
+
+
+def register():
+    from vllm.logger import init_logger
+    from vllm.model_executor.layers.quantization import register_quantization_config
+
+    logger = init_logger("vllm.quantpack")
+    found = _conflicts()
+    if found:
+        # They register the same quant_method and patch the same classes; which one wins would depend on plugin order.
+        raise RuntimeError(f"vllm-quantpack replaces the vLLM plugin(s) {', '.join(found)}: uninstall them from this "
+                           "environment, or leave them out with VLLM_PLUGINS=quantpack")
+
+    from .exl3.layers import patch_embedding_layers
+    from .exl3.linear import Exl3Config
+
+    try:
+        register_quantization_config("exl3")(Exl3Config)
+    except ValueError:
+        return  # already registered in this process
+    patch_embedding_layers(logger)
+
+    from .exl3.spec_decode import patch_should_share
+    from .families import flint, nemotron, qwen, spark
+    patch_should_share(logger)
+    qwen.register(logger)
+    if flint.register(logger):
+        logger.info("vllm-quantpack: Flint model registered")
+    if spark.register(logger):
+        logger.info("vllm-quantpack: Spark model registered")
+    if nemotron.register(logger):
+        logger.info("vllm-quantpack: Nemotron-3.5-Super-VL model registered")
+    logger.info("vllm-quantpack %s: quant_method 'exl3' is loadable (linear, int8 embedding, MoE, TP)", __version__)