Defuser 0.0.26__tar.gz → 0.0.27__tar.gz

{defuser-0.0.26 → defuser-0.0.27}/Defuser.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: Defuser
-Version: 0.0.26
+Version: 0.0.27
 Summary: Model defuser helper for HF Transformers.
 Author-email: ModelCloud <qubitium@modelcloud.ai>
 License-Expression: Apache-2.0
@@ -93,7 +93,7 @@ Defuser currently supports the following `transformers>=5.3.0` `model_type` valu
 | Pattern | Supported model types                                                                                                                                                                                                               | Defused op performed ⚙️ |
 | --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| --- |
 | Standard routed expert tensors 🧱 | `deepseek_v2`, `dots1`, `ernie4_5_moe`, `ernie4_5_vl_moe`, `exaone_moe`, `flex_olmo`, `glm4_moe_lite`, `glm4v_moe`, `hunyuan_v1_moe`, `jamba`, `laguna`, `lfm2_moe`, `minimax`, `minimax_m2`, `olmoe`, `qwen3_vl_moe`, `solar_open`, `solar_open2` | Splits fused expert tensors or registered expert buffers into numbered expert `nn.Linear` modules with per-expert `gate_proj`, `up_proj`, and `down_proj`. |
-| Mixed sparse and shared experts | `deepseek_v3`, `deepseek_v32`, `deepseek_v4`, `glm_moe_dsa`, `qwen3_5_moe`, `qwen3_5_moe_text`, `qwen4_exp`, `qwen4_exp_text`                                                                                                                           | Runtime expert tensor defusion for routed experts while preserving the model's shared-expert path. |
+| Mixed sparse and shared experts | `deepseek_v3`, `deepseek_v32`, `deepseek_v4`, `glm5_next`, `glm_moe_dsa`, `qwen3_5_moe`, `qwen3_5_moe_text`, `qwen4_exp`, `qwen4_exp_text`                                                                                                              | Runtime expert tensor defusion for routed experts while preserving the model's shared-expert path. |
 | Transposed or packed expert tensors | `gpt_oss`, `phimoe`                                                                                                                                                                                                                 | Splits transposed fused expert `gate_up_proj` tensors into per-expert `gate_proj` + `up_proj`, preserves expert bias when present, and converts expert tensors into numbered expert `nn.Linear` modules. |
 | Flattened expert layout | `dbrx`                                                                                                                                                                                                                              | Rebuilds the flattened DBRX expert FFN weights into numbered expert `gate_proj`, `up_proj`, and `down_proj` `nn.Linear` modules. |
 | Batched expert-input execution | `llama4`                                                                                                                                                                                                                            | Runtime expert tensor defusion plus preservation of the llama4 batched expert-input execution contract. |

{defuser-0.0.26 → defuser-0.0.27}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: Defuser
-Version: 0.0.26
+Version: 0.0.27
 Summary: Model defuser helper for HF Transformers.
 Author-email: ModelCloud <qubitium@modelcloud.ai>
 License-Expression: Apache-2.0
@@ -93,7 +93,7 @@ Defuser currently supports the following `transformers>=5.3.0` `model_type` valu
 | Pattern | Supported model types                                                                                                                                                                                                               | Defused op performed ⚙️ |
 | --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| --- |
 | Standard routed expert tensors 🧱 | `deepseek_v2`, `dots1`, `ernie4_5_moe`, `ernie4_5_vl_moe`, `exaone_moe`, `flex_olmo`, `glm4_moe_lite`, `glm4v_moe`, `hunyuan_v1_moe`, `jamba`, `laguna`, `lfm2_moe`, `minimax`, `minimax_m2`, `olmoe`, `qwen3_vl_moe`, `solar_open`, `solar_open2` | Splits fused expert tensors or registered expert buffers into numbered expert `nn.Linear` modules with per-expert `gate_proj`, `up_proj`, and `down_proj`. |
-| Mixed sparse and shared experts | `deepseek_v3`, `deepseek_v32`, `deepseek_v4`, `glm_moe_dsa`, `qwen3_5_moe`, `qwen3_5_moe_text`, `qwen4_exp`, `qwen4_exp_text`                                                                                                                           | Runtime expert tensor defusion for routed experts while preserving the model's shared-expert path. |
+| Mixed sparse and shared experts | `deepseek_v3`, `deepseek_v32`, `deepseek_v4`, `glm5_next`, `glm_moe_dsa`, `qwen3_5_moe`, `qwen3_5_moe_text`, `qwen4_exp`, `qwen4_exp_text`                                                                                                              | Runtime expert tensor defusion for routed experts while preserving the model's shared-expert path. |
 | Transposed or packed expert tensors | `gpt_oss`, `phimoe`                                                                                                                                                                                                                 | Splits transposed fused expert `gate_up_proj` tensors into per-expert `gate_proj` + `up_proj`, preserves expert bias when present, and converts expert tensors into numbered expert `nn.Linear` modules. |
 | Flattened expert layout | `dbrx`                                                                                                                                                                                                                              | Rebuilds the flattened DBRX expert FFN weights into numbered expert `gate_proj`, `up_proj`, and `down_proj` `nn.Linear` modules. |
 | Batched expert-input execution | `llama4`                                                                                                                                                                                                                            | Runtime expert tensor defusion plus preservation of the llama4 batched expert-input execution contract. |

{defuser-0.0.26 → defuser-0.0.27}/README.md

@@ -67,7 +67,7 @@ Defuser currently supports the following `transformers>=5.3.0` `model_type` valu
 | Pattern | Supported model types                                                                                                                                                                                                               | Defused op performed ⚙️ |
 | --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| --- |
 | Standard routed expert tensors 🧱 | `deepseek_v2`, `dots1`, `ernie4_5_moe`, `ernie4_5_vl_moe`, `exaone_moe`, `flex_olmo`, `glm4_moe_lite`, `glm4v_moe`, `hunyuan_v1_moe`, `jamba`, `laguna`, `lfm2_moe`, `minimax`, `minimax_m2`, `olmoe`, `qwen3_vl_moe`, `solar_open`, `solar_open2` | Splits fused expert tensors or registered expert buffers into numbered expert `nn.Linear` modules with per-expert `gate_proj`, `up_proj`, and `down_proj`. |
-| Mixed sparse and shared experts | `deepseek_v3`, `deepseek_v32`, `deepseek_v4`, `glm_moe_dsa`, `qwen3_5_moe`, `qwen3_5_moe_text`, `qwen4_exp`, `qwen4_exp_text`                                                                                                                           | Runtime expert tensor defusion for routed experts while preserving the model's shared-expert path. |
+| Mixed sparse and shared experts | `deepseek_v3`, `deepseek_v32`, `deepseek_v4`, `glm5_next`, `glm_moe_dsa`, `qwen3_5_moe`, `qwen3_5_moe_text`, `qwen4_exp`, `qwen4_exp_text`                                                                                                              | Runtime expert tensor defusion for routed experts while preserving the model's shared-expert path. |
 | Transposed or packed expert tensors | `gpt_oss`, `phimoe`                                                                                                                                                                                                                 | Splits transposed fused expert `gate_up_proj` tensors into per-expert `gate_proj` + `up_proj`, preserves expert bias when present, and converts expert tensors into numbered expert `nn.Linear` modules. |
 | Flattened expert layout | `dbrx`                                                                                                                                                                                                                              | Rebuilds the flattened DBRX expert FFN weights into numbered expert `gate_proj`, `up_proj`, and `down_proj` `nn.Linear` modules. |
 | Batched expert-input execution | `llama4`                                                                                                                                                                                                                            | Runtime expert tensor defusion plus preservation of the llama4 batched expert-input execution contract. |

{defuser-0.0.26 → defuser-0.0.27}/defuser/model_registry.py

@@ -199,6 +199,15 @@ MODEL_CONFIG = {
     "glm_image": {
         "min_transformers_version": MIN_SUPPORTED_TRANSFORMERS_VERSION,
     },
+    "glm5_next": {
+        "min_transformers_version": MIN_SUPPORTED_TRANSFORMERS_VERSION,
+        PATCH.EXPERTS_DEFUSE: [
+            {
+                "module_class": "transformers.models.glm5_next.modeling_glm5_next.Glm5NextTextExperts",
+                "forward_impl": "linear_loop",
+            }
+        ],
+    },
     "glm_moe_dsa": {
         "min_transformers_version": MIN_SUPPORTED_TRANSFORMERS_VERSION,
     },

{defuser-0.0.26 → defuser-0.0.27}/defuser/modeling/moe_experts_interface.py

@@ -282,6 +282,10 @@ def _matching_experts_defuse_spec(module: nn.Module, specs: list[dict]) -> dict
 
 def _install_instance_forward(module: nn.Module, implementation: str) -> None:
     """Attach a generic forward implementation directly to one experts module."""
+    if implementation == LINEAR_LOOP_IMPL:
+        module.forward = MethodType(linear_loop_experts_forward, module)
+        return
+
     if implementation == BATCHED_INPUT_IMPL:
         module.forward = MethodType(batched_input_experts_forward, module)
         return

{defuser-0.0.26 → defuser-0.0.27}/pyproject.toml

@@ -9,7 +9,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "Defuser"
-version = "0.0.26"
+version = "0.0.27"
 description = "Model defuser helper for HF Transformers."
 readme = "README.md"
 requires-python = ">=3.9"

{defuser-0.0.26 → defuser-0.0.27}/tests/test_candidate_coverage.py

@@ -177,6 +177,24 @@ STANDARD_MOE_CASES = [
         "route_indices": [[0], [1], [2], [3], [0]],
         "expert_attrs": ("gate_proj", "up_proj", "down_proj"),
     },
+    {
+        "model_type": "glm5_next",
+        "module_path": "transformers.models.glm5_next.modeling_glm5_next",
+        "class_name": "Glm5NextTextExperts",
+        "config_module": "transformers.models.glm5_next.configuration_glm5_next",
+        "config_name": "Glm5NextTextConfig",
+        "config_updates": {
+            "hidden_size": 64,
+            "moe_intermediate_size": 32,
+            "n_routed_experts": 4,
+            "hidden_act": "silu",
+            "swiglu_limit": 1.0,
+            "_experts_implementation": "grouped_mm",
+        },
+        "input_dim": 64,
+        "route_indices": [[0], [1], [2], [3], [0]],
+        "expert_attrs": ("gate_proj", "up_proj", "down_proj"),
+    },
     {
         "model_type": "dots1",
         "module_path": "transformers.models.dots1.modeling_dots1",
@@ -576,6 +594,7 @@ ALL_CANDIDATE_MODEL_TYPES = {
     "glm4_moe_lite",
     "glm4v",
     "glm4v_moe",
+    "glm5_next",
     "glm_image",
     "glm_moe_dsa",
     "glm_ocr",

{defuser-0.0.26 → defuser-0.0.27}/tests/test_meta_model_defusion.py

@@ -166,6 +166,8 @@ def _build_model_config(case: dict):
         }
     elif model_type == "glm4_moe":
         config.first_k_dense_replace = -1
+    elif model_type == "glm5_next":
+        config.text_config.mlp_layer_types = ["sparse"] * config.text_config.num_hidden_layers
     elif model_type == "glm_moe_dsa":
         config.mlp_layer_types = ["sparse"] * config.num_hidden_layers
     elif model_type == "granitemoehybrid":
@@ -500,6 +502,19 @@ META_MODEL_CASES = [
         "target_class_paths": ("transformers.models.glm_image.modeling_glm_image.GlmImageTextMLP",),
         "validator": "dense_split",
     },
+    {
+        "model_type": "glm5_next",
+        "mode": "convert",
+        "model_module": "transformers.models.glm5_next.modeling_glm5_next",
+        "model_class": "Glm5NextForConditionalGeneration",
+        "config_module": "transformers.models.glm5_next.configuration_glm5_next",
+        "config_class": "Glm5NextConfig",
+        "target_class_paths": (
+            "transformers.models.glm5_next.modeling_glm5_next.Glm5NextTextExperts",
+        ),
+        "validator": "experts",
+        "min_targets": 2,
+    },
     {
         "model_type": "glm_moe_dsa",
         "mode": "convert",