Defuser 0.0.26__tar.gz → 0.0.28__tar.gz

{defuser-0.0.26 → defuser-0.0.28}/Defuser.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: Defuser
-Version: 0.0.26
+Version: 0.0.28
 Summary: Model defuser helper for HF Transformers.
 Author-email: ModelCloud <qubitium@modelcloud.ai>
 License-Expression: Apache-2.0
@@ -92,8 +92,8 @@ Defuser currently supports the following `transformers>=5.3.0` `model_type` valu
 
 | Pattern | Supported model types                                                                                                                                                                                                               | Defused op performed ⚙️ |
 | --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| --- |
-| Standard routed expert tensors 🧱 | `deepseek_v2`, `dots1`, `ernie4_5_moe`, `ernie4_5_vl_moe`, `exaone_moe`, `flex_olmo`, `glm4_moe_lite`, `glm4v_moe`, `hunyuan_v1_moe`, `jamba`, `laguna`, `lfm2_moe`, `minimax`, `minimax_m2`, `olmoe`, `qwen3_vl_moe`, `solar_open`, `solar_open2` | Splits fused expert tensors or registered expert buffers into numbered expert `nn.Linear` modules with per-expert `gate_proj`, `up_proj`, and `down_proj`. |
-| Mixed sparse and shared experts | `deepseek_v3`, `deepseek_v32`, `deepseek_v4`, `glm_moe_dsa`, `qwen3_5_moe`, `qwen3_5_moe_text`, `qwen4_exp`, `qwen4_exp_text`                                                                                                                           | Runtime expert tensor defusion for routed experts while preserving the model's shared-expert path. |
+| Standard routed expert tensors 🧱 | `deepseek_v2`, `diffusion_gemma`, `dots1`, `ernie4_5_moe`, `ernie4_5_vl_moe`, `exaone_moe`, `flex_olmo`, `glm4_moe_lite`, `glm4v_moe`, `hunyuan_v1_moe`, `jamba`, `laguna`, `lfm2_moe`, `minimax`, `minimax_m2`, `olmoe`, `qwen3_vl_moe`, `solar_open`, `solar_open2` | Splits fused expert tensors or registered expert buffers into numbered expert `nn.Linear` modules with per-expert `gate_proj`, `up_proj`, and `down_proj`. |
+| Mixed sparse and shared experts | `deepseek_v3`, `deepseek_v32`, `deepseek_v4`, `glm5_next`, `glm_moe_dsa`, `qwen3_5_moe`, `qwen3_5_moe_text`, `qwen4_exp`, `qwen4_exp_text`                                                                                                              | Runtime expert tensor defusion for routed experts while preserving the model's shared-expert path. |
 | Transposed or packed expert tensors | `gpt_oss`, `phimoe`                                                                                                                                                                                                                 | Splits transposed fused expert `gate_up_proj` tensors into per-expert `gate_proj` + `up_proj`, preserves expert bias when present, and converts expert tensors into numbered expert `nn.Linear` modules. |
 | Flattened expert layout | `dbrx`                                                                                                                                                                                                                              | Rebuilds the flattened DBRX expert FFN weights into numbered expert `gate_proj`, `up_proj`, and `down_proj` `nn.Linear` modules. |
 | Batched expert-input execution | `llama4`                                                                                                                                                                                                                            | Runtime expert tensor defusion plus preservation of the llama4 batched expert-input execution contract. |

{defuser-0.0.26 → defuser-0.0.28}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: Defuser
-Version: 0.0.26
+Version: 0.0.28
 Summary: Model defuser helper for HF Transformers.
 Author-email: ModelCloud <qubitium@modelcloud.ai>
 License-Expression: Apache-2.0
@@ -92,8 +92,8 @@ Defuser currently supports the following `transformers>=5.3.0` `model_type` valu
 
 | Pattern | Supported model types                                                                                                                                                                                                               | Defused op performed ⚙️ |
 | --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| --- |
-| Standard routed expert tensors 🧱 | `deepseek_v2`, `dots1`, `ernie4_5_moe`, `ernie4_5_vl_moe`, `exaone_moe`, `flex_olmo`, `glm4_moe_lite`, `glm4v_moe`, `hunyuan_v1_moe`, `jamba`, `laguna`, `lfm2_moe`, `minimax`, `minimax_m2`, `olmoe`, `qwen3_vl_moe`, `solar_open`, `solar_open2` | Splits fused expert tensors or registered expert buffers into numbered expert `nn.Linear` modules with per-expert `gate_proj`, `up_proj`, and `down_proj`. |
-| Mixed sparse and shared experts | `deepseek_v3`, `deepseek_v32`, `deepseek_v4`, `glm_moe_dsa`, `qwen3_5_moe`, `qwen3_5_moe_text`, `qwen4_exp`, `qwen4_exp_text`                                                                                                                           | Runtime expert tensor defusion for routed experts while preserving the model's shared-expert path. |
+| Standard routed expert tensors 🧱 | `deepseek_v2`, `diffusion_gemma`, `dots1`, `ernie4_5_moe`, `ernie4_5_vl_moe`, `exaone_moe`, `flex_olmo`, `glm4_moe_lite`, `glm4v_moe`, `hunyuan_v1_moe`, `jamba`, `laguna`, `lfm2_moe`, `minimax`, `minimax_m2`, `olmoe`, `qwen3_vl_moe`, `solar_open`, `solar_open2` | Splits fused expert tensors or registered expert buffers into numbered expert `nn.Linear` modules with per-expert `gate_proj`, `up_proj`, and `down_proj`. |
+| Mixed sparse and shared experts | `deepseek_v3`, `deepseek_v32`, `deepseek_v4`, `glm5_next`, `glm_moe_dsa`, `qwen3_5_moe`, `qwen3_5_moe_text`, `qwen4_exp`, `qwen4_exp_text`                                                                                                              | Runtime expert tensor defusion for routed experts while preserving the model's shared-expert path. |
 | Transposed or packed expert tensors | `gpt_oss`, `phimoe`                                                                                                                                                                                                                 | Splits transposed fused expert `gate_up_proj` tensors into per-expert `gate_proj` + `up_proj`, preserves expert bias when present, and converts expert tensors into numbered expert `nn.Linear` modules. |
 | Flattened expert layout | `dbrx`                                                                                                                                                                                                                              | Rebuilds the flattened DBRX expert FFN weights into numbered expert `gate_proj`, `up_proj`, and `down_proj` `nn.Linear` modules. |
 | Batched expert-input execution | `llama4`                                                                                                                                                                                                                            | Runtime expert tensor defusion plus preservation of the llama4 batched expert-input execution contract. |

{defuser-0.0.26 → defuser-0.0.28}/README.md

@@ -66,8 +66,8 @@ Defuser currently supports the following `transformers>=5.3.0` `model_type` valu
 
 | Pattern | Supported model types                                                                                                                                                                                                               | Defused op performed ⚙️ |
 | --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| --- |
-| Standard routed expert tensors 🧱 | `deepseek_v2`, `dots1`, `ernie4_5_moe`, `ernie4_5_vl_moe`, `exaone_moe`, `flex_olmo`, `glm4_moe_lite`, `glm4v_moe`, `hunyuan_v1_moe`, `jamba`, `laguna`, `lfm2_moe`, `minimax`, `minimax_m2`, `olmoe`, `qwen3_vl_moe`, `solar_open`, `solar_open2` | Splits fused expert tensors or registered expert buffers into numbered expert `nn.Linear` modules with per-expert `gate_proj`, `up_proj`, and `down_proj`. |
-| Mixed sparse and shared experts | `deepseek_v3`, `deepseek_v32`, `deepseek_v4`, `glm_moe_dsa`, `qwen3_5_moe`, `qwen3_5_moe_text`, `qwen4_exp`, `qwen4_exp_text`                                                                                                                           | Runtime expert tensor defusion for routed experts while preserving the model's shared-expert path. |
+| Standard routed expert tensors 🧱 | `deepseek_v2`, `diffusion_gemma`, `dots1`, `ernie4_5_moe`, `ernie4_5_vl_moe`, `exaone_moe`, `flex_olmo`, `glm4_moe_lite`, `glm4v_moe`, `hunyuan_v1_moe`, `jamba`, `laguna`, `lfm2_moe`, `minimax`, `minimax_m2`, `olmoe`, `qwen3_vl_moe`, `solar_open`, `solar_open2` | Splits fused expert tensors or registered expert buffers into numbered expert `nn.Linear` modules with per-expert `gate_proj`, `up_proj`, and `down_proj`. |
+| Mixed sparse and shared experts | `deepseek_v3`, `deepseek_v32`, `deepseek_v4`, `glm5_next`, `glm_moe_dsa`, `qwen3_5_moe`, `qwen3_5_moe_text`, `qwen4_exp`, `qwen4_exp_text`                                                                                                              | Runtime expert tensor defusion for routed experts while preserving the model's shared-expert path. |
 | Transposed or packed expert tensors | `gpt_oss`, `phimoe`                                                                                                                                                                                                                 | Splits transposed fused expert `gate_up_proj` tensors into per-expert `gate_proj` + `up_proj`, preserves expert bias when present, and converts expert tensors into numbered expert `nn.Linear` modules. |
 | Flattened expert layout | `dbrx`                                                                                                                                                                                                                              | Rebuilds the flattened DBRX expert FFN weights into numbered expert `gate_proj`, `up_proj`, and `down_proj` `nn.Linear` modules. |
 | Batched expert-input execution | `llama4`                                                                                                                                                                                                                            | Runtime expert tensor defusion plus preservation of the llama4 batched expert-input execution contract. |

{defuser-0.0.26 → defuser-0.0.28}/defuser/model_registry.py

@@ -46,6 +46,21 @@ MODEL_CONFIG = {
     "dia": {
         "min_transformers_version": MIN_SUPPORTED_TRANSFORMERS_VERSION,
     },
+    "diffusion_gemma": {
+        "min_transformers_version": MIN_SUPPORTED_TRANSFORMERS_VERSION,
+        PATCH.EXPERTS_DEFUSE: [
+            {
+                # DiffusionGemma's fused experts need an instance-level
+                # implementation after defusion so their per-expert Linear
+                # leaves remain compatible with the upstream routing call.
+                "module_class": (
+                    "transformers.models.diffusion_gemma.modeling_diffusion_gemma."
+                    "DiffusionGemmaTextExperts"
+                ),
+                "forward_impl": "linear_loop",
+            }
+        ],
+    },
     "dots1": {
         "min_transformers_version": MIN_SUPPORTED_TRANSFORMERS_VERSION,
     },
@@ -199,6 +214,15 @@ MODEL_CONFIG = {
     "glm_image": {
         "min_transformers_version": MIN_SUPPORTED_TRANSFORMERS_VERSION,
     },
+    "glm5_next": {
+        "min_transformers_version": MIN_SUPPORTED_TRANSFORMERS_VERSION,
+        PATCH.EXPERTS_DEFUSE: [
+            {
+                "module_class": "transformers.models.glm5_next.modeling_glm5_next.Glm5NextTextExperts",
+                "forward_impl": "linear_loop",
+            }
+        ],
+    },
     "glm_moe_dsa": {
         "min_transformers_version": MIN_SUPPORTED_TRANSFORMERS_VERSION,
     },

{defuser-0.0.26 → defuser-0.0.28}/defuser/modeling/moe_experts_interface.py

@@ -282,6 +282,10 @@ def _matching_experts_defuse_spec(module: nn.Module, specs: list[dict]) -> dict
 
 def _install_instance_forward(module: nn.Module, implementation: str) -> None:
     """Attach a generic forward implementation directly to one experts module."""
+    if implementation == LINEAR_LOOP_IMPL:
+        module.forward = MethodType(linear_loop_experts_forward, module)
+        return
+
     if implementation == BATCHED_INPUT_IMPL:
         module.forward = MethodType(batched_input_experts_forward, module)
         return

{defuser-0.0.26 → defuser-0.0.28}/pyproject.toml

@@ -9,7 +9,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "Defuser"
-version = "0.0.26"
+version = "0.0.28"
 description = "Model defuser helper for HF Transformers."
 readme = "README.md"
 requires-python = ">=3.9"

{defuser-0.0.26 → defuser-0.0.28}/tests/test_candidate_coverage.py

@@ -177,6 +177,24 @@ STANDARD_MOE_CASES = [
         "route_indices": [[0], [1], [2], [3], [0]],
         "expert_attrs": ("gate_proj", "up_proj", "down_proj"),
     },
+    {
+        "model_type": "glm5_next",
+        "module_path": "transformers.models.glm5_next.modeling_glm5_next",
+        "class_name": "Glm5NextTextExperts",
+        "config_module": "transformers.models.glm5_next.configuration_glm5_next",
+        "config_name": "Glm5NextTextConfig",
+        "config_updates": {
+            "hidden_size": 64,
+            "moe_intermediate_size": 32,
+            "n_routed_experts": 4,
+            "hidden_act": "silu",
+            "swiglu_limit": 1.0,
+            "_experts_implementation": "grouped_mm",
+        },
+        "input_dim": 64,
+        "route_indices": [[0], [1], [2], [3], [0]],
+        "expert_attrs": ("gate_proj", "up_proj", "down_proj"),
+    },
     {
         "model_type": "dots1",
         "module_path": "transformers.models.dots1.modeling_dots1",
@@ -576,6 +594,7 @@ ALL_CANDIDATE_MODEL_TYPES = {
     "glm4_moe_lite",
     "glm4v",
     "glm4v_moe",
+    "glm5_next",
     "glm_image",
     "glm_moe_dsa",
     "glm_ocr",

{defuser-0.0.26 → defuser-0.0.28}/tests/test_meta_model_defusion.py

@@ -138,7 +138,49 @@ def _mutate_common_config_tree(config, visited: set[int] | None = None) -> None:
 
 def _build_model_config(case: dict):
     """Construct a small config tree for one registered public model type."""
-    config = _load(case["config_module"], case["config_class"])()
+    config_cls = _load(case["config_module"], case["config_class"])
+    if case["model_type"] == "diffusion_gemma":
+        # Unlike the generic cases, DiffusionGemma validates its nested text /
+        # vision configs during construction; provide a minimal valid pair so
+        # this registry test remains a meta-only structural check.
+        return config_cls(
+            text_config={
+                "vocab_size": 128,
+                "hidden_size": 64,
+                "intermediate_size": 128,
+                "moe_intermediate_size": 32,
+                "num_hidden_layers": 2,
+                "num_attention_heads": 4,
+                "num_key_value_heads": 1,
+                "num_global_key_value_heads": 1,
+                "head_dim": 16,
+                "global_head_dim": 16,
+                "num_experts": 4,
+                "top_k_experts": 1,
+                "max_position_embeddings": 64,
+                "sliding_window": 8,
+                "layer_types": ["sliding_attention", "full_attention"],
+                "use_bidirectional_attention": "vision",
+            },
+            vision_config={
+                "hidden_size": 64,
+                "intermediate_size": 128,
+                "num_hidden_layers": 1,
+                "num_attention_heads": 4,
+                "num_key_value_heads": 4,
+                "head_dim": 16,
+                "max_position_embeddings": 64,
+                "patch_size": 4,
+                "pooling_kernel_size": 1,
+                "position_embedding_size": 16,
+            },
+            canvas_length=4,
+            image_token_id=124,
+            boi_token_id=125,
+            eoi_token_id=126,
+        )
+
+    config = config_cls()
     _mutate_common_config_tree(config)
 
     model_type = case["model_type"]
@@ -166,6 +208,8 @@ def _build_model_config(case: dict):
         }
     elif model_type == "glm4_moe":
         config.first_k_dense_replace = -1
+    elif model_type == "glm5_next":
+        config.text_config.mlp_layer_types = ["sparse"] * config.text_config.num_hidden_layers
     elif model_type == "glm_moe_dsa":
         config.mlp_layer_types = ["sparse"] * config.num_hidden_layers
     elif model_type == "granitemoehybrid":
@@ -388,6 +432,21 @@ META_MODEL_CASES = [
         "target_class_paths": ("transformers.models.dots1.modeling_dots1.Dots1NaiveMoe",),
         "validator": "experts",
     },
+    {
+        "model_type": "diffusion_gemma",
+        "mode": "convert",
+        "model_module": "transformers.models.diffusion_gemma.modeling_diffusion_gemma",
+        "model_class": "DiffusionGemmaForBlockDiffusion",
+        "config_module": "transformers.models.diffusion_gemma.configuration_diffusion_gemma",
+        "config_class": "DiffusionGemmaConfig",
+        "target_class_paths": (
+            "transformers.models.diffusion_gemma.modeling_diffusion_gemma.DiffusionGemmaTextExperts",
+        ),
+        "validator": "experts",
+        # There are separate experts blocks in the encoder and decoder; seeing
+        # both ensures the declarative class-path patch is applied everywhere.
+        "min_targets": 2,
+    },
     {
         "model_type": "ernie4_5_moe",
         "mode": "convert",
@@ -500,6 +559,19 @@ META_MODEL_CASES = [
         "target_class_paths": ("transformers.models.glm_image.modeling_glm_image.GlmImageTextMLP",),
         "validator": "dense_split",
     },
+    {
+        "model_type": "glm5_next",
+        "mode": "convert",
+        "model_module": "transformers.models.glm5_next.modeling_glm5_next",
+        "model_class": "Glm5NextForConditionalGeneration",
+        "config_module": "transformers.models.glm5_next.configuration_glm5_next",
+        "config_class": "Glm5NextConfig",
+        "target_class_paths": (
+            "transformers.models.glm5_next.modeling_glm5_next.Glm5NextTextExperts",
+        ),
+        "validator": "experts",
+        "min_targets": 2,
+    },
     {
         "model_type": "glm_moe_dsa",
         "mode": "convert",