onecomp 1.2.2__tar.gz → 1.3.0__tar.gz

{onecomp-1.2.2/onecomp.egg-info → onecomp-1.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: onecomp
-Version: 1.2.2
+Version: 1.3.0
 Summary: Python package for LLM compression
 Author: Keiji Kimura
 License: MIT License
@@ -89,6 +89,14 @@ Provides-Extra: hydra
 Requires-Dist: hydra-core; extra == "hydra"
 Provides-Extra: vllm
 Requires-Dist: vllm<0.22,>=0.10; extra == "vllm"
+Requires-Dist: conch-triton-kernels; extra == "vllm"
+Provides-Extra: gguf
+Requires-Dist: gguf>=0.10; extra == "gguf"
+Provides-Extra: llamacpp
+Requires-Dist: gguf>=0.10; extra == "llamacpp"
+Requires-Dist: llama-cpp-python>=0.3.0; extra == "llamacpp"
+Requires-Dist: sentencepiece; extra == "llamacpp"
+Requires-Dist: protobuf; extra == "llamacpp"
 Provides-Extra: eval
 Requires-Dist: hydra-core>=1.3; extra == "eval"
 Requires-Dist: omegaconf>=2.3; extra == "eval"
@@ -143,10 +151,12 @@ Full documentation is available at **[https://FujitsuResearch.github.io/OneCompr
 - **vLLM Plugin Integration**: Serve OneComp-quantized models with [vLLM](https://docs.vllm.ai/) via built-in plugins for DBF and Mixed-GPTQ quantization methods. Pair with [Open WebUI](https://github.com/open-webui/open-webui) for a ChatGPT-like chat experience on your local machine.
 - **AutoBit**: Mixed-precision quantization with ILP-based bitwidth assignment. Automatically estimates the target bitwidth from available VRAM and assigns per-layer bitwidths to minimize quantization error under the memory budget.
 - **JointQ**: Joint quantization method that optimizes weight assignments and scale parameters simultaneously for improved quantization accuracy. Supports group-wise quantization (e.g., 4-bit, groupsize=128).
+- **MDBF (Multi-Envelope Double Binary Factorization)**: A binary factorization quantizer that approximates each weight matrix as a sum of multi-path sign matrices with multi-scale FP16 envelopes, generalizing DBF and LittleBit for aggressive low-bit (sub-1-bit) compression. Supports ADMM/gradient refinement, activation-aware initialization, and a GemLite-accelerated 1-bit inference path. See the [MDBF guide](https://FujitsuResearch.github.io/OneCompression/algorithms/mdbf/) for details.
 - **Block-wise PTQ**: Post-quantization block-wise distillation that minimises intermediate-representation MSE against an FP16 teacher model at Transformer-block granularity. Includes Phase 1 (greedy per-block optimisation) and Phase 2 CBQ (cross-block sliding-window optimisation). Supports GPTQ, DBF, and OneBit quantizers.
 - **LoRA SFT Post-Process**: Fine-tune quantized models with LoRA adapters for accuracy recovery or domain-specific knowledge injection. Supports SFT loss, teacher distillation, and intermediate block alignment.
 - **Rotation Preprocessing**: SpinQuant/OstQuant-based rotation preprocessing that reduces quantization error by learning optimal rotation matrices before quantization. Rotation/scaling matrices are absorbed into model weights, with online Hadamard hooks automatically registered at load time. Supports Llama and Qwen3 architectures.
 - **Web Dashboard (HPC)**: A browser-based dashboard for launching quantization jobs, deploying models, and validating chat-based inference in HPC environments. See [dashboard/README.md](dashboard/README.md) for details.
+- **GGUF Export & Hugging Face Hub Integration**: Convert models to the GGUF v3 format (F16) for llama.cpp/Ollama with a dependency-free built-in writer, generate model cards with quantization recipes and evaluation results, and push save directories to the Hugging Face Hub. Supports Llama (SentencePiece or Llama-3-style BPE) and Qwen2 (BPE) architectures, including multi-EOS stop-token mapping. See the [GGUF Export guide](https://FujitsuResearch.github.io/OneCompression/user-guide/gguf-export/).
 - (TBD)
 
 ## 🤖 Supported Models
@@ -159,6 +169,8 @@ Other Hugging Face-compatible models may work but are currently untested.
 | 1 | Llama | TinyLlama, Llama-2, Llama-3 | ✅ Verified |
 | 2 | Qwen3 | Qwen3-0.6B ~ 32B | ✅ Verified |
 | 3 | Gemma | Gemma 2, Gemma 3, Gemma 4  | ✅ Verified |
+| 4 | [GPT-OSS](docs/user-guide/gptoss.md) | openai/gpt-oss-20b, gpt-oss-120b | ✅ Verified |
+| 5 | Qwen3.6 | Qwen3.6-27B, Qwen3.6-35B-A3B | ✅ Verified |
 
 
 > **Note:** Support for additional architectures is planned. Contributions and test reports are welcome.
@@ -379,10 +391,19 @@ uv run mkdocs serve
 
 Then open [http://127.0.0.1:8000](http://127.0.0.1:8000) in your browser.
 
+## 📓 Tutorial Notebook
+
+Interactive walkthrough in Jupyter or [Google Colab](https://colab.research.google.com/github/FujitsuResearch/OneCompression/blob/main/notebook/01_tutorial.ipynb)
+— RTN visualization, `Runner.auto_run`, and vLLM chat inference.
+
+See [`notebook/README.md`](./notebook/README.md) for local setup, or the
+[Tutorial Notebook guide](https://FujitsuResearch.github.io/OneCompression/getting-started/tutorial-notebook/) in the docs.
+
 ## 🚀 Examples
 
 | Category | Script | Description |
 |----------|--------|-------------|
+| Tutorial | [01_tutorial.ipynb](./notebook/01_tutorial.ipynb) | Interactive notebook (Jupyter / Colab) |
 | Quantization | [example_gptq.py](./example/example_gptq.py) | GPTQ quantization |
 | | [example_qep_gptq.py](./example/example_qep_gptq.py) | GPTQ + QEP (error propagation) |
 | | [example_lpcd_gptq.py](./example/example_lpcd_gptq.py) | GPTQ + QEP + LPCD quantization |
@@ -391,14 +412,25 @@ Then open [http://127.0.0.1:8000](http://127.0.0.1:8000) in your browser.
 | | [example_auto_run.py](./example/example_auto_run.py) | AutoBit with automatic VRAM estimation |
 | Calibration | [example_custom_calibration.py](./example/example_custom_calibration.py) | Custom calibration dataset with CalibrationConfig |
 | Save / Load | [example_save_load.py](./example/example_save_load.py) | Save and load quantized models |
+| | [example_reload_post_process_resave.py](./example/post_process/example_reload_post_process_resave.py) | Reload, post-process, and re-save a quantized checkpoint |
 | Rotation Preprocessing | [example_llama_preprocess_rtn.py](./example/pre_process/example_llama_preprocess_rtn.py) | Rotation preprocessing + RTN (TinyLlama) |
 | | [example_preprocess_save_load.py](./example/pre_process/example_preprocess_save_load.py) | Save and load rotation-preprocessed quantized models |
-| Post-Process | [example_blockwise_ptq.py](./example/post_process/example_blockwise_ptq.py) | Block-wise PTQ (GPTQ + Phase 1 & CBQ) |
+| Post-Process | [example_blockwise_ptq.py](./example/post_process/example_blockwise_ptq.py) | Block-wise PTQ via `Runner.run()` with packed buffers by default |
+| | [example_blockwise_global_ptq.py](./example/post_process/example_blockwise_global_ptq.py) | BlockWisePTQ → GlobalPTQ in a single Runner, followed by safetensors save/load |
+| | [example_blockwise_global_ptq_staged.py](./example/post_process/example_blockwise_global_ptq_staged.py) | Staged BlockWisePTQ → GlobalPTQ across save/load boundaries with accumulated post-process metadata |
+| | [example_global_ptq.py](./example/post_process/example_global_ptq.py) | Global PTQ with packed buffers by default and HF-compatible safetensors output |
+| | [example_global_ptq_dbf.py](./example/post_process/example_global_ptq_dbf.py) | Global PTQ with the DBF backend and HF-compatible safetensors output |
+| | [example_global_ptq_distributed.py](./example/post_process/example_global_ptq_distributed.py) | Multi-GPU Global PTQ with DeepSpeed / torchrun and safetensors output |
 | | [example_lora_sft.py](./example/post_process/example_lora_sft.py) | LoRA SFT post-quantization fine-tuning |
 | | [example_lora_sft_knowledge.py](./example/post_process/example_lora_sft_knowledge.py) | LoRA SFT knowledge injection |
+| | [example_lora_sft_knowledge_jointq.py](./example/post_process/example_lora_sft_knowledge_jointq.py) | LoRA SFT knowledge injection on a JointQ-quantized model |
+| | [example_lora_gptq_vllm_inference.py](./example/post_process/example_lora_gptq_vllm_inference.py) | GPTQ + LoRA SFT, HF-compatible safetensors + PEFT sidecar, and vLLM inference |
 | vLLM | [example_gptq_vllm_inference.py](./example/vllm_inference/example_gptq_vllm_inference.py) | GPTQ + QEP quantization and vLLM inference |
+| | [example_gptq_vllm_qwen36_inference.py](./example/vllm_inference/example_gptq_vllm_qwen36_inference.py) | GPTQ quantization and vLLM inference for Qwen3.6 (full-wrapper save) |
+| | [example_gptq_vllm_gptoss_inference.py](./example/vllm_inference/example_gptq_vllm_gptoss_inference.py) | GPTQ quantization and vLLM inference for gpt-oss |
 | | [example_jointq_vllm_inference.py](./example/vllm_inference/example_jointq_vllm_inference.py) | JointQ quantization and vLLM inference |
 | | [example_autobit_vllm_inference.py](./example/vllm_inference/example_autobit_vllm_inference.py) | AutoBit quantization and vLLM inference |
+| | [example_dbf_vllm_inference.py](./example/vllm_inference/example_dbf_vllm_inference.py) | DBF quantization and vLLM inference |
 
 ## 🔌 vLLM Inference
 
@@ -415,6 +447,22 @@ pip install vllm
 
 See the [vLLM Inference guide](https://FujitsuResearch.github.io/OneCompression/user-guide/vllm-inference/) for details, including Open WebUI setup instructions.
 
+### GPT-OSS (mixed_gptq)
+
+[GPT-OSS models](docs/user-guide/gptoss.md) (`openai/gpt-oss-20b`, `openai/gpt-oss-120b`) need extra steps beyond the
+standard vLLM plugin flow:
+
+1. Quantize with **`GPTQ(wbits=4)`** and keep the MoE experts 4-bit via `Runner(..., moe_quant_experts=True)` with `groupsize=64`. GPT-OSS `hidden_size` (2880) is not divisible by 128, so the experts must use **`group_size=64`** for vLLM's WNA16 MoE kernel.
+2. Before `LLM(...)`, apply vLLM runtime patches: `python -m vllm_plugins.patches.apply_all`
+3. Set `export VLLM_USE_DEEP_GEMM=0`
+
+The `--extra vllm` dependency set includes **`conch-triton-kernels`**. On NVIDIA Blackwell (B200, sm100),
+vLLM 0.20 needs Conch for `mixed_gptq` linear layers when the Marlin kernel cannot handle GPT-OSS weight shapes.
+If you install vLLM manually with `pip install vllm`, also run `pip install conch-triton-kernels`.
+
+See the [GPT-OSS guide](docs/user-guide/gptoss.md) for HF save/load, patch details, and verification scripts.
+
+
 
 ## 📬 Contact Us
 
@@ -466,3 +514,17 @@ year={2025},
 url={https://arxiv.org/abs/2512.01546}
 }
 ```
+
+MDBF (Multi-Envelope Double Binary Factorization):
+
+```
+@misc{ichikawa2025bitsmultienvelopedoublebinary,
+      title={More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantization}, 
+      author={Yuma Ichikawa and Yoshihiko Fujisawa and Yudai Fujimoto and Akira Sakai and Katsuki Fujisawa},
+      year={2025},
+      eprint={2512.24545},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2512.24545}, 
+}
+```

{onecomp-1.2.2 → onecomp-1.3.0}/README.md

@@ -38,10 +38,12 @@ Full documentation is available at **[https://FujitsuResearch.github.io/OneCompr
 - **vLLM Plugin Integration**: Serve OneComp-quantized models with [vLLM](https://docs.vllm.ai/) via built-in plugins for DBF and Mixed-GPTQ quantization methods. Pair with [Open WebUI](https://github.com/open-webui/open-webui) for a ChatGPT-like chat experience on your local machine.
 - **AutoBit**: Mixed-precision quantization with ILP-based bitwidth assignment. Automatically estimates the target bitwidth from available VRAM and assigns per-layer bitwidths to minimize quantization error under the memory budget.
 - **JointQ**: Joint quantization method that optimizes weight assignments and scale parameters simultaneously for improved quantization accuracy. Supports group-wise quantization (e.g., 4-bit, groupsize=128).
+- **MDBF (Multi-Envelope Double Binary Factorization)**: A binary factorization quantizer that approximates each weight matrix as a sum of multi-path sign matrices with multi-scale FP16 envelopes, generalizing DBF and LittleBit for aggressive low-bit (sub-1-bit) compression. Supports ADMM/gradient refinement, activation-aware initialization, and a GemLite-accelerated 1-bit inference path. See the [MDBF guide](https://FujitsuResearch.github.io/OneCompression/algorithms/mdbf/) for details.
 - **Block-wise PTQ**: Post-quantization block-wise distillation that minimises intermediate-representation MSE against an FP16 teacher model at Transformer-block granularity. Includes Phase 1 (greedy per-block optimisation) and Phase 2 CBQ (cross-block sliding-window optimisation). Supports GPTQ, DBF, and OneBit quantizers.
 - **LoRA SFT Post-Process**: Fine-tune quantized models with LoRA adapters for accuracy recovery or domain-specific knowledge injection. Supports SFT loss, teacher distillation, and intermediate block alignment.
 - **Rotation Preprocessing**: SpinQuant/OstQuant-based rotation preprocessing that reduces quantization error by learning optimal rotation matrices before quantization. Rotation/scaling matrices are absorbed into model weights, with online Hadamard hooks automatically registered at load time. Supports Llama and Qwen3 architectures.
 - **Web Dashboard (HPC)**: A browser-based dashboard for launching quantization jobs, deploying models, and validating chat-based inference in HPC environments. See [dashboard/README.md](dashboard/README.md) for details.
+- **GGUF Export & Hugging Face Hub Integration**: Convert models to the GGUF v3 format (F16) for llama.cpp/Ollama with a dependency-free built-in writer, generate model cards with quantization recipes and evaluation results, and push save directories to the Hugging Face Hub. Supports Llama (SentencePiece or Llama-3-style BPE) and Qwen2 (BPE) architectures, including multi-EOS stop-token mapping. See the [GGUF Export guide](https://FujitsuResearch.github.io/OneCompression/user-guide/gguf-export/).
 - (TBD)
 
 ## 🤖 Supported Models
@@ -54,6 +56,8 @@ Other Hugging Face-compatible models may work but are currently untested.
 | 1 | Llama | TinyLlama, Llama-2, Llama-3 | ✅ Verified |
 | 2 | Qwen3 | Qwen3-0.6B ~ 32B | ✅ Verified |
 | 3 | Gemma | Gemma 2, Gemma 3, Gemma 4  | ✅ Verified |
+| 4 | [GPT-OSS](docs/user-guide/gptoss.md) | openai/gpt-oss-20b, gpt-oss-120b | ✅ Verified |
+| 5 | Qwen3.6 | Qwen3.6-27B, Qwen3.6-35B-A3B | ✅ Verified |
 
 
 > **Note:** Support for additional architectures is planned. Contributions and test reports are welcome.
@@ -274,10 +278,19 @@ uv run mkdocs serve
 
 Then open [http://127.0.0.1:8000](http://127.0.0.1:8000) in your browser.
 
+## 📓 Tutorial Notebook
+
+Interactive walkthrough in Jupyter or [Google Colab](https://colab.research.google.com/github/FujitsuResearch/OneCompression/blob/main/notebook/01_tutorial.ipynb)
+— RTN visualization, `Runner.auto_run`, and vLLM chat inference.
+
+See [`notebook/README.md`](./notebook/README.md) for local setup, or the
+[Tutorial Notebook guide](https://FujitsuResearch.github.io/OneCompression/getting-started/tutorial-notebook/) in the docs.
+
 ## 🚀 Examples
 
 | Category | Script | Description |
 |----------|--------|-------------|
+| Tutorial | [01_tutorial.ipynb](./notebook/01_tutorial.ipynb) | Interactive notebook (Jupyter / Colab) |
 | Quantization | [example_gptq.py](./example/example_gptq.py) | GPTQ quantization |
 | | [example_qep_gptq.py](./example/example_qep_gptq.py) | GPTQ + QEP (error propagation) |
 | | [example_lpcd_gptq.py](./example/example_lpcd_gptq.py) | GPTQ + QEP + LPCD quantization |
@@ -286,14 +299,25 @@ Then open [http://127.0.0.1:8000](http://127.0.0.1:8000) in your browser.
 | | [example_auto_run.py](./example/example_auto_run.py) | AutoBit with automatic VRAM estimation |
 | Calibration | [example_custom_calibration.py](./example/example_custom_calibration.py) | Custom calibration dataset with CalibrationConfig |
 | Save / Load | [example_save_load.py](./example/example_save_load.py) | Save and load quantized models |
+| | [example_reload_post_process_resave.py](./example/post_process/example_reload_post_process_resave.py) | Reload, post-process, and re-save a quantized checkpoint |
 | Rotation Preprocessing | [example_llama_preprocess_rtn.py](./example/pre_process/example_llama_preprocess_rtn.py) | Rotation preprocessing + RTN (TinyLlama) |
 | | [example_preprocess_save_load.py](./example/pre_process/example_preprocess_save_load.py) | Save and load rotation-preprocessed quantized models |
-| Post-Process | [example_blockwise_ptq.py](./example/post_process/example_blockwise_ptq.py) | Block-wise PTQ (GPTQ + Phase 1 & CBQ) |
+| Post-Process | [example_blockwise_ptq.py](./example/post_process/example_blockwise_ptq.py) | Block-wise PTQ via `Runner.run()` with packed buffers by default |
+| | [example_blockwise_global_ptq.py](./example/post_process/example_blockwise_global_ptq.py) | BlockWisePTQ → GlobalPTQ in a single Runner, followed by safetensors save/load |
+| | [example_blockwise_global_ptq_staged.py](./example/post_process/example_blockwise_global_ptq_staged.py) | Staged BlockWisePTQ → GlobalPTQ across save/load boundaries with accumulated post-process metadata |
+| | [example_global_ptq.py](./example/post_process/example_global_ptq.py) | Global PTQ with packed buffers by default and HF-compatible safetensors output |
+| | [example_global_ptq_dbf.py](./example/post_process/example_global_ptq_dbf.py) | Global PTQ with the DBF backend and HF-compatible safetensors output |
+| | [example_global_ptq_distributed.py](./example/post_process/example_global_ptq_distributed.py) | Multi-GPU Global PTQ with DeepSpeed / torchrun and safetensors output |
 | | [example_lora_sft.py](./example/post_process/example_lora_sft.py) | LoRA SFT post-quantization fine-tuning |
 | | [example_lora_sft_knowledge.py](./example/post_process/example_lora_sft_knowledge.py) | LoRA SFT knowledge injection |
+| | [example_lora_sft_knowledge_jointq.py](./example/post_process/example_lora_sft_knowledge_jointq.py) | LoRA SFT knowledge injection on a JointQ-quantized model |
+| | [example_lora_gptq_vllm_inference.py](./example/post_process/example_lora_gptq_vllm_inference.py) | GPTQ + LoRA SFT, HF-compatible safetensors + PEFT sidecar, and vLLM inference |
 | vLLM | [example_gptq_vllm_inference.py](./example/vllm_inference/example_gptq_vllm_inference.py) | GPTQ + QEP quantization and vLLM inference |
+| | [example_gptq_vllm_qwen36_inference.py](./example/vllm_inference/example_gptq_vllm_qwen36_inference.py) | GPTQ quantization and vLLM inference for Qwen3.6 (full-wrapper save) |
+| | [example_gptq_vllm_gptoss_inference.py](./example/vllm_inference/example_gptq_vllm_gptoss_inference.py) | GPTQ quantization and vLLM inference for gpt-oss |
 | | [example_jointq_vllm_inference.py](./example/vllm_inference/example_jointq_vllm_inference.py) | JointQ quantization and vLLM inference |
 | | [example_autobit_vllm_inference.py](./example/vllm_inference/example_autobit_vllm_inference.py) | AutoBit quantization and vLLM inference |
+| | [example_dbf_vllm_inference.py](./example/vllm_inference/example_dbf_vllm_inference.py) | DBF quantization and vLLM inference |
 
 ## 🔌 vLLM Inference
 
@@ -310,6 +334,22 @@ pip install vllm
 
 See the [vLLM Inference guide](https://FujitsuResearch.github.io/OneCompression/user-guide/vllm-inference/) for details, including Open WebUI setup instructions.
 
+### GPT-OSS (mixed_gptq)
+
+[GPT-OSS models](docs/user-guide/gptoss.md) (`openai/gpt-oss-20b`, `openai/gpt-oss-120b`) need extra steps beyond the
+standard vLLM plugin flow:
+
+1. Quantize with **`GPTQ(wbits=4)`** and keep the MoE experts 4-bit via `Runner(..., moe_quant_experts=True)` with `groupsize=64`. GPT-OSS `hidden_size` (2880) is not divisible by 128, so the experts must use **`group_size=64`** for vLLM's WNA16 MoE kernel.
+2. Before `LLM(...)`, apply vLLM runtime patches: `python -m vllm_plugins.patches.apply_all`
+3. Set `export VLLM_USE_DEEP_GEMM=0`
+
+The `--extra vllm` dependency set includes **`conch-triton-kernels`**. On NVIDIA Blackwell (B200, sm100),
+vLLM 0.20 needs Conch for `mixed_gptq` linear layers when the Marlin kernel cannot handle GPT-OSS weight shapes.
+If you install vLLM manually with `pip install vllm`, also run `pip install conch-triton-kernels`.
+
+See the [GPT-OSS guide](docs/user-guide/gptoss.md) for HF save/load, patch details, and verification scripts.
+
+
 
 ## 📬 Contact Us
 
@@ -361,3 +401,17 @@ year={2025},
 url={https://arxiv.org/abs/2512.01546}
 }
 ```
+
+MDBF (Multi-Envelope Double Binary Factorization):
+
+```
+@misc{ichikawa2025bitsmultienvelopedoublebinary,
+      title={More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantization}, 
+      author={Yuma Ichikawa and Yoshihiko Fujisawa and Yudai Fujimoto and Akira Sakai and Katsuki Fujisawa},
+      year={2025},
+      eprint={2512.24545},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2512.24545}, 
+}
+```

onecomp-1.3.0/envs/vllm/v0_24_0_rocm/src/onecomp_vllm_v0_24_0_rocm/__init__.py

@@ -0,0 +1,12 @@
+"""ROCm-only workaround for vLLM 0.24.0 TritonW4A16LinearKernel.
+
+Copyright 2025-2026 Fujitsu Ltd.
+
+The entry point installed via ``pip install``
+(declared in this package's ``pyproject.toml``) is
+:func:`onecomp_vllm_v0_24_0_rocm.patch.apply`.
+"""
+
+from .patch import apply
+
+__all__ = ["apply"]

onecomp-1.3.0/envs/vllm/v0_24_0_rocm/src/onecomp_vllm_v0_24_0_rocm/patch.py

@@ -0,0 +1,238 @@
+"""ROCm-only workaround for vLLM 0.24.x TritonW4A16LinearKernel.
+
+Copyright 2025-2026 Fujitsu Ltd.
+
+"""
+
+from __future__ import annotations
+
+_APPLIED_MARKER_AUTO_GPTQ = "_onecomp_vllm_0_24_0_rocm_applied_auto_gptq"
+_APPLIED_MARKER_KERNEL = "_onecomp_vllm_0_24_0_rocm_applied_kernel"
+
+# Set after the first inference-time fixup so logs stay readable (once per process).
+_LOGGED_AUTO_GPTQ_QZEROS_FIXUP = False
+_LOGGED_TRITON_QZEROS_PERMUTE = False
+
+# vLLM only attaches handlers to the ``vllm`` logger tree; ``__name__`` logs are silently dropped.
+# Use a child of ``vllm`` so INFO lines appear in engine output (including EngineCore subprocesses).
+_VLLM_LOGGER_NAME = "vllm.onecomp_v0_24_0_rocm"
+
+
+def _get_logger():
+    from vllm.logger import init_logger
+
+    return init_logger(_VLLM_LOGGER_NAME)
+
+
+def _is_target_env() -> tuple[bool, str]:
+    """Return ``(is_target, reason)`` for the current Python / vLLM env."""
+    try:
+        import vllm  # noqa: F401
+    except ImportError as exc:
+        return False, f"vllm not importable ({exc})"
+    try:
+        from vllm.platforms import current_platform
+    except ImportError as exc:
+        return False, f"vllm.platforms not importable ({exc})"
+
+    if not current_platform.is_rocm():
+        return False, "not ROCm"
+
+    # Accept "0.24.0", "0.24.0+rocm723", "0.24.1", "0.24.0rc3.dev3", etc.
+    base_version = vllm.__version__.split("+", 1)[0]
+    if not base_version.startswith("0.24."):
+        return False, f"vllm base version {base_version!r} not in 0.24.x"
+
+    return True, "vllm 0.24.x on ROCm"
+
+
+def _unbias_v1_zeros(zp_packed):
+    """Add 1 (mod 16) to every 4-bit nibble of a GPTQ-packed int32 tensor.
+
+    Parameters
+    ----------
+    zp_packed
+        ``[K // G, N // 8]`` int32 tensor.  GPTQ sequential packing: each
+        int32 holds 8 consecutive N-values at bit offsets
+        ``[0, 4, 8, ..., 28]``.
+
+    Returns
+    -------
+    Same shape / dtype / device, with every nibble incremented by 1 mod 16.
+    """
+    import torch
+
+    shifts = torch.arange(8, device=zp_packed.device, dtype=torch.int32) * 4
+    nibbles = ((zp_packed.unsqueeze(-1) >> shifts) & 0xF) + 1
+    nibbles &= 0xF
+    return torch.sum(nibbles << shifts, dim=-1, dtype=torch.int32)
+
+
+def _patch_auto_gptq_process_weights(logger) -> None:
+    """Wrap ``AutoGPTQLinearMethod.process_weights_after_loading`` with a +1
+    fixup on ``qzeros`` for the GPTQv1 convention.
+
+    Rationale
+    ---------
+    GPTQv1 checkpoints store ``stored_zero = real_zero - 1``.  Marlin/Machete
+    kernels re-add the ``-1`` bias internally, but ``TritonW4A16LinearKernel``
+    consumes ``qzeros`` verbatim, so we need to add 1 (mod 16) to every
+    packed nibble before the kernel sees the parameter.  This is scoped to
+    the AutoGPTQ path (i.e. GPTQv1 checkpoints); other quantization methods
+    that reuse ``TritonW4A16LinearKernel`` (e.g. compressed-tensors) do not
+    have the ``-1`` convention and are left untouched.
+    """
+    from vllm.model_executor.kernels.linear.mixed_precision.triton_w4a16 import (
+        TritonW4A16LinearKernel,
+    )
+    from vllm.model_executor.layers.quantization.auto_gptq import (
+        AutoGPTQConfig,
+        AutoGPTQLinearMethod,
+    )
+    from vllm.scalar_type import scalar_types
+
+    # Additive: unblock asymmetric GPTQ checkpoints.  Stock vLLM 0.24 still
+    # only registers (n_bits, True) in TYPE_MAP, which makes AutoGPTQConfig
+    # .from_config raise ValueError for sym=False models.  TritonW4A16
+    # already advertises uint4 (asymmetric) in SUPPORTED_QUANT_TYPES, so
+    # this extension does not change kernel selection for sym=True.
+    AutoGPTQConfig.TYPE_MAP.setdefault((4, False), scalar_types.uint4)
+    AutoGPTQConfig.TYPE_MAP.setdefault((8, False), scalar_types.uint8)
+
+    if getattr(
+        AutoGPTQLinearMethod.process_weights_after_loading,
+        _APPLIED_MARKER_AUTO_GPTQ,
+        False,
+    ):
+        logger.debug("onecomp env-patch already installed on AutoGPTQLinearMethod; skip")
+        return
+
+    _orig = AutoGPTQLinearMethod.process_weights_after_loading
+
+    def _patched(self, layer):
+        global _LOGGED_AUTO_GPTQ_QZEROS_FIXUP
+        # +1 fixup runs BEFORE the kernel's process_weights_after_loading so
+        # that we operate on the AutoGPTQ-native layout [K//G, N//8] where
+        # the packed axis is dim -1 (which is what ``_unbias_v1_zeros``
+        # assumes).  The subsequent shape/permute juggling in the kernel
+        # PWAL preserves per-nibble values.
+        if isinstance(self.kernel, TritonW4A16LinearKernel):
+            zp = getattr(layer, "qzeros", None)
+            if zp is not None and getattr(zp, "data", None) is not None:
+                zp.data = _unbias_v1_zeros(zp.data)
+                if not _LOGGED_AUTO_GPTQ_QZEROS_FIXUP:
+                    _LOGGED_AUTO_GPTQ_QZEROS_FIXUP = True
+                    logger.info(
+                        "onecomp env-patch active at inference: "
+                        "AutoGPTQLinearMethod qzeros +1 fixup "
+                        "(TritonW4A16LinearKernel, qzeros shape=%s)",
+                        tuple(zp.data.shape),
+                    )
+        return _orig(self, layer)
+
+    setattr(_patched, _APPLIED_MARKER_AUTO_GPTQ, True)
+    AutoGPTQLinearMethod.process_weights_after_loading = _patched
+    logger.info("onecomp env-patch: wrapped AutoGPTQLinearMethod.process_weights_after_loading")
+
+
+def _patch_triton_w4a16_kernel_process_weights(logger) -> None:
+    """Wrap ``TritonW4A16LinearKernel.process_weights_after_loading`` to fix
+    a missing ``permute_param_layout_`` call on ``qzeros``.
+
+    Rationale
+    ---------
+    In vLLM 0.24.x the stock ``TritonW4A16LinearKernel.process_weights_after_loading``
+    normalizes ``qweight`` and ``scales`` via ``permute_param_layout_`` so
+    that both compressed-tensors (``output_dim=0, packed_dim=0``) and
+    AutoGPTQ (``input_dim=0, output_dim=1, packed_dim=1``) checkpoint
+    layouts land in the same physical form before the ``.t().contiguous()``
+    that the kernel expects.  The block for ``qzeros`` skips this step and
+    calls ``.t()`` unconditionally::
+
+        # Checkpoint: [N//8, K//G] int32 (N packed at dim 0, K//G at dim 1)
+        # Kernel needs: [K//G, N//8] -- just transpose
+        replace_parameter(
+            layer, self.w_zp_name,
+            torch.nn.Parameter(zp.data.t().contiguous(), requires_grad=False),
+        )
+
+    That works only for the compressed-tensors layout.  For an AutoGPTQ
+    checkpoint the parameter is already ``[K//G, N//8]``, so the ``.t()``
+    flips it to ``[N//8, K//G]`` and the downstream shape assertion in
+    ``triton_w4a16_gemm`` fires (observed as
+    ``AssertionError: qzeros shape mismatch: torch.Size([N//8, K//G])``).
+
+    The one-liner upstream is missing is a ``permute_param_layout_(zp,
+    input_dim=1, output_dim=0, packed_dim=0)`` call before the ``.t()``.
+    Injecting it via a wrapper here fixes AutoGPTQ without regressing the
+    compressed-tensors path (that path already satisfies the target
+    layout, so the permute is a no-op).
+    """
+    from vllm.model_executor.kernels.linear.mixed_precision.triton_w4a16 import (
+        TritonW4A16LinearKernel,
+    )
+    from vllm.model_executor.parameter import permute_param_layout_
+
+    if getattr(
+        TritonW4A16LinearKernel.process_weights_after_loading,
+        _APPLIED_MARKER_KERNEL,
+        False,
+    ):
+        logger.debug("onecomp env-patch already installed on TritonW4A16LinearKernel; skip")
+        return
+
+    _orig = TritonW4A16LinearKernel.process_weights_after_loading
+
+    def _patched(self, layer):
+        global _LOGGED_TRITON_QZEROS_PERMUTE
+        if self.w_zp_name is not None:
+            zp = getattr(layer, self.w_zp_name, None)
+            if zp is not None:
+                # Normalise both compressed-tensors and AutoGPTQ layouts to
+                # [N//8, K//G] with (input_dim=1, output_dim=0, packed_dim=0)
+                # so the stock code's subsequent zp.data.t().contiguous()
+                # ends up at the kernel's expected [K//G, N//8].
+                permute_param_layout_(zp, input_dim=1, output_dim=0, packed_dim=0)
+                if not _LOGGED_TRITON_QZEROS_PERMUTE:
+                    _LOGGED_TRITON_QZEROS_PERMUTE = True
+                    logger.info(
+                        "onecomp env-patch active at inference: "
+                        "TritonW4A16LinearKernel qzeros permute_param_layout_ "
+                        "(param=%s, shape=%s)",
+                        self.w_zp_name,
+                        tuple(zp.data.shape),
+                    )
+        return _orig(self, layer)
+
+    setattr(_patched, _APPLIED_MARKER_KERNEL, True)
+    TritonW4A16LinearKernel.process_weights_after_loading = _patched
+    logger.info("onecomp env-patch: wrapped TritonW4A16LinearKernel.process_weights_after_loading")
+
+
+def apply() -> None:
+    """vLLM ``vllm.general_plugins`` entry point.
+
+    No-op on every platform / version other than ROCm + vLLM 0.24.x.
+    """
+    logger = _get_logger()
+
+    is_target, reason = _is_target_env()
+    if not is_target:
+        logger.debug("onecomp env-patch vllm_v0_24_0_rocm skipped: %s", reason)
+        return
+
+    import vllm
+
+    logger.info(
+        "onecomp env-patch: loading for %s (vllm %s)",
+        reason,
+        vllm.__version__,
+    )
+    _patch_auto_gptq_process_weights(logger)
+    _patch_triton_w4a16_kernel_process_weights(logger)
+    logger.info(
+        "onecomp env-patch installed: "
+        "AutoGPTQLinearMethod qzeros +1 fixup for TritonW4A16LinearKernel; "
+        "TritonW4A16LinearKernel qzeros layout permute injected; "
+        "AutoGPTQConfig.TYPE_MAP extended with asymmetric uint4/uint8.",
+    )

onecomp-1.3.0/example/cpu_inference/example_gptq_gguf_cpu.py

@@ -0,0 +1,57 @@
+"""GPTQ quantization -> direct GGUF export -> CPU inference with llama.cpp.
+
+This example shows the OneComp CPU inference path:
+  1. Quantize a model with GPTQ (4-bit symmetric, group size 128).
+  2. Export it to GGUF without re-quantization (preserving the GPTQ codes).
+  3. Run CPU text generation via llama-cpp-python.
+
+Only the GGUF export and inference are CPU-bound; quantization runs on the GPU
+when one is available and falls back to CPU (ModelConfig default
+``device="auto"``; both are supported).
+
+Run:
+    python example/cpu_inference/example_gptq_gguf_cpu.py
+
+Requires: pip install 'onecomp[llamacpp]'  (gguf + llama-cpp-python)
+
+Copyright 2025-2026 Fujitsu Ltd.
+
+Author: Yuma Ichikawa
+
+"""
+
+import os
+
+from onecomp import GPTQ, CalibrationConfig, ModelConfig, Runner
+from onecomp.cpu import LlamaCppModel, convert_gptq_to_gguf
+from onecomp.log import setup_logger
+
+MODEL_ID = "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T"
+SAVE_DIR = "./TinyLlama-1.1B-gptq-4bit"
+GGUF_PATH = "./TinyLlama-1.1B-gptq-4bit.gguf"
+
+
+def main():
+    setup_logger()
+
+    # 1. Quantize (4-bit symmetric, group size 128, no actorder => lossless to Q4_0).
+    runner = Runner(
+        model_config=ModelConfig(model_id=MODEL_ID),
+        quantizer=GPTQ(wbits=4, groupsize=128, sym=True, actorder=False),
+        calibration_config=CalibrationConfig(num_calibration_samples=64, max_length=512),
+        qep=True,
+    )
+    runner.run()
+    runner.save_quantized_model(SAVE_DIR)
+
+    # 2. Direct GGUF export (no re-quantization; QEP-corrected codes preserved).
+    summary = convert_gptq_to_gguf(quantized_dir=SAVE_DIR, out_gguf=GGUF_PATH)
+    print("Export summary:", summary)
+
+    # 3. CPU inference.
+    model = LlamaCppModel(GGUF_PATH, n_ctx=1024)
+    print(model.generate("Fujitsu is", max_tokens=64, temperature=0.0))
+
+
+if __name__ == "__main__":
+    main()

onecomp-1.3.0/example/cpu_inference/example_mixed_gptq_gguf_cpu.py

@@ -0,0 +1,63 @@
+"""Example: mixed-bit GPTQ -> mixed-precision GGUF -> llama.cpp CPU inference.
+
+Quantizes a model with per-module bit-widths (4-bit attention, 8-bit MLP gate,
+3-bit MLP up, 2-bit MLP down) in a single GPTQ run, then exports it to ONE GGUF
+where each tensor carries its own quantization type:
+
+    4-bit sym -> Q4_0   8-bit sym -> Q8_0   3-bit -> Q3_K   2-bit -> Q2_K
+
+The 4/8-bit layers keep the exact GPTQ codes (lossless); the 2/3-bit layers are
+re-quantized to K-quants via llama-quantize (requires the llama-quantize binary).
+
+Copyright 2025-2026 Fujitsu Ltd.
+
+Author: Yuma Ichikawa
+"""
+
+import json
+import os
+
+from transformers import AutoConfig
+
+from llamacpp_plugins.gptq import export_mixed_gptq_gguf, plan_mixed_export
+from onecomp import GPTQ, CalibrationConfig, ModelConfig, Runner, setup_logger
+from onecomp.cpu import LlamaCppModel
+
+MODEL_ID = "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T"
+OUT_DIR = "./results/mixed-gptq"
+OUT_GGUF = "./results/mixed-gptq.gguf"
+
+
+def main():
+    setup_logger()
+
+    num_layers = AutoConfig.from_pretrained(MODEL_ID).num_hidden_layers
+    module_wbits = {}
+    for i in range(num_layers):
+        module_wbits[f"model.layers.{i}.mlp.gate_proj"] = 8
+        module_wbits[f"model.layers.{i}.mlp.up_proj"] = 3
+        module_wbits[f"model.layers.{i}.mlp.down_proj"] = 2
+
+    quantizer = GPTQ(wbits=4, groupsize=128, sym=True, module_wbits=module_wbits)
+    runner = Runner(
+        model_config=ModelConfig(model_id=MODEL_ID, device="cpu"),
+        quantizer=quantizer,
+        calibration_config=CalibrationConfig(num_calibration_samples=32, max_length=512),
+        qep=False,
+    )
+    runner.run()
+    runner.save_quantized_model(OUT_DIR)
+
+    # Preview the per-module GGUF routing.
+    for p in plan_mixed_export(OUT_DIR)[:6]:
+        print(f"{p.name:45s} bits={p.bits} -> {p.route}/{p.ggml_type}")
+
+    summary = export_mixed_gptq_gguf(OUT_DIR, OUT_GGUF)
+    print(json.dumps(summary["plan"], indent=2))
+
+    model = LlamaCppModel(OUT_GGUF, n_ctx=512)
+    print(model.generate("The capital of Japan is", max_tokens=32, temperature=0.0))
+
+
+if __name__ == "__main__":
+    main()

onecomp-1.3.0/example/cpu_inference/example_serve_cpu.py

@@ -0,0 +1,43 @@
+"""Serve a packed OneComp checkpoint (or GGUF) on CPU with one command.
+
+The barrier-free path: drop in a *packed quantized* model directory and anyone
+can run it on CPU behind an OpenAI-compatible API. If given a packed OneComp
+checkpoint, the GGUF is auto-exported (losslessly, cached) on first launch.
+
+Run:
+    # Packed OneComp GPTQ checkpoint (auto-exports to GGUF once):
+    python example/cpu_inference/example_serve_cpu.py ./TinyLlama-1.1B-gptq-4bit
+    # ...or an existing GGUF:
+    python example/cpu_inference/example_serve_cpu.py ./model.gguf
+
+Then, from another shell:
+    curl http://localhost:8080/v1/chat/completions \
+      -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":64}'
+
+Equivalent CLI: onecomp-gguf serve --model <path> --port 8080
+
+Requires: pip install 'onecomp[llamacpp]'  (no FastAPI/uvicorn needed)
+
+Copyright 2025-2026 Fujitsu Ltd.
+
+Author: Yuma Ichikawa
+
+"""
+
+import sys
+
+from onecomp.cpu import serve
+from onecomp.log import setup_logger
+
+
+def main():
+    setup_logger()
+    model_path = sys.argv[1] if len(sys.argv) > 1 else "./TinyLlama-1.1B-gptq-4bit"
+    port = int(sys.argv[2]) if len(sys.argv) > 2 else 8080
+    # host=0.0.0.0 to expose on the network; chat_format is auto-detected from
+    # the GGUF architecture when no chat template is embedded.
+    serve(model_path, host="0.0.0.0", port=port, n_ctx=2048)
+
+
+if __name__ == "__main__":
+    main()