vllm-ascend 0.9.1rc2__tar.gz → 0.9.1rc3__tar.gz

{vllm_ascend-0.9.1rc2 → vllm_ascend-0.9.1rc3}/Dockerfile

@@ -53,7 +53,7 @@ RUN source /usr/local/Ascend/ascend-toolkit/set_env.sh && \
     python3 -m pip cache purge
 
 # Install modelscope (for fast download) and ray (for multinode)
-RUN python3 -m pip install modelscope ray && \
+RUN python3 -m pip install modelscope 'ray>=2.47.1' 'protobuf>3.20.0' && \
     python3 -m pip cache purge
 
 CMD ["/bin/bash"]

{vllm_ascend-0.9.1rc2 → vllm_ascend-0.9.1rc3}/Dockerfile.openEuler

@@ -50,7 +50,7 @@ RUN source /usr/local/Ascend/ascend-toolkit/set_env.sh && \
     python3 -m pip cache purge
 
 # Install modelscope (for fast download) and ray (for multinode)
-RUN python3 -m pip install modelscope ray && \
+RUN python3 -m pip install modelscope 'ray>=2.47.1' 'protobuf>3.20.0' && \
     python3 -m pip cache purge
 
 CMD ["/bin/bash"]

{vllm_ascend-0.9.1rc2 → vllm_ascend-0.9.1rc3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: vllm_ascend
-Version: 0.9.1rc2
+Version: 0.9.1rc3
 Summary: vLLM Ascend backend plugin
 Home-page: https://github.com/vllm-project/vllm-ascend
 Author: vLLM-Ascend team
@@ -40,6 +40,10 @@ vLLM Ascend Plugin
 
 ---
 *Latest News* 🔥
+
+- [2025/06] [User stories](https://vllm-ascend.readthedocs.io/en/latest/community/user_stories/index.html) page is now live! It kicks off with ‌LLaMA-Factory/verl//TRL/GPUStack‌ to demonstrate how ‌vLLM Ascend‌ assists Ascend users in enhancing their experience across fine-tuning, evaluation, reinforcement learning (RL), and deployment scenarios.
+- [2025/06] [Contributors](https://vllm-ascend.readthedocs.io/en/latest/community/contributors.html) page is now live! All contributions deserve to be recorded, thanks for all contributors.
+- [2025/05] We've released first official version [v0.7.3](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.7.3)! We collaborated with the vLLM community to publish a blog post sharing our practice: [Introducing vLLM Hardware Plugin, Best Practice from Ascend NPU](https://blog.vllm.ai/2025/05/12/hardware-plugin.html).
 - [2025/03] We hosted the [vLLM Beijing Meetup](https://mp.weixin.qq.com/s/VtxO9WXa5fC-mKqlxNUJUQ) with vLLM team! Please find the meetup slides [here](https://drive.google.com/drive/folders/1Pid6NSFLU43DZRi0EaTcPgXsAzDvbBqF).
 - [2025/02] vLLM community officially created [vllm-project/vllm-ascend](https://github.com/vllm-project/vllm-ascend) repo for running vLLM seamlessly on the Ascend NPU.
 - [2024/12] We are working with the vLLM community to support [[RFC]: Hardware pluggable](https://github.com/vllm-project/vllm/issues/11162).
@@ -64,10 +68,16 @@ By using vLLM Ascend plugin, popular open-source models, including Transformer-l
 
 ## Getting Started
 
-Please refer to [QuickStart](https://vllm-ascend.readthedocs.io/en/latest/quick_start.html) and [Installation](https://vllm-ascend.readthedocs.io/en/latest/installation.html) for more details.
+Please use the following recommended versions to get started quickly:
+
+| Version    | Release type | Doc                                  |
+|------------|--------------|--------------------------------------|
+|v0.9.2rc1|Latest release candidate|[QuickStart](https://vllm-ascend.readthedocs.io/en/latest/quick_start.html) and [Installation](https://vllm-ascend.readthedocs.io/en/latest/installation.html) for more details|
+|v0.9.1rc3|Next stable release|[QuickStart](https://vllm-ascend.readthedocs.io/en/v0.9.1-dev/quick_start.html) and [Installation](https://vllm-ascend.readthedocs.io/en/v0.9.1-dev/installation.html) for more details|
+|v0.7.3.post1|Latest stable version|[QuickStart](https://vllm-ascend.readthedocs.io/en/stable/quick_start.html) and [Installation](https://vllm-ascend.readthedocs.io/en/stable/installation.html) for more details|
 
 ## Contributing
-See [CONTRIBUTING](https://vllm-ascend.readthedocs.io/en/main/developer_guide/contributing.html) for more details, which is a step-by-step guide to help you set up development environment, build and test.
+See [CONTRIBUTING](https://vllm-ascend.readthedocs.io/en/latest/developer_guide/contribution/index.html) for more details, which is a step-by-step guide to help you set up development environment, build and test.
 
 We welcome and value any contributions and collaborations:
 - Please let us know if you encounter a bug by [filing an issue](https://github.com/vllm-project/vllm-ascend/issues)
@@ -86,9 +96,10 @@ Below is maintained branches:
 |------------|--------------|--------------------------------------|
 | main       | Maintained   | CI commitment for vLLM main branch and vLLM 0.9.x branch   |
 | v0.7.1-dev | Unmaintained | Only doc fixed is allowed |
-| v0.7.3-dev | Maintained   | CI commitment for vLLM 0.7.3 version |
+| v0.7.3-dev | Maintained   | CI commitment for vLLM 0.7.3 version, only bug fix is allowed and no new release tag any more. |
+| v0.9.1-dev | Maintained   | CI commitment for vLLM 0.9.1 version |
 
-Please refer to [Versioning policy](https://vllm-ascend.readthedocs.io/en/main/developer_guide/versioning_policy.html) for more details.
+Please refer to [Versioning policy](https://vllm-ascend.readthedocs.io/en/latest/community/versioning_policy.html) for more details.
 
 ## Weekly Meeting
 

{vllm_ascend-0.9.1rc2 → vllm_ascend-0.9.1rc3}/README.md

@@ -19,6 +19,10 @@ vLLM Ascend Plugin
 
 ---
 *Latest News* 🔥
+
+- [2025/06] [User stories](https://vllm-ascend.readthedocs.io/en/latest/community/user_stories/index.html) page is now live! It kicks off with ‌LLaMA-Factory/verl//TRL/GPUStack‌ to demonstrate how ‌vLLM Ascend‌ assists Ascend users in enhancing their experience across fine-tuning, evaluation, reinforcement learning (RL), and deployment scenarios.
+- [2025/06] [Contributors](https://vllm-ascend.readthedocs.io/en/latest/community/contributors.html) page is now live! All contributions deserve to be recorded, thanks for all contributors.
+- [2025/05] We've released first official version [v0.7.3](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.7.3)! We collaborated with the vLLM community to publish a blog post sharing our practice: [Introducing vLLM Hardware Plugin, Best Practice from Ascend NPU](https://blog.vllm.ai/2025/05/12/hardware-plugin.html).
 - [2025/03] We hosted the [vLLM Beijing Meetup](https://mp.weixin.qq.com/s/VtxO9WXa5fC-mKqlxNUJUQ) with vLLM team! Please find the meetup slides [here](https://drive.google.com/drive/folders/1Pid6NSFLU43DZRi0EaTcPgXsAzDvbBqF).
 - [2025/02] vLLM community officially created [vllm-project/vllm-ascend](https://github.com/vllm-project/vllm-ascend) repo for running vLLM seamlessly on the Ascend NPU.
 - [2024/12] We are working with the vLLM community to support [[RFC]: Hardware pluggable](https://github.com/vllm-project/vllm/issues/11162).
@@ -43,10 +47,16 @@ By using vLLM Ascend plugin, popular open-source models, including Transformer-l
 
 ## Getting Started
 
-Please refer to [QuickStart](https://vllm-ascend.readthedocs.io/en/latest/quick_start.html) and [Installation](https://vllm-ascend.readthedocs.io/en/latest/installation.html) for more details.
+Please use the following recommended versions to get started quickly:
+
+| Version    | Release type | Doc                                  |
+|------------|--------------|--------------------------------------|
+|v0.9.2rc1|Latest release candidate|[QuickStart](https://vllm-ascend.readthedocs.io/en/latest/quick_start.html) and [Installation](https://vllm-ascend.readthedocs.io/en/latest/installation.html) for more details|
+|v0.9.1rc3|Next stable release|[QuickStart](https://vllm-ascend.readthedocs.io/en/v0.9.1-dev/quick_start.html) and [Installation](https://vllm-ascend.readthedocs.io/en/v0.9.1-dev/installation.html) for more details|
+|v0.7.3.post1|Latest stable version|[QuickStart](https://vllm-ascend.readthedocs.io/en/stable/quick_start.html) and [Installation](https://vllm-ascend.readthedocs.io/en/stable/installation.html) for more details|
 
 ## Contributing
-See [CONTRIBUTING](https://vllm-ascend.readthedocs.io/en/main/developer_guide/contributing.html) for more details, which is a step-by-step guide to help you set up development environment, build and test.
+See [CONTRIBUTING](https://vllm-ascend.readthedocs.io/en/latest/developer_guide/contribution/index.html) for more details, which is a step-by-step guide to help you set up development environment, build and test.
 
 We welcome and value any contributions and collaborations:
 - Please let us know if you encounter a bug by [filing an issue](https://github.com/vllm-project/vllm-ascend/issues)
@@ -65,9 +75,10 @@ Below is maintained branches:
 |------------|--------------|--------------------------------------|
 | main       | Maintained   | CI commitment for vLLM main branch and vLLM 0.9.x branch   |
 | v0.7.1-dev | Unmaintained | Only doc fixed is allowed |
-| v0.7.3-dev | Maintained   | CI commitment for vLLM 0.7.3 version |
+| v0.7.3-dev | Maintained   | CI commitment for vLLM 0.7.3 version, only bug fix is allowed and no new release tag any more. |
+| v0.9.1-dev | Maintained   | CI commitment for vLLM 0.9.1 version |
 
-Please refer to [Versioning policy](https://vllm-ascend.readthedocs.io/en/main/developer_guide/versioning_policy.html) for more details.
+Please refer to [Versioning policy](https://vllm-ascend.readthedocs.io/en/latest/community/versioning_policy.html) for more details.
 
 ## Weekly Meeting
 

{vllm_ascend-0.9.1rc2 → vllm_ascend-0.9.1rc3}/README.zh.md

@@ -20,6 +20,9 @@ vLLM Ascend Plugin
 ---
 *最新消息* 🔥
 
+- [2025/06] [用户案例](https://vllm-ascend.readthedocs.io/en/latest/community/user_stories/index.html)现已上线！展示了LLaMA-Factory/verl/TRL/GPUStack等用户案例，展示了vLLM Ascend如何帮助昇腾用户在模型微调、评估、强化学习 (RL) 以及部署等场景中提升体验。
+- [2025/06] [贡献者](https://vllm-ascend.readthedocs.io/en/latest/community/contributors.html)页面现已上线！所有的贡献都值得被记录，感谢所有的贡献者。
+- [2025/05] 我们发布了首个正式版本 [v0.7.3](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.7.3)！我们与 vLLM 社区合作发布了一篇博客文章，分享了我们的实践：[Introducing vLLM Hardware Plugin, Best Practice from Ascend NPU](https://blog.vllm.ai/2025/05/12/hardware-plugin.html)。
 - [2025/03] 我们和vLLM团队举办了[vLLM Beijing Meetup](https://mp.weixin.qq.com/s/CGDuMoB301Uytnrkc2oyjg)! 你可以在[这里](https://drive.google.com/drive/folders/1Pid6NSFLU43DZRi0EaTcPgXsAzDvbBqF)找到演讲材料.
 - [2025/02] vLLM社区正式创建了[vllm-project/vllm-ascend](https://github.com/vllm-project/vllm-ascend)仓库，让vLLM可以无缝运行在Ascend NPU。
 - [2024/12] 我们正在与 vLLM 社区合作，以支持 [[RFC]: Hardware pluggable](https://github.com/vllm-project/vllm/issues/11162).
@@ -44,10 +47,16 @@ vLLM 昇腾插件 (`vllm-ascend`) 是一个由社区维护的让vLLM在Ascend NP
 
 ## 开始使用
 
-请查看[快速开始](https://vllm-ascend.readthedocs.io/en/latest/quick_start.html)和[安装指南](https://vllm-ascend.readthedocs.io/en/latest/installation.html)了解更多.
+推荐您使用以下版本快速开始使用：
+
+| Version    | Release type | Doc                                  |
+|------------|--------------|--------------------------------------|
+|v0.9.2rc1| 最新RC版本 |请查看[快速开始](https://vllm-ascend.readthedocs.io/en/latest/quick_start.html)和[安装指南](https://vllm-ascend.readthedocs.io/en/latest/installation.html)了解更多|
+|v0.9.1rc3| 下一个正式/稳定版 |[快速开始](https://vllm-ascend.readthedocs.io/en/v0.9.1-dev/quick_start.html) and [安装指南](https://vllm-ascend.readthedocs.io/en/v0.9.1-dev/installation.html)了解更多|
+|v0.7.3.post1| 最新正式/稳定版本 |请查看[快速开始](https://vllm-ascend.readthedocs.io/en/stable/quick_start.html)和[安装指南](https://vllm-ascend.readthedocs.io/en/stable/installation.html)了解更多|
 
 ## 贡献
-请参考 [CONTRIBUTING]((https://vllm-ascend.readthedocs.io/en/main/developer_guide/contributing.html)) 文档了解更多关于开发环境搭建、功能测试以及 PR 提交规范的信息。
+请参考 [CONTRIBUTING]((https://vllm-ascend.readthedocs.io/en/latest/developer_guide/contribution/index.html)) 文档了解更多关于开发环境搭建、功能测试以及 PR 提交规范的信息。
 
 我们欢迎并重视任何形式的贡献与合作：
 - 请通过[Issue](https://github.com/vllm-project/vllm-ascend/issues)来告知我们您遇到的任何Bug。
@@ -65,9 +74,10 @@ vllm-ascend有主干分支和开发分支。
 |------------|------------|---------------------|
 | main       | Maintained | 基于vLLM main分支CI看护   |
 | v0.7.1-dev | Unmaintained | 只允许文档修复 |
-| v0.7.3-dev | Maintained | 基于vLLM v0.7.3版本CI看护 |
+| v0.7.3-dev | Maintained | 基于vLLM v0.7.3版本CI看护, 只允许Bug修复，不会再发布新版本 |
+| v0.9.1-dev | Maintained | 基于vLLM v0.9.1版本CI看护 |
 
-请参阅[版本策略](https://vllm-ascend.readthedocs.io/en/main/developer_guide/versioning_policy.html)了解更多详细信息。
+请参阅[版本策略](https://vllm-ascend.readthedocs.io/en/latest/community/versioning_policy.html)了解更多详细信息。
 
 ## 社区例会
 

{vllm_ascend-0.9.1rc2 → vllm_ascend-0.9.1rc3}/csrc/kernels/pos_encoding_kernels.cpp

@@ -15,11 +15,6 @@
  */
 
 #include "kernel_operator.h"
-#include "kernel_tpipe_impl.h"
-#include "kernel_tensor_impl.h"
-#include "kernel_type.h"
-#include "kernel_operator_intf.h"
-#include "inner_interface/inner_kernel_operator_intf.h"
 #include <stdio.h>
 #include "types.h"
 #include "utils.h"

{vllm_ascend-0.9.1rc2 → vllm_ascend-0.9.1rc3}/docs/source/community/versioning_policy.md

@@ -23,6 +23,8 @@ Following is the Release Compatibility Matrix for vLLM Ascend Plugin:
 | vLLM Ascend | vLLM         | Python           | Stable CANN | PyTorch/torch_npu  | MindIE Turbo |
 |-------------|--------------|------------------|-------------|--------------------|--------------|
 | v0.9.2rc1   | v0.9.2       | >= 3.9, < 3.12   | 8.1.RC1     | 2.5.1 / 2.5.1.post1.dev20250619      |              |
+| v0.9.1rc3   | v0.9.1       | >= 3.9, < 3.12   | 8.1.RC1     | 2.5.1 / 2.5.1.post1 |              |
+| v0.9.1rc2   | v0.9.1       | >= 3.9, < 3.12   | 8.1.RC1     | 2.5.1 / 2.5.1.post1|              |
 | v0.9.1rc1   | v0.9.1       | >= 3.9, < 3.12   | 8.1.RC1     | 2.5.1 / 2.5.1.post1.dev20250528      |              |
 | v0.9.0rc2   | v0.9.0       | >= 3.9, < 3.12   | 8.1.RC1     | 2.5.1 / 2.5.1      |              |
 | v0.9.0rc1   | v0.9.0       | >= 3.9, < 3.12   | 8.1.RC1     | 2.5.1 / 2.5.1      |              |
@@ -37,6 +39,8 @@ Following is the Release Compatibility Matrix for vLLM Ascend Plugin:
 
 | Date       | Event                                     |
 |------------|-------------------------------------------|
+| 2025.08.22 | Release candidates, v0.9.1rc3             |
+| 2025.08.06 | Release candidates, v0.9.1rc2             |
 | 2025.07.11 | Release candidates, v0.9.2rc1             |
 | 2025.06.22 | Release candidates, v0.9.1rc1             |
 | 2025.06.10 | Release candidates, v0.9.0rc2             |

{vllm_ascend-0.9.1rc2 → vllm_ascend-0.9.1rc3}/docs/source/conf.py

@@ -65,15 +65,15 @@ myst_substitutions = {
     # the branch of vllm, used in vllm clone
     # - main branch: 'main'
     # - vX.Y.Z branch: 'vX.Y.Z'
-    'vllm_version': 'v0.9.0',
+    'vllm_version': 'v0.9.1',
     # the branch of vllm-ascend, used in vllm-ascend clone and image tag
     # - main branch: 'main'
     # - vX.Y.Z branch: latest vllm-ascend release tag
-    'vllm_ascend_version': 'v0.9.0rc2',
+    'vllm_ascend_version': 'v0.9.1rc3',
     # the newest release version of vllm-ascend and matched vLLM, used in pip install.
     # This value should be updated when cut down release.
-    'pip_vllm_ascend_version': "0.9.0rc2",
-    'pip_vllm_version': "0.9.0",
+    'pip_vllm_ascend_version': "0.9.1rc3",
+    'pip_vllm_version': "0.9.1",
     # CANN image tag
     'cann_image_tag': "8.2.rc1-910b-ubuntu22.04-py3.11",
 }

vllm_ascend-0.9.1rc3/docs/source/developer_guide/performance/distributed_dp_server_with_large_ep.md

@@ -0,0 +1,253 @@
+# Distributed DP Server With Large EP (DeepSeek)
+
+## Getting Start
+
+vLLM-Ascend now supports prefill-decode (PD) disaggregation in the large **Expert  Parallelism (EP)** scenario. To achieve better performance，the distributed DP server is applied in vLLM-Ascend. In the PD separation scenario, different optimization strategies can be implemented based on the distinct characteristics of PD nodes, thereby enabling more flexible model deployment.
+
+## Verify Multi-Node Communication Environment
+
+### Physical Layer Requirements:
+
+- The physical machines must be located on the same WLAN, with network connectivity.
+- All NPUs are connected with optical modules, and the connection status must be normal.
+
+### Verification Process:
+
+Execute the following commands on each node in sequence. The results must all be `success` and the status must be `UP`:
+
+```bash
+ # Check the remote switch ports
+ for i in {0..15}; do hccn_tool -i $i -lldp -g | grep Ifname; done 
+ # Get the link status of the Ethernet ports (UP or DOWN)
+ for i in {0..15}; do hccn_tool -i $i -link -g ; done
+ # Check the network health status
+ for i in {0..15}; do hccn_tool -i $i -net_health -g ; done
+ # View the network detected IP configuration
+ for i in {0..15}; do hccn_tool -i $i -netdetect -g ; done
+ # View gateway configuration
+ for i in {0..15}; do hccn_tool -i $i -gateway -g ; done
+ # View NPU network configuration
+ cat /etc/hccn.conf
+```
+
+### NPU Interconnect Verification:
+
+#### 1. Get NPU IP Addresses
+
+```bash
+for i in {0..15}; do hccn_tool -i $i -vnic -g;done
+```
+
+#### 2. Get superpodid and SDID
+
+```bash
+for i in {0..7}; do npu-smi info -t spod-info -i $i -c 0;npu-smi info -t spod-info -i $i -c 1;done
+```
+
+#### 3. Cross-Node PING Test
+
+```bash
+# Execute on the target node (replace with actual IP)
+for i in {0..15}; do hccn_tool -i $i -hccs_ping -g address x.x.x.x;done
+```
+
+## Generate Ranktable
+
+You need to generate a ranktable to make  mulit nodes to communicate with each other. For more details please refer to the [vllm-ascend examples](https://github.com/vllm-project/vllm-ascend/blob/v0.9.1-dev/examples/disaggregate_prefill_v1/README.md). Execute the following commands for reference.
+
+```shell
+cd vllm-ascend/examples/disaggregate_prefill_v1/
+bash gen_ranktable.sh --ips prefiller_node1_local_ip prefiller_node2_local_ip decoder_node1_local_ip decoder_node2_local_ip \
+  --npus-per-node  npu_clips --network-card-name nic_name --prefill-device-cnt prefiller_npu_clips --decode-device-cnt decode_npu_clips
+```
+
+|Parameter  | meaning |
+| --- | --- |
+| --ips | Each node's local ip (prefiller nodes should be front of decoder nodes) |
+| --npus-per-node | Each node's npu clips  |
+| --network-card-name | The physical machines' NIC |
+|--prefill-device-cnt  | Npu clips used for prefill |
+|--decode-device-cnt |Npu clips used for decode |
+
+## Use the Distributed DP Server
+
+Execute the following commands to use the distributed DP server. (We recommend using this feature on the v0.9.1-dev branch)
+
+```python
+import multiprocessing
+import os
+import sys
+dp_size = "total number of DP workers for decode/prefill"
+dp_size_local = "number of DP workers on the current node"
+dp_rank_start = "starting DP rank for the current node"
+dp_ip = "master node ip"
+dp_port = "port used for communication"
+engine_port = "the starting port for all DP groups on the current node"
+template_path = "./run_dp_template.sh"
+if not os.path.exists(template_path):
+  print(f"Template file {template_path} does not exist.")
+  sys.exit(1)
+def run_command(dp_rank_local, dp_rank, engine_port_):
+  command = f"bash ./run_dp_template.sh {dp_size} {dp_ip} {dp_port} {dp_rank_local} {dp_rank} {engine_port_} {dp_size_local}"
+  os.system(command)
+processes = []
+for i in range(dp_size_local):
+  dp_rank = dp_rank_start + i
+  dp_rank_local = i
+  engine_port_ = engine_port + i
+  process = multiprocessing.Process(target=run_command, args=(dp_rank_local, dp_rank, engine_port_))
+  processes.append(process)
+  process.start()
+for process in processes:
+  process.join()
+```
+
+Note that the prefiller nodes and the decoder nodes may have differenet configurations. You can use the following shell script for configuring the prefiller and decoder nodes respectively.
+
+```shell
+# run_dp_template.sh
+#!/bin/sh
+
+# this obtained through ifconfig
+# nic_name is the network interface name corresponding to local_ip
+nic_name="xxxx"
+local_ip="xxxx"
+
+# basic configuration for HCCL and connection
+export HCCL_IF_IP=$local_ip
+export GLOO_SOCKET_IFNAME=$nic_name
+export TP_SOCKET_IFNAME=$nic_name
+export HCCL_SOCKET_IFNAME=$nic_name
+export OMP_PROC_BIND=false
+export OMP_NUM_THREADS=10
+export HCCL_BUFFSIZE=256
+export DISAGGREGATED_PREFILL_RANK_TABLE_PATH='ranktable you generate'
+
+# obtain parameters from distributed DP server
+export VLLM_DP_SIZE=$1
+export VLLM_DP_MASTER_IP=$2
+export VLLM_DP_MASTER_PORT=$3
+export VLLM_DP_RANK_LOCAL=$4
+export VLLM_DP_RANK=$5
+export VLLM_DP_SIZE_LOCAL=$7
+
+#pytorch_npu settings and vllm settings
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export TASK_QUEUE_ENABLE=1
+export VLLM_USE_V1=1
+
+# enable the distributed DP server 
+export VLLM_WORKER_MULTIPROC_METHOD="fork"
+export VLLM_ASCEND_EXTERNAL_DP_LB_ENABLED=1
+
+# The w8a8 weight can obtained from https://www.modelscope.cn/models/vllm-ascend/DeepSeek-R1-W8A8
+# "--additional-config" is used to enable characteristics from vllm-ascend
+vllm serve /root/.cache/ds_r1 \
+    --host 0.0.0.0 \
+    --port $6 \
+    --tensor-parallel-size 8 \
+    --enable-expert-parallel \
+    --seed 1024 \
+    --served-model-name deepseek_r1 \
+    --max-model-len 17000 \
+    --max-num-batched-tokens 16384 \
+    --trust-remote-code \
+    --max-num-seqs 4 \
+    --gpu-memory-utilization 0.9 \
+    --quantization ascend \
+    --speculative-config '{"num_speculative_tokens": 1, "method":"deepseek_mtp"}' \
+    --kv-transfer-config \
+        '{"kv_connector": "LLMDataDistCMgrConnector",
+        "kv_buffer_device": "npu",
+        "kv_role": "kv_consumer",
+        "kv_parallel_size": "1",
+        "kv_port": "20001",
+        "engine_id": "0",
+        "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
+        }' \
+    --additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled":true}}'
+```
+
+In the PD separation scenario, we provide a recommended optimized configuration.
+
+- **prefiller node**
+
+1. set HCCL_BUFFSIZE=256
+2. add '--enforce-eager' command to 'vllm serve'
+3. Take '--additional-config' as follow
+
+```shell
+--additional-config '{"ascend_scheduler_config":{"enabled":false}, "torchair_graph_config":{"enabled":false},"enable_weight_nz_layout":true,"enable_prefill_optimizations":true}'
+```
+
+- **decoder node**
+
+1. set HCCL_BUFFSIZE=1024
+2. Take '--additional-config' as follow
+
+```shell
+--additional-config '{"ascend_scheduler_config":{"enabled":false}, "torchair_graph_config":{"enabled":true,"enable_multistream_mla":true,"enable_multistream_moe":true,"graph_batch_sizes":[28], "enable_super_kernel":true, "use_cached_graph":true},"enable_weight_nz_layout":true}'
+```
+
+<br>
+
+'--additional-config'  Parameter Introduction:
+
+- **"torchair_graph_config"：** The config options for torchair graph mode.
+- **"ascend_scheduler_config"：** The config options for ascend scheduler.
+- **"enable_weight_nz_layout"：** Whether to convert quantized weights to NZ format to accelerate matrix multiplication.
+- **"enable_prefill_optimizations"：** Whether to enable DeepSeek models' prefill optimizations.
+  <br>
+
+"torchair_graph_config" Parameter Introduction:
+
+- **"enable_multistream_mla"：** Whether to put vector ops of MLA to another stream. This option only takes effects on models using MLA.
+- **"enable_multistream_moe"：** Whether to enable multistream shared expert. This option only takes effects on DeepSeek moe models.
+- **"graph_batch_sizes"：**  The batch size for torchair graph cache.
+- **"enable_super_kernel"：** Whether to enable super kernel.
+- **"use_cached_graph"：** Whether to use cached graph
+
+## Toy proxy for Distributed DP Server
+
+In the PD separation scenario, we need a proxy to distribute requests. Execute the following commands to enable the toy proxy:
+
+```shell
+python load_balance_proxy_server_example.py \
+  --port "proxy port" \
+  --host 0.0.0.0 \
+  --prefiller-hosts \
+    prefiller node1 local ip \
+    prefiller node2 local ip \
+  --prefiller-ports  \
+    engine_port engine_port \
+  --decoder-hosts \
+    decoder node1 local ip  \
+    decoder node1 local ip  \
+    decoder node2 local ip  \
+    decoder node2 local ip  \
+  --decoder-ports  \
+    engine_port ...  \ # Increase by dp_size_local e.g. 9000 9001
+    engine_port ...  \ # Increase by dp_size_local e.g. 9000 9001
+```
+
+:::{note}
+Each node local ip should repeat the same times as its '**dp_size_local**', at the same time, each node has the same number of ports as '**dp_size_local**', and their ports increase sequentially starting from '**engine_port**'.
+:::
+
+You can get the proxy program in the repository's examples, [load\_balance\_proxy\_server\_example.py](https://github.com/vllm-project/vllm-ascend/blob/v0.9.1-dev/examples/disaggregate_prefill_v1/load_balance_proxy_server_example.py)
+
+## Recommended Configuration
+
+For example，if the average input length is 3.5k, and the output length is 1.1k, the context length is 16k, the max length of the input dataset is 7K. In this scenario, we give a recommended configuration for distributed DP server with high EP. Here we use 4 nodes for prefill and 4 nodes for decode.
+<br>
+
+| node     | DP | TP | EP | max-model-len | max-num-batched-tokens | max-num-seqs |  gpu-memory-utilization |
+|----------|----|----|----|---------------|------------------------|--------------|-----------|
+| prefill  | 2  |  8 | 16 |     17000     |         16384          |      4       |    0.9    |
+| decode   | 64 |  1 | 64 |     17000     |          256           |      28      |    0.9    |
+
+## FAQ
+
+### 1. Prefiller nodes need to warmup
+
+Since the computation of some NPU operators requires several rounds of warm-up to achieve best performance, we recommend preheating the service with some requests before conducting performance tests to achieve the best end-to-end throughput.

{vllm_ascend-0.9.1rc2 → vllm_ascend-0.9.1rc3}/docs/source/developer_guide/performance/index.md

@@ -6,4 +6,5 @@
 performance_benchmark
 profile_execute_duration
 optimization_and_tuning
+distributed_dp_server_with_large_ep
 :::

{vllm_ascend-0.9.1rc2 → vllm_ascend-0.9.1rc3}/docs/source/faqs.md

@@ -3,7 +3,7 @@
 ## Version Specific FAQs
 
 - [[v0.7.3.post1] FAQ & Feedback](https://github.com/vllm-project/vllm-ascend/issues/1007)
-- [[v0.9.0rc2] FAQ & Feedback](https://github.com/vllm-project/vllm-ascend/issues/1115)
+- [[v0.9.1rc3] FAQ & Feedback](https://github.com/vllm-project/vllm-ascend/issues/2410)
 
 ## General FAQs
 
@@ -158,12 +158,29 @@ for output in outputs:
 2. Set the following enveriments parameters:
 
 ```bash
-export LCCL_DETERMINISTIC = 1
-export HCCL_DETERMINISTIC = 1
-export ATB_MATMUL_SHUFFLE_K_ENABLE = 0
-export ATB_LLM_LCOC_ENABLE = 0
+export LCCL_DETERMINISTIC=1
+export HCCL_DETERMINISTIC=true
+export ATB_MATMUL_SHUFFLE_K_ENABLE=0
+export ATB_LLM_LCOC_ENABLE=0
 ```
 
 ### 19. How to fix the error "ImportError: Please install vllm[audio] for audio support" for Qwen2.5-Omni model？
 The `Qwen2.5-Omni` model requires the `librosa` package to be installed, you need to install the `qwen-omni-utils` package to ensure all dependencies are met `pip install qwen-omni-utils`,
 this package will install `librosa` and its related dependencies, resolving the `ImportError: No module named 'librosa'` issue and ensuring audio processing functionality works correctly.
+
+### 20. Failed to run with `ray` distributed backend?
+You might facing the following errors when running with ray backend in distributed scenarios:
+
+```
+TypeError: can't convert npu:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
+```
+
+```
+AttributeError: 'str' object has no attribute 'DESCRIPTOR' when packaging message to dict
+```
+
+This has been solved in `ray>=2.47.1`, thus we could solve this as following:
+
+```
+python3 -m pip install modelscope 'ray>=2.47.1' 'protobuf>3.20.0'
+``` 

{vllm_ascend-0.9.1rc2 → vllm_ascend-0.9.1rc3}/docs/source/user_guide/configuration/additional_config.md

@@ -30,7 +30,6 @@ The following table lists the additional configuration options available in vLLM
 | `ascend_scheduler_config`     | dict | `{}` | The config options for ascend scheduler                                                       |
 | `refresh`                     | bool | `false` | Whether to refresh global ascend config content. This value is usually used by rlhf or ut/e2e test case.     |
 | `expert_map_path`             | str  | `None` | When using expert load balancing for the MOE model, an expert map path needs to be passed in. |
-| `chunked_prefill_for_mla`     | bool | `False` | Whether to enable the fused operator-like chunked_prefill. |
 | `kv_cache_dtype`     | str | `None` | When using the kv cache quantization method, kv cache dtype needs to be set, currently only int8 is supported. |
 
 The details of each config option are as follows: