xfuser 0.3.2__tar.gz → 0.3.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {xfuser-0.3.2 → xfuser-0.3.3}/PKG-INFO +102 -61
- {xfuser-0.3.2 → xfuser-0.3.3}/README.md +100 -59
- {xfuser-0.3.2 → xfuser-0.3.3}/setup.py +1 -1
- xfuser-0.3.3/xfuser/__version__.py +1 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/config/args.py +74 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/config/config.py +21 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/distributed/runtime_state.py +5 -5
- xfuser-0.3.3/xfuser/core/fast_attention/__init__.py +37 -0
- xfuser-0.3.3/xfuser/core/fast_attention/attn_layer.py +220 -0
- xfuser-0.3.3/xfuser/core/fast_attention/fast_attn_state.py +111 -0
- xfuser-0.3.3/xfuser/core/fast_attention/utils.py +232 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/long_ctx_attention/hybrid/attn_layer.py +16 -1
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/base_wrapper.py +2 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/layers/attention_processor.py +41 -25
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/layers/embeddings.py +42 -18
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/models/transformers/base_transformer.py +80 -33
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/models/transformers/cogvideox_transformer_3d.py +1 -1
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/models/transformers/hunyuan_transformer_2d.py +2 -1
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/models/transformers/transformer_flux.py +37 -15
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/models/transformers/transformer_sd3.py +1 -1
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/pipelines/base_pipeline.py +118 -8
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/pipelines/pipeline_cogvideox.py +15 -37
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/pipelines/pipeline_flux.py +315 -41
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/pipelines/pipeline_hunyuandit.py +15 -4
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/pipelines/pipeline_latte.py +3 -2
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/pipelines/pipeline_pixart_alpha.py +16 -4
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/pipelines/pipeline_pixart_sigma.py +18 -4
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/pipelines/pipeline_stable_diffusion_3.py +21 -10
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser.egg-info/PKG-INFO +102 -61
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser.egg-info/SOURCES.txt +4 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser.egg-info/requires.txt +1 -1
- xfuser-0.3.2/xfuser/__version__.py +0 -1
- {xfuser-0.3.2 → xfuser-0.3.3}/LICENSE.txt +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/comfyui-xdit/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/comfyui-xdit/host.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/comfyui-xdit/launch_host.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/comfyui-xdit/nodes.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/comfyui-xdit/utils.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/setup.cfg +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/config/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/cache_manager/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/cache_manager/cache_manager.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/distributed/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/distributed/group_coordinator.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/distributed/parallel_state.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/distributed/utils.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/long_ctx_attention/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/long_ctx_attention/hybrid/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/long_ctx_attention/hybrid/utils.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/long_ctx_attention/ring/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/long_ctx_attention/ring/ring_flash_attn.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/long_ctx_attention/ulysses/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/long_ctx_attention/ulysses/attn_layer.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/utils/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/core/utils/timer.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/envs.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/logger.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/layers/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/layers/base_layer.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/layers/conv.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/layers/feedforward.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/layers/register.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/models/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/models/base_model.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/models/transformers/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/models/transformers/latte_transformer_3d.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/models/transformers/pixart_transformer_2d.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/models/transformers/register.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/pipelines/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/pipelines/register.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/schedulers/__init__.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/schedulers/base_scheduler.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/schedulers/register.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/schedulers/scheduling_ddim.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/schedulers/scheduling_ddim_cogvideox.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/schedulers/scheduling_ddpm.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/schedulers/scheduling_dpm_cogvideox.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/schedulers/scheduling_dpmsolver_multistep.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/model_executor/schedulers/scheduling_flow_match_euler_discrete.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser/parallel.py +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser.egg-info/dependency_links.txt +0 -0
- {xfuser-0.3.2 → xfuser-0.3.3}/xfuser.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: xfuser
|
|
3
|
-
Version: 0.3.
|
|
3
|
+
Version: 0.3.3
|
|
4
4
|
Summary: xDiT: A Scalable Inference Engine for Diffusion Transformers (DiTs) on multi-GPU Clusters
|
|
5
5
|
Home-page: https://github.com/xdit-project/xDiT.
|
|
6
6
|
Author: xDiT Team
|
|
@@ -12,7 +12,7 @@ Description-Content-Type: text/markdown
|
|
|
12
12
|
License-File: LICENSE.txt
|
|
13
13
|
Requires-Dist: torch>=2.1.0
|
|
14
14
|
Requires-Dist: accelerate>=0.33.0
|
|
15
|
-
Requires-Dist: diffusers>=0.
|
|
15
|
+
Requires-Dist: diffusers>=0.31
|
|
16
16
|
Requires-Dist: transformers>=4.39.1
|
|
17
17
|
Requires-Dist: sentencepiece>=0.1.99
|
|
18
18
|
Requires-Dist: beautifulsoup4>=4.12.3
|
|
@@ -29,12 +29,11 @@ Requires-Dist: flash_attn>=2.6.3; extra == "flash-attn"
|
|
|
29
29
|
<p align="center">
|
|
30
30
|
|
|
31
31
|
<picture>
|
|
32
|
-
<img alt="xDiT" src="
|
|
33
|
-
</picture>
|
|
32
|
+
<img alt="xDiT" src="https://raw.githubusercontent.com/xdit-project/xdit_assets/main/XDiTlogo.png" width="50%">
|
|
34
33
|
|
|
35
34
|
</p>
|
|
36
35
|
<h3>A Scalable Inference Engine for Diffusion Transformers (DiTs) on multi-GPU Clusters</h3>
|
|
37
|
-
<strong><a href="https://arxiv.org/abs/2405.14430">📃 Paper</a> | <a href="#QuickStart">🚀 Quick Start</a> | <a href="#support-dits">🎯 Supported DiTs</a> | <a href="#dev-guide">📚 Dev Guide </a> | <a href="https://github.com/xdit-project/xDiT/discussions">📈 Discussion </a> </strong>
|
|
36
|
+
<strong><a href="https://arxiv.org/abs/2405.14430">📃 Paper</a> | <a href="#QuickStart">🚀 Quick Start</a> | <a href="#support-dits">🎯 Supported DiTs</a> | <a href="#dev-guide">📚 Dev Guide </a> | <a href="https://github.com/xdit-project/xDiT/discussions">📈 Discussion </a> | <a href="https://medium.com/@xditproject">📝 Blogs</a></strong>
|
|
38
37
|
<p></p>
|
|
39
38
|
|
|
40
39
|
[](https://discord.gg/YEWzWfCF9S)
|
|
@@ -47,12 +46,14 @@ Requires-Dist: flash_attn>=2.6.3; extra == "flash-attn"
|
|
|
47
46
|
- [📢 Updates](#updates)
|
|
48
47
|
- [🎯 Supported DiTs](#support-dits)
|
|
49
48
|
- [📈 Performance](#perf)
|
|
49
|
+
- [CogVideoX](#perf_cogvideox)
|
|
50
50
|
- [Flux.1](#perf_flux)
|
|
51
51
|
- [HunyuanDiT](#perf_hunyuandit)
|
|
52
52
|
- [SD3](#perf_sd3)
|
|
53
53
|
- [Pixart](#perf_pixart)
|
|
54
54
|
- [Latte](#perf_latte)
|
|
55
55
|
- [🚀 QuickStart](#QuickStart)
|
|
56
|
+
- [🖼️ ComfyUI with xDiT](#comfyui)
|
|
56
57
|
- [✨ xDiT's Arsenal](#secrets)
|
|
57
58
|
- [Parallel Methods](#parallel)
|
|
58
59
|
- [1. PipeFusion](#PipeFusion)
|
|
@@ -60,7 +61,9 @@ Requires-Dist: flash_attn>=2.6.3; extra == "flash-attn"
|
|
|
60
61
|
- [3. Hybrid Parallel](#hybrid_parallel)
|
|
61
62
|
- [4. CFG Parallel](#cfg_parallel)
|
|
62
63
|
- [5. Parallel VAE](#parallel_vae)
|
|
63
|
-
- [
|
|
64
|
+
- [Single GPU Acceleration](#1gpuacc)
|
|
65
|
+
- [Compilation Acceleration](#compilation)
|
|
66
|
+
- [DiTFastAttn](#dittfastattn)
|
|
64
67
|
- [📚 Develop Guide](#dev-guide)
|
|
65
68
|
- [🚧 History and Looking for Contributions](#history)
|
|
66
69
|
- [📝 Cite Us](#cite-us)
|
|
@@ -72,14 +75,23 @@ Diffusion Transformers (DiTs) are driving advancements in high-quality image and
|
|
|
72
75
|
With the escalating input context length in DiTs, the computational demand of the Attention mechanism grows **quadratically**!
|
|
73
76
|
Consequently, multi-GPU and multi-machine deployments are essential to meet the **real-time** requirements in online services.
|
|
74
77
|
|
|
78
|
+
|
|
79
|
+
<h3 id="meet-xdit-parallel">Parallel Inference</h3>
|
|
80
|
+
|
|
75
81
|
To meet real-time demand for DiTs applications, parallel inference is a must.
|
|
76
82
|
xDiT is an inference engine designed for the parallel deployment of DiTs on large scale.
|
|
77
|
-
xDiT provides a suite of efficient parallel approaches for Diffusion Models, as well as
|
|
83
|
+
xDiT provides a suite of efficient parallel approaches for Diffusion Models, as well as computation accelerations.
|
|
84
|
+
|
|
85
|
+
The overview of xDiT is shown as follows.
|
|
78
86
|
|
|
87
|
+
<picture>
|
|
88
|
+
<img alt="xDiT" src="https://raw.githubusercontent.com/xdit-project/xdit_assets/main/methods/xdit_overview.png">
|
|
89
|
+
</picture>
|
|
79
90
|
|
|
80
|
-
1. Sequence Parallelism, [USP](https://arxiv.org/abs/2405.07719) is a unified sequence parallel approach combining DeepSpeed-Ulysses, Ring-Attention.
|
|
81
91
|
|
|
82
|
-
|
|
92
|
+
1. Sequence Parallelism, [USP](https://arxiv.org/abs/2405.07719) is a unified sequence parallel approach combining DeepSpeed-Ulysses, Ring-Attention proposed by use3.
|
|
93
|
+
|
|
94
|
+
2. [PipeFusion](https://arxiv.org/abs/2405.14430), a sequence-level pipeline parallelism, similar to [TeraPipe](https://arxiv.org/abs/2102.07988) but takes advantage of the input temporal redundancy characteristics of diffusion models.
|
|
83
95
|
|
|
84
96
|
3. Data Parallel: Processes multiple prompts or generates multiple images from a single prompt in parallel across images.
|
|
85
97
|
|
|
@@ -96,28 +108,27 @@ We also have implemented the following parallel stategies for reference:
|
|
|
96
108
|
2. [DistriFusion](https://arxiv.org/abs/2402.19481)
|
|
97
109
|
|
|
98
110
|
|
|
99
|
-
|
|
100
|
-
In addition to utilizing well-known Attention optimization libraries, we leverage compilation acceleration technologies such as `torch.compile` and `onediff`.
|
|
111
|
+
<h3 id="meet-xdit-perf">Computing Acceleration</h3>
|
|
101
112
|
|
|
102
|
-
|
|
113
|
+
Optimization orthogonal to parallel focuses on accelerating single GPU performance.
|
|
103
114
|
|
|
104
|
-
|
|
105
|
-
<img alt="xDiT" src="./assets/methods/xdit_overview.png">
|
|
106
|
-
</picture>
|
|
115
|
+
First, xDiT employs a series of kernel acceleration methods. In addition to utilizing well-known Attention optimization libraries, we leverage compilation acceleration technologies such as `torch.compile` and `onediff`.
|
|
107
116
|
|
|
117
|
+
Furthermore, xDiT incorporates optimization techniques from [DiTFastAttn](https://github.com/thu-nics/DiTFastAttn), which exploits computational redundancies between different steps of the Diffusion Model to accelerate inference on a single GPU.
|
|
108
118
|
|
|
109
119
|
<h2 id="updates">📢 Updates</h2>
|
|
110
120
|
|
|
111
|
-
*
|
|
121
|
+
* 🎉**October 10, 2024**: xDiT applied DiTFastAttn to accelerate single GPU inference for Pixart Models!
|
|
122
|
+
* 🎉**September 26, 2024**: xDiT has been officially used by [THUDM/CogVideo](https://github.com/THUDM/CogVideo)! The inference scripts are placed in [parallel_inference/](https://github.com/THUDM/CogVideo/blob/main/tools/parallel_inference) at their repository.
|
|
123
|
+
* 🎉**September 23, 2024**: Support CogVideoX. The inference scripts are [examples/cogvideox_example.py](examples/cogvideox_example.py).
|
|
112
124
|
* 🎉**August 26, 2024**: We apply torch.compile and [onediff](https://github.com/siliconflow/onediff) nexfort backend to accelerate GPU kernels speed.
|
|
113
|
-
* 🎉**August
|
|
114
|
-
* 🎉**August
|
|
115
|
-
* 🎉**August
|
|
125
|
+
* 🎉**August 15, 2024**: Support Hunyuan-DiT hybrid parallel version. The inference scripts are [examples/hunyuandit_example.py](examples/hunyuandit_example.py).
|
|
126
|
+
* 🎉**August 9, 2024**: Support Latte sequence parallel version. The inference scripts are [examples/latte_example.py](examples/latte_example.py).
|
|
127
|
+
* 🎉**August 8, 2024**: Support Flux sequence parallel version. The inference scripts are [examples/flux_example.py](examples/flux_example.py).
|
|
128
|
+
* 🎉**August 2, 2024**: Support Stable Diffusion 3 hybrid parallel version. The inference scripts are [examples/sd3_example.py](examples/sd3_example.py).
|
|
116
129
|
* 🎉**July 18, 2024**: Support PixArt-Sigma and PixArt-Alpha. The inference scripts are [examples/pixartsigma_example.py](examples/pixartsigma_example.py), [examples/pixartalpha_example.py](examples/pixartalpha_example.py).
|
|
117
130
|
* 🎉**July 17, 2024**: Rename the project to xDiT. The project has evolved from a collection of parallel methods into a unified inference framework and supported the hybrid parallel for DiTs.
|
|
118
|
-
* 🎉**
|
|
119
|
-
* 🎉**June 26, 2024**: Support Stable Diffusion 3. The inference script is [legacy/scripts/sd3_example.py](./legacy/scripts/sd3_example.py).
|
|
120
|
-
* 🎉**May 24, 2024**: PipeFusion is public released. It supports PixArt-alpha [legacy/scripts/pixart_example.py](./legacy/scripts/pixart_example.py), DiT [legacy/scripts/ditxl_example.py](./legacy/scripts/ditxl_example.py) and SDXL [legacy/scripts/sdxl_example.py](./legacy/scripts/sdxl_example.py).
|
|
131
|
+
* 🎉**May 24, 2024**: PipeFusion is public released. It supports PixArt-alpha [scripts/pixart_example.py](./scripts/pixart_example.py), DiT [scripts/ditxl_example.py](./scripts/ditxl_example.py) and SDXL [scripts/sdxl_example.py](./scripts/sdxl_example.py). This version is currently in the `legacy` branch.
|
|
121
132
|
|
|
122
133
|
|
|
123
134
|
<h2 id="support-dits">🎯 Supported DiTs</h2>
|
|
@@ -126,10 +137,10 @@ The overview of xDiT is shown as follows.
|
|
|
126
137
|
|
|
127
138
|
| Model Name | CFG | SP | PipeFusion |
|
|
128
139
|
| --- | --- | --- | --- |
|
|
129
|
-
| [🎬 CogVideoX](https://huggingface.co/THUDM/CogVideoX-2b) |
|
|
140
|
+
| [🎬 CogVideoX](https://huggingface.co/THUDM/CogVideoX-2b) | ✔️ | ✔️ | ❎ |
|
|
130
141
|
| [🎬 Latte](https://huggingface.co/maxin-cn/Latte-1) | ❎ | ✔️ | ❎ |
|
|
131
142
|
| [🔵 HunyuanDiT-v1.2-Diffusers](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2-Diffusers) | ✔️ | ✔️ | ✔️ |
|
|
132
|
-
| [🟠 Flux](https://huggingface.co/black-forest-labs/FLUX.1-schnell) | NA | ✔️ |
|
|
143
|
+
| [🟠 Flux](https://huggingface.co/black-forest-labs/FLUX.1-schnell) | NA | ✔️ | ✔️ |
|
|
133
144
|
| [🔴 PixArt-Sigma](https://huggingface.co/PixArt-alpha/PixArt-Sigma-XL-2-1024-MS) | ✔️ | ✔️ | ✔️ |
|
|
134
145
|
| [🟢 PixArt-alpha](https://huggingface.co/PixArt-alpha/PixArt-alpha) | ✔️ | ✔️ | ✔️ |
|
|
135
146
|
| [🟠 Stable Diffusion 3](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers) | ✔️ | ✔️ | ✔️ |
|
|
@@ -145,30 +156,34 @@ The overview of xDiT is shown as follows.
|
|
|
145
156
|
|
|
146
157
|
<h2 id="perf">📈 Performance</h2>
|
|
147
158
|
|
|
159
|
+
<h3 id="perf_cogvideox">CogVideo</h3>
|
|
160
|
+
|
|
161
|
+
1. [CogVideo Performance Report](./docs/performance/cogvideo.md)
|
|
162
|
+
|
|
148
163
|
<h3 id="perf_flux">Flux.1</h3>
|
|
149
164
|
|
|
150
|
-
|
|
165
|
+
2. [Flux Performance Report](./docs/performance/flux.md)
|
|
166
|
+
|
|
167
|
+
<h3 id="perf_latte">Latte</h3>
|
|
168
|
+
|
|
169
|
+
3. [Latte Performance Report](./docs/performance/latte.md)
|
|
151
170
|
|
|
152
171
|
<h3 id="perf_hunyuandit">HunyuanDiT</h3>
|
|
153
172
|
|
|
154
|
-
|
|
173
|
+
4. [HunyuanDiT Performance Report](./docs/performance/hunyuandit.md)
|
|
155
174
|
|
|
156
175
|
<h3 id="perf_sd3">SD3</h3>
|
|
157
176
|
|
|
158
|
-
|
|
177
|
+
5. [Stable Diffusion 3 Performance Report](./docs/performance/sd3.md)
|
|
159
178
|
|
|
160
179
|
<h3 id="perf_pixart">Pixart</h3>
|
|
161
180
|
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
<h3 id="perf_latte">Pixart</h3>
|
|
165
|
-
|
|
166
|
-
5. [Latte Performance Report](./docs/performance/latte.md)
|
|
181
|
+
6. [Pixart-Alpha Performance Report (legacy)](./docs/performance/pixart_alpha_legacy.md)
|
|
167
182
|
|
|
168
183
|
|
|
169
184
|
<h2 id="QuickStart">🚀 QuickStart</h2>
|
|
170
185
|
|
|
171
|
-
### 1. Install from pip
|
|
186
|
+
### 1. Install from pip
|
|
172
187
|
|
|
173
188
|
```
|
|
174
189
|
pip install xfuser
|
|
@@ -200,7 +215,10 @@ You can easily modify the model type, model directory, and parallel options in t
|
|
|
200
215
|
bash examples/run.sh
|
|
201
216
|
```
|
|
202
217
|
|
|
203
|
-
|
|
218
|
+
---
|
|
219
|
+
|
|
220
|
+
<details>
|
|
221
|
+
<summary>Click to see available options for the PixArt-alpha example</summary>
|
|
204
222
|
|
|
205
223
|
```bash
|
|
206
224
|
python ./examples/pixartalpha_example.py -h
|
|
@@ -260,10 +278,14 @@ Input Options:
|
|
|
260
278
|
Number of inference steps.
|
|
261
279
|
```
|
|
262
280
|
|
|
281
|
+
</details>
|
|
282
|
+
|
|
283
|
+
---
|
|
284
|
+
|
|
263
285
|
Hybriding multiple parallelism techniques togather is essential for efficiently scaling.
|
|
264
286
|
It's important that the product of all parallel degrees matches the number of devices.
|
|
265
|
-
For instance, you can combine CFG, PipeFusion, and sequence parallelism with the command below to generate an image of a cute dog through hybrid parallelism.
|
|
266
|
-
Here ulysses_degree * pipefusion_parallel_degree * cfg_degree(
|
|
287
|
+
Note use_cfg_parallel means cfg_parallel=2. For instance, you can combine CFG, PipeFusion, and sequence parallelism with the command below to generate an image of a cute dog through hybrid parallelism.
|
|
288
|
+
Here ulysses_degree * pipefusion_parallel_degree * cfg_degree(use_cfg_parallel) == number of devices == 8.
|
|
267
289
|
|
|
268
290
|
|
|
269
291
|
```bash
|
|
@@ -284,14 +306,26 @@ We observed that a warmup of 0 had no effect on the PixArt model.
|
|
|
284
306
|
Users can tune this value according to their specific tasks.
|
|
285
307
|
|
|
286
308
|
|
|
309
|
+
<h2 id="comfyui">🖼️ ComfyUI with xDiT</h2>
|
|
287
310
|
|
|
288
|
-
###
|
|
311
|
+
### 1. Launch ComfyUI
|
|
289
312
|
|
|
290
|
-
|
|
313
|
+
ComfyUI, is the most popular web-based Diffusion Model interface optimized for workflow.
|
|
314
|
+
It provides users with a UI platform for image generation, supporting plugins like LoRA, ControlNet, and IPAdaptor.
|
|
315
|
+
Yet, its design for native single-GPU usage leaves it struggling with the demands of today’s large DiTs, resulting in unacceptably high latency for users like Flux.1. Leveraging the power of xDiT, we’ve successfully implemented a multi-GPU parallel processing workflow within ComfyUI, effectively addressing Flux.1’s performance challenges.
|
|
316
|
+
Below is an example of using xDiT to accelerate a Flux workflow with LoRA:
|
|
317
|
+
|
|
318
|
+

|
|
319
|
+
|
|
320
|
+
More details can be found in our Medium article: [Supercharge Your AIGC Experience: Leverage xDiT for Multiple GPU Parallel in ComfyUI Flux.1 Workflow](https://medium.com/@xditproject/supercharge-your-aigc-experience-leverage-xdit-for-multiple-gpu-parallel-in-comfyui-flux-1-54b34e4bca05).
|
|
321
|
+
|
|
322
|
+
Currently, if you need the parallel version of ComfyUI applying xDiT, please contact us via email [jiaruifang@tencent.com](jiaruifang@tencent.com).
|
|
323
|
+
|
|
324
|
+
### 2. Launch a Http Service
|
|
291
325
|
|
|
292
|
-
|
|
326
|
+
You can also launch a http service to generate images with xDiT.
|
|
293
327
|
|
|
294
|
-
[Launching
|
|
328
|
+
[Launching a Text-to-Image Http Service](./docs/developer/Http_Service.md)
|
|
295
329
|
|
|
296
330
|
|
|
297
331
|
<h2 id="secrets">✨ The xDiT's Arsenal</h2>
|
|
@@ -308,34 +342,32 @@ For the VAE module, xDiT offers a parallel implementation, [DistVAE](https://git
|
|
|
308
342
|
The (<span style="color: red;">xDiT</span>) highlights the methods first proposed by use.
|
|
309
343
|
|
|
310
344
|
<div align="center">
|
|
311
|
-
<img src="
|
|
345
|
+
<img src="https://raw.githubusercontent.com/xdit-project/xdit_assets/main/methods/xdit_method.png" alt="xdit methods">
|
|
312
346
|
</div>
|
|
313
347
|
|
|
314
348
|
The communication and memory costs associated with the aforementioned intra-image parallelism, except for the CFG and DP (they are inter-image parallel), in DiTs are detailed in the table below. (* denotes that communication can be overlapped with computation.)
|
|
315
349
|
|
|
316
350
|
As we can see, PipeFusion and Sequence Parallel achieve lowest communication cost on different scales and hardware configurations, making them suitable foundational components for a hybrid approach.
|
|
317
351
|
|
|
318
|
-
𝒑: Number of pixels
|
|
319
|
-
𝒉𝒔: Model hidden size
|
|
320
|
-
𝑳: Number of model layers
|
|
321
|
-
𝑷: Total model parameters
|
|
322
|
-
𝑵: Number of parallel devices
|
|
323
|
-
𝑴: Number of patch splits
|
|
324
|
-
𝑸𝑶: Query and Output parameter count
|
|
325
|
-
𝑲𝑽: KV Activation parameter count
|
|
352
|
+
𝒑: Number of pixels;\
|
|
353
|
+
𝒉𝒔: Model hidden size;\
|
|
354
|
+
𝑳: Number of model layers;\
|
|
355
|
+
𝑷: Total model parameters;\
|
|
356
|
+
𝑵: Number of parallel devices;\
|
|
357
|
+
𝑴: Number of patch splits;\
|
|
358
|
+
𝑸𝑶: Query and Output parameter count;\
|
|
359
|
+
𝑲𝑽: KV Activation parameter count;\
|
|
326
360
|
𝑨 = 𝑸 = 𝑶 = 𝑲 = 𝑽: Equal parameters for Attention, Query, Output, Key, and Value;
|
|
327
361
|
|
|
328
|
-
<div align="center">
|
|
329
362
|
|
|
330
|
-
|
|
|
331
|
-
|
|
332
|
-
| Tensor Parallel
|
|
333
|
-
| DistriFusion*
|
|
334
|
-
| Ring Sequence Parallel*
|
|
335
|
-
| Ulysses Sequence Parallel | fresh
|
|
336
|
-
| PipeFusion*
|
|
363
|
+
| | attn-KV | communication cost | param memory | activations memory | extra buff memory |
|
|
364
|
+
|:-------------------------:|:-------:|:----------------------------:|:--------------:|:------------------------------:|:----------------------------------:|
|
|
365
|
+
| Tensor Parallel | fresh | $4O(p \times hs)L$ | $\frac{1}{N}P$ | $\frac{2}{N}A = \frac{1}{N}QO$ | $\frac{2}{N}A = \frac{1}{N}KV$ |
|
|
366
|
+
| DistriFusion* | stale | $2O(p \times hs)L$ | $P$ | $\frac{2}{N}A = \frac{1}{N}QO$ | $2AL = (KV)L$ |
|
|
367
|
+
| Ring Sequence Parallel* | fresh | $2O(p \times hs)L$ | $P$ | $\frac{2}{N}A = \frac{1}{N}QO$ | $\frac{2}{N}A = \frac{1}{N}KV$ |
|
|
368
|
+
| Ulysses Sequence Parallel | fresh | $\frac{4}{N}O(p \times hs)L$ | $P$ | $\frac{2}{N}A = \frac{1}{N}QO$ | $\frac{2}{N}A = \frac{1}{N}KV$ |
|
|
369
|
+
| PipeFusion* | stale- | $2O(p \times hs)$ | $\frac{1}{N}P$ | $\frac{2}{M}A = \frac{1}{M}QO$ | $\frac{2L}{N}A = \frac{1}{N}(KV)L$ |
|
|
337
370
|
|
|
338
|
-
</div>
|
|
339
371
|
|
|
340
372
|
<h4 id="PipeFusion">1.1. PipeFusion</h4>
|
|
341
373
|
|
|
@@ -357,7 +389,10 @@ As we can see, PipeFusion and Sequence Parallel achieve lowest communication cos
|
|
|
357
389
|
|
|
358
390
|
[Patch Parallel VAE](./docs/methods/parallel_vae.md)
|
|
359
391
|
|
|
360
|
-
<h3 id="
|
|
392
|
+
<h3 id="1gpuacc">Single GPU Acceleration</h3>
|
|
393
|
+
|
|
394
|
+
|
|
395
|
+
<h4 id="compilation">Compilation Acceleration</h4>
|
|
361
396
|
|
|
362
397
|
We utilize two compilation acceleration techniques, [torch.compile](https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html) and [onediff](https://github.com/siliconflow/onediff), to enhance runtime speed on GPUs. These compilation accelerations are used in conjunction with parallelization methods.
|
|
363
398
|
|
|
@@ -371,6 +406,12 @@ pip install -U nexfort
|
|
|
371
406
|
For usage instructions, refer to the [example/run.sh](./examples/run.sh). Simply append `--use_torch_compile` or `--use_onediff` to your command. Note that these options are mutually exclusive, and their performance varies across different scenarios.
|
|
372
407
|
|
|
373
408
|
|
|
409
|
+
<h4 id="dittfastattn">DiTFastAttn</h4>
|
|
410
|
+
|
|
411
|
+
xDiT also provides DiTFastAttn for single GPU acceleration. It can reduce computation cost of attention layer by leveraging redundancies between different steps of the Diffusion Model.
|
|
412
|
+
|
|
413
|
+
[DiTFastAttn: Attention Compression for Diffusion Transformer Models](./docs/methods/ditfastattn.md)
|
|
414
|
+
|
|
374
415
|
<h2 id="dev-guide">📚 Develop Guide</h2>
|
|
375
416
|
|
|
376
417
|
[The implement and design of xdit framework](./docs/developer/The_implement_design_of_xdit_framework.md)
|
|
@@ -383,9 +424,9 @@ We conducted a major upgrade of this project in August 2024.
|
|
|
383
424
|
|
|
384
425
|
The latest APIs is located in the [xfuser/](./xfuser/) directory, supports hybrid parallelism. It offers clearer and more structured code but currently supports fewer models.
|
|
385
426
|
|
|
386
|
-
The legacy APIs is in the [legacy
|
|
427
|
+
The legacy APIs is in the [legacy](https://github.com/xdit-project/xDiT/tree/legacy) branch, limited to single parallelism. It supports a richer of parallel methods, including PipeFusion, Sequence Parallel, DistriFusion, and Tensor Parallel. CFG Parallel can be hybrid with PipeFusion but not with other parallel methods.
|
|
387
428
|
|
|
388
|
-
For models not yet supported by the latest APIs, you can run the examples in the [
|
|
429
|
+
For models not yet supported by the latest APIs, you can run the examples in the [scripts/](https://github.com/xdit-project/xDiT/tree/legacy/scripts) directory under branch `legacy`. If you wish to develop new features on a model or require hybrid parallelism, stay tuned for further project updates.
|
|
389
430
|
|
|
390
431
|
We also welcome developers to join and contribute more features and models to the project. Tell us which model you need in xDiT in [discussions](https://github.com/xdit-project/xDiT/discussions).
|
|
391
432
|
|