xfuser 0.2__tar.gz

xfuser-0.2/LICENSE.txt

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [2024] [PipeFusion Team]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

xfuser-0.2/PKG-INFO

@@ -0,0 +1,322 @@
+Metadata-Version: 2.1
+Name: xfuser
+Version: 0.2
+Summary: xDiT: A Scalable Inference Engine for Diffusion Transformers (DiTs) on multi-GPU Clusters
+Home-page: https://github.com/xdit-project/xDiT.
+Author: Jiannan Wang, Jiarui Fang, Jinzhe Pan, Aoyu Li, Pengcheng Yang
+Author-email: fangjiarui123@gmail.com
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: torch>=2.2
+Requires-Dist: diffusers>=0.30.0
+Requires-Dist: transformers
+Requires-Dist: sentencepiece
+Requires-Dist: accelerate==0.33.0
+Requires-Dist: beautifulsoup4
+Requires-Dist: distvae
+Requires-Dist: ftfy
+
+<div align="center">
+  <!-- <h1>KTransformers</h1> -->
+  <p align="center">
+  
+  <picture>
+    <img alt="xDiT" src="./assets/XDiTlogo.png" width=50%>
+
+  </picture>
+
+  </p>
+  <h3>A Scalable Inference Engine for Diffusion Transformers (DiTs) on multi-GPU Clusters</h3>
+  <strong><a href="https://arxiv.org/abs/2405.14430">📃 Paper</a> | <a href="#QuickStart">🚀 Quick Start</a> | <a href="#support-dits">🎯 Supported DiTs</a> | <a href="#dev-guide">📚 Dev Guide </a> | <a href="https://github.com/xdit-project/xDiT/discussions">📈  Discussion </a> </strong>
+</div>
+
+<h2 id="agenda">Table of Contents</h2>
+
+- [🔥 Meet xDiT](#meet-xdit)
+- [📢 Updates](#updates)
+- [🎯 Supported DiTs](#support-dits)
+- [📈 Performance](#perf)
+- [🚀 QuickStart](#QuickStart)
+- [✨ the xDiT's secret weapons](#secrets)
+  - [1. PipeFusion](#PipeFusion)
+  - [2. USP](#USP)
+  - [3. Hybrid Parallel](#hybrid_parallel)
+  - [4. CFG Parallel](#cfg_parallel)
+  - [5. Parallel VAE](#parallel_vae)
+- [📚  Develop Guide](#dev-guide)
+- [🚧  History and Looking for Contributions](#history)
+- [📝 Cite Us](#cite-us)
+
+
+<h2 id="meet-xdit">🔥 Meet xDiT</h2>
+
+Diffusion Transformers (DiTs), pivotal in text-to-image and text-to-video models, are driving advancements in high-quality image and video generation. 
+With the escalating input sequence length in DiTs, the computational demand of the Attention mechanism grows **quadratically**! 
+Consequently, multi-GPU and multi-machine deployments are essential to maintain real-time performance in online services.
+
+To meet real-time demand for DiTs applications, parallel inference is a must.
+xDiT is an inference engine designed for the parallel deployment of DiTs on large scale. 
+xDiT provides a suite of efficient parallel inference approaches for Diffusion Models.
+
+1. Sequence Parallelism, [USP](https://arxiv.org/abs/2405.07719) is a unified sequence parallel approach combining DeepSpeed-Ulysses, Ring-Attention.
+
+2. [PipeFusion](https://arxiv.org/abs/2405.14430), a patch level pipeline parallelism using displaced patch by taking advantage of the diffusion model characteristics.
+
+3. Data Parallel: Processes multiple prompts or generates multiple images from a single prompt in parallel across images.
+
+4. CFG Parallel, also known as Split Batch: Activates when using classifier-free guidance (CFG) with a constant parallelism of 2.
+
+The four parallel methods in xDiT can be configured in a hybrid manner, optimizing communication patterns to best suit the underlying network hardware.
+
+xDiT offers a set of APIs to adapt DiT models in [huggingface/diffusers](https://github.com/huggingface/diffusers) to hybrid parallel implementation through simple wrappers. 
+If the model you require is not available in the model zoo, developing it yourself is straightforward; please refer to our [Dev Guide](#dev-guide).
+
+
+We also have implemented the following parallel stategies for reference:
+
+1. Tensor Parallelism
+2. [DistriFusion](https://arxiv.org/abs/2402.19481)
+
+The communication and memory costs associated with the aforementioned parallelism, except for the CFG and DP, in DiTs are detailed in the table below. (* denotes that communication can be overlapped with computation.)
+
+
+As we can see, PipeFusion and Sequence Parallel achieve lowest communication cost on different scales and hardware configurations, making them suitable foundational components for a hybrid approach.
+
+𝒑: Number of pixels;
+𝒉𝒔: Model hidden size;
+𝑳: Number of model layers;
+𝑷: Total model parameters;
+𝑵: Number of parallel devices;
+𝑴: Number of patch splits;
+𝑸𝑶: Query and Output parameter count;
+𝑲𝑽: KV Activation parameter count;
+𝑨 = 𝑸 = 𝑶 = 𝑲 = 𝑽: Equal parameters for Attention, Query, Output, Key, and Value;
+
+<div align="center">
+
+|          | attn-KV | communication cost | param memory | activations memory | extra buff memory |
+|:--------:|:-------:|:-----------------:|:-----:|:-----------:|:----------:|
+| Tensor Parallel | fresh | $4O(p \times hs)L$ | $\frac{1}{N}P$ | $\frac{2}{N}A = \frac{1}{N}QO$ | $\frac{2}{N}A = \frac{1}{N}KV$ |
+| DistriFusion* | stale | $2O(p \times hs)L$ | $P$ | $\frac{2}{N}A = \frac{1}{N}QO$ | $2AL = (KV)L$ |
+| Ring Sequence Parallel* | fresh | $2O(p \times hs)L$ | $P$ | $\frac{2}{N}A = \frac{1}{N}QO$ | $\frac{2}{N}A = \frac{1}{N}KV$ |
+| Ulysses Sequence Parallel | fresh | $\frac{4}{N}O(p \times hs)L$ | $P$ | $\frac{2}{N}A = \frac{1}{N}QO$ | $\frac{2}{N}A = \frac{1}{N}KV$ |
+| PipeFusion* | stale- | $2O(p \times hs)$ | $\frac{1}{N}P$ | $\frac{2}{M}A = \frac{1}{M}QO$ | $\frac{2L}{N}A = \frac{1}{N}(KV)L$ |
+
+</div>
+
+<h2 id="updates">📢 Updates</h2>
+
+* 🎉**August 8, 2024**: Support Flux sequence parallel version. The inference scripts are [examples/flux_example](examples/flux_example.py).
+* 🎉**August 2, 2024**: Support Stable Diffusion 3 hybrid parallel version. The inference scripts are [examples/sd3_example](examples/sd3_example.py).
+* 🎉**July 18, 2024**: Support PixArt-Sigma and PixArt-Alpha. The inference scripts are [examples/pixartsigma_example.py](examples/pixartsigma_example.py), [examples/pixartalpha_example.py](examples/pixartalpha_example.py).
+* 🎉**July 17, 2024**: Rename the project to xDiT. The project has evolved from a collection of parallel methods into a unified inference framework and supported the hybrid parallel for DiTs.
+* 🎉**July 10, 2024**: Support HunyuanDiT. The inference script is [legacy/scripts/hunyuandit_example.py](./legacy/scripts/hunyuandit_example.py).
+* 🎉**June 26, 2024**: Support Stable Diffusion 3. The inference script is [legacy/scripts/sd3_example.py](./legacy/scripts/sd3_example.py).
+* 🎉**May 24, 2024**: PipeFusion is public released. It supports PixArt-alpha [legacy/scripts/pixart_example.py](./legacy/scripts/pixart_example.py), DiT [legacy/scripts/ditxl_example.py](./legacy/scripts/ditxl_example.py) and SDXL [legacy/scripts/sdxl_example.py](./legacy/scripts/sdxl_example.py).
+
+
+<h2 id="support-dits">🎯 Supported DiTs</h2>
+
+-  [🟠 Flux](https://huggingface.co/black-forest-labs/FLUX.1-schnell)
+-  [🔴 PixArt-Sigma](https://huggingface.co/PixArt-alpha/PixArt-Sigma-XL-2-1024-MS)
+-  [🔵 HunyuanDiT-v1.2-Diffusers](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2-Diffusers)
+-  [🟢 PixArt-alpha](https://huggingface.co/PixArt-alpha/PixArt-alpha)
+-  [🟠 Stable Diffusion 3](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers)
+-  [🔴 DiT-XL](https://huggingface.co/facebook/DiT-XL-2-256)
+
+
+<h2 id="perf">📈 Performance</h2>
+
+Here are the benchmark results for Pixart-Alpha using the 20-step DPM solver as the scheduler across various image resolutions. To replicate these findings, please refer to the script at [./legacy/scripts/benchmark.sh](./legacy/scripts/benchmark.sh).
+
+**TBD**: Updates results on hybrid parallelism.
+
+1. The Latency on 4xA100-80GB (PCIe)
+
+<div align="center">
+    <img src="./assets/latency-A100-PCIe.png" alt="A100 PCIe latency">
+</div>
+
+2. The Latency on 8xL20-48GB (PCIe)
+
+<div align="center">
+    <img src="./assets/latency-L20.png" alt="L20 latency">
+</div>
+
+3. The Latency on 8xA100-80GB (NVLink)
+
+<div align="center">
+    <img src="./assets/latency-A100-NVLink.png" alt="latency-A100-NVLink">
+</div>
+
+4. The Latency on 4xT4-16GB (PCIe)
+
+<div align="center">
+    <img src="./assets/latency-T4.png" 
+    alt="latency-T4">
+</div>
+
+
+<h2 id="QuickStart">🚀 QuickStart</h2>
+
+1. Install yunchang for sequence parallel.
+
+Install yunchang from [feifeibear/long-context-attention](https://github.com/feifeibear/long-context-attention).
+Please note that it has a dependency on flash attention and specific GPU model requirements. We recommend installing yunchang from the source code rather than using `pip install yunchang==0.2.0`.
+
+2. Install xDiT
+
+```
+python setup.py install
+```
+
+3. Usage
+
+We provide examples demonstrating how to run models with xDiT in the [./examples/](./examples/) directory. 
+You can easily modify the model type, model directory, and parallel options in the [examples/run.sh](examples/run.sh) within the script to run some already supported DiT models.
+
+```bash
+bash examples/run.sh
+```
+
+To inspect the available options for the PixArt-alpha example, use the following command:
+
+```bash
+python ./examples/pixartalpha_example.py -h
+
+...
+
+xFuser Arguments
+
+options:
+  -h, --help            show this help message and exit
+
+Model Options:
+  --model MODEL         Name or path of the huggingface model to use.
+  --download-dir DOWNLOAD_DIR
+                        Directory to download and load the weights, default to the default cache dir of huggingface.
+  --trust-remote-code   Trust remote code from huggingface.
+
+Runtime Options:
+  --warmup_steps WARMUP_STEPS
+                        Warmup steps in generation.
+  --use_parallel_vae
+  --seed SEED           Random seed for operations.
+  --output_type OUTPUT_TYPE
+                        Output type of the pipeline.
+
+Parallel Processing Options:
+  --do_classifier_free_guidance
+  --use_split_batch     Use split batch in classifier_free_guidance. cfg_degree will be 2 if set
+  --data_parallel_degree DATA_PARALLEL_DEGREE
+                        Data parallel degree.
+  --ulysses_degree ULYSSES_DEGREE
+                        Ulysses sequence parallel degree. Used in attention layer.
+  --ring_degree RING_DEGREE
+                        Ring sequence parallel degree. Used in attention layer.
+  --pipefusion_parallel_degree PIPEFUSION_PARALLEL_DEGREE
+                        Pipefusion parallel degree. Indicates the number of pipeline stages.
+  --num_pipeline_patch NUM_PIPELINE_PATCH
+                        Number of patches the feature map should be segmented in pipefusion parallel.
+  --attn_layer_num_for_pp [ATTN_LAYER_NUM_FOR_PP ...]
+                        List representing the number of layers per stage of the pipeline in pipefusion parallel
+  --tensor_parallel_degree TENSOR_PARALLEL_DEGREE
+                        Tensor parallel degree.
+  --split_scheme SPLIT_SCHEME
+                        Split scheme for tensor parallel.
+
+Input Options:
+  --height HEIGHT       The height of image
+  --width WIDTH         The width of image
+  --prompt [PROMPT ...]
+                        Prompt for the model.
+  --no_use_resolution_binning
+  --negative_prompt [NEGATIVE_PROMPT ...]
+                        Negative prompt for the model.
+  --num_inference_steps NUM_INFERENCE_STEPS
+                        Number of inference steps.
+```
+
+Hybriding multiple parallelism techniques togather is essential for efficiently scaling. 
+It's important that the product of all parallel degrees matches the number of devices. 
+For instance, you can combine CFG, PipeFusion, and sequence parallelism with the command below to generate an image of a cute dog through hybrid parallelism. 
+Here ulysses_degree * pipefusion_parallel_degree * cfg_degree(use_split_batch) == number of devices == 8.
+
+
+```bash
+torchrun --nproc_per_node=8 \
+examples/pixartalpha_example.py \
+--model models/PixArt-XL-2-1024-MS \
+--pipefusion_parallel_degree 2 \
+--ulysses_degree 2 \
+--num_inference_steps 20 \
+--warmup_steps 0 \
+--prompt "A small dog" \
+--use_split_batch
+```
+
+
+⚠️ Applying PipeFusion requires setting `warmup_steps`, also required in DistriFusion, typically set to a small number compared with `num_inference_steps`.
+The warmup step impacts the efficiency of PipeFusion as it cannot be executed in parallel, thus degrading to a serial execution. 
+We observed that a warmup of 0 had no effect on the PixArt model.
+Users can tune this value according to their specific tasks.
+
+
+<h2 id="secrets">✨ The xDiT's Secret Weapons</h2>
+
+The exceptional capabilities of xDiT stem from our innovative technologies.
+
+<h3 id="PipeFusion">1. PipeFusion</h3>
+
+[PipeFusion: Displaced Patch Pipeline Parallelism for Diffusion Models](./docs/methods/pipefusion.md)
+
+<h3 id="USP">2. USP: Unified Sequence Parallelism</h3>
+
+[USP: A Unified Sequence Parallelism Approach for Long Context Generative AI](./docs/methods/usp.md)
+
+<h3 id="hybrid_parallel">3. Hybrid Parallel</h3>
+
+[Hybrid Parallelism](./docs/methods/hybrid.md)
+
+<h3 id="cfg_parallel">4. CFG Parallel</h3>
+
+[CFG Parallel](./docs/methods/cfg_parallel.md)
+
+<h3 id="parallel_vae">5. Parallel VAE</h3>
+
+[Patch Parallel VAE](./docs/methods/parallel_vae.md)
+
+<h2 id="dev-guide">📚  Develop Guide</h2>
+
+[The implement and design of xdit framework](./docs/developer/The_implement_design_of_xdit_framework.md)
+
+[Manual for adding new models](./docs/developer/Manual_for_Adding_New_Models.md)
+
+<h2 id="history">🚧  History and Looking for Contributions</h2>
+
+We conducted a major upgrade of this project in August 2024.
+
+The latest APIs is located in the [xfuser/](./xfuser/) directory, supports hybrid parallelism. It offers clearer and more structured code but currently supports fewer models.
+
+The legacy APIs is in the [legacy/](./legacy/) directory, limited to single parallelism. It supports a richer of parallel methods, including PipeFusion, Sequence Parallel, DistriFusion, and Tensor Parallel. CFG Parallel can be hybrid with PipeFusion but not with other parallel methods.
+
+For models not yet supported by the latest APIs, you can run the examples in the [legacy/scripts/](./legacy/scripts/) directory. If you wish to develop new features on a model or require hybrid parallelism, stay tuned for further project updates. 
+
+We also welcome developers to join and contribute more features and models to the project. Tell us which model you need in xDiT in [discussions](https://github.com/xdit-project/xDiT/discussions).
+
+<h2 id="cite-us">📝 Cite Us</h2>
+
+```
+@article{wang2024pipefusion,
+      title={PipeFusion: Displaced Patch Pipeline Parallelism for Inference of Diffusion Transformer Models}, 
+      author={Jiannan Wang and Jiarui Fang and Jinzhe Pan and Aoyu Li and PengCheng Yang},
+      year={2024},
+      eprint={2405.07719},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+