lycoris-lora 2.3.0.dev11__tar.gz → 3.0.0__tar.gz

lycoris_lora-3.0.0/PKG-INFO

@@ -0,0 +1,25 @@
+Metadata-Version: 2.1
+Name: lycoris_lora
+Version: 3.0.0
+Summary: Lora beYond Conventional methods, Other Rank adaptation Implementations for Stable diffusion
+Home-page: https://github.com/KohakuBlueleaf/LyCORIS
+Author: Shih-Ying Yeh(KohakuBlueLeaf), Yu-Guan Hsieh, Zhidong Gao
+Author-email: apolloyeh0123@gmail.com
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: GPU :: NVIDIA CUDA :: 11.8
+Classifier: Environment :: GPU :: NVIDIA CUDA :: 12
+Classifier: Environment :: GPU :: NVIDIA CUDA :: 12 :: 12.1
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+License-File: LICENSE.md
+Requires-Dist: torch
+Requires-Dist: einops
+Requires-Dist: toml
+Requires-Dist: tqdm

{lycoris_lora-2.3.0.dev11 → lycoris_lora-3.0.0}/README.md

@@ -1,6 +1,15 @@
+
+![pypi](https://img.shields.io/pypi/v/lycoris-lora.svg)
+![versions](https://img.shields.io/pypi/pyversions/lycoris-lora.svg)
+![PyPI - License](https://img.shields.io/pypi/l/lycoris-lora)
+![downloads](https://img.shields.io/pypi/dm/lycoris-lora)
+![commits](https://img.shields.io/github/commit-activity/m/KohakuBlueleaf/LyCORIS/dev)
+![Discord](https://img.shields.io/discord/1082218577395986452)
+
+
 # LyCORIS - Lora beYond Conventional methods, Other Rank adaptation Implementations for Stable diffusion. (ICLR'24)
 
-![banner image](docs/images/banner.png)
+![banner image](docs/images/banner2.png)
 
 A project that implements different parameter-efficient fine-tuning algorithms for Stable Diffusion.
 
@@ -20,21 +29,21 @@ Please check [List of Implemented Algorithms](docs/Algo-List.md) and [Guidelines
 
 A simple comparison of some of these methods are provided below (to be taken with a grain of salt)
 
-|                       | Full | LoRA | LoHa | LoKr low factor | LoKr high factor $^+$ |
-| --------------------- | ---- | ---- | ---- | --------------- | ---------------- |
-| Fidelity              | ★   | ●   | ▲   | ◉              | ▲               |
-| Flexibility $^*$     | ★   | ●   | ◉   | ▲              | ● $^†$        |
-| Diversity             | ▲   | ◉   | ★   | ●              | ★               |
-| Size                  | ▲   | ●   | ●   | ●              | ★               |
-| Training Speed Linear | ★   | ●   | ●   | ★              | ★               |
-| Training Speed Conv   | ●   | ★   | ▲   | ●              | ●               |
+|                       | Full | LoRA | LoHa | LoKr low factor | LoKr high factor$^+$ |
+| --------------------- | ---- | ---- | ---- | --------------- | ---------------------- |
+| Fidelity              | ★   | ●   | ▲   | ◉              | ▲                     |
+| Flexibility$^*$     | ★   | ●   | ◉   | ▲              | ●$^†$              |
+| Diversity             | ▲   | ◉   | ★   | ●              | ★                     |
+| Size                  | ▲   | ●   | ●   | ●              | ★                     |
+| Training Speed Linear | ★   | ●   | ●   | ★              | ★                     |
+| Training Speed Conv   | ●   | ★   | ▲   | ●              | ●                     |
 
 ★ > ◉ > ● > ▲
 [> means better and smaller size is better]
 
-$^+$ Usually we take `factor <= 0.5 * sqrt(dim)` as low factor and `factor >= sqrt(dim` as high factor. For example, factor<=8 for SD1.x/SD2.x/SDXL can be seen as low factor, and, factor>=16 can be seen as high factor. <br>
-$^*$ Flexibility means anything related to generating images not similar to those in the training set, and combination of multiple concepts, whether they are trained together or not <br>
-$^†$ It may become more difficult to switch base model or combine multiple concepts in this situation <br>
+$^+$ Usually we take `factor <= 0.5 * sqrt(dim)` as low factor and `factor >= sqrt(dim` as high factor. For example, factor<=8 for SD1.x/SD2.x/SDXL can be seen as low factor, and, factor>=16 can be seen as high factor. `<br>`
+$^*$ Flexibility means anything related to generating images not similar to those in the training set, and combination of multiple concepts, whether they are trained together or not `<br>`
+$^†$ It may become more difficult to switch base model or combine multiple concepts in this situation `<br>`
 
 **The actual performance may vary depending on the datasets, tasks, and hyperparameters used. It is recommended to experiment with different settings to achieve optimal results.**
 
@@ -66,7 +75,7 @@ However, newer model types may not always be supported. If you encounter this is
 There are three different ways to train LyCORIS models.
 
 - With [kohya-ss/sd-scripts](https://github.com/kohya-ss/sd-scripts) (see a list of compatible graphical interfaces and colabs at the end of the section)
-- With [HCP-Diffusion](https://github.com/IrisRainbowNeko/HCP-Diffusion)
+- With [Naifu-Diffusion](https://github.com/Mikubill/naifu)
 - With your own script by using LyCORIS as standalone wrappers for **ANY** pytorch modules.
 
 In any case, please install this package in the corresponding virtual environment. You can either install it
@@ -74,7 +83,7 @@ In any case, please install this package in the corresponding virtual environmen
 - through pip
 
   ```bash
-  pip install lycoris_lora
+  pip install lycoris-lora
   ```
 - or from source
 
@@ -111,6 +120,8 @@ You can use this package's kohya module to run kohya's training script to train
 
 #### HCP-Diffusion
 
+**The support for HCP-Diffusion has been dropped on LyCORIS3.0.0, we will wait until HCP side finish the implementation of new wrapper**
+
 You can use this package's hcp module to run HCP-Diffusion's training script to train lycoris module for SD models
 
 ```bash
@@ -156,6 +167,13 @@ forward_with_lyco = your_model(x)
 
 You can check my [HakuPhi](https://github.com/KohakuBlueleaf/HakuPhi) project to see how I utilize LyCORIS to finetune the Phi-1.5 models.
 
+#### Other method
+
+After LyCORIS3.0.0, Parametrize API and Functional API have been added, which provide more different ways on utilizing LyCORIS library.
+
+Check API reference for more informations.
+You can also take the [test suites](test/) as a kind of examples.
+
 #### Bitsandbytes support
 
 See [bnb_example.py](bnb_example.py) for example. Basically as same as standalone wrapper.
@@ -243,44 +261,61 @@ See [docs/Conversion-scripts.md](docs/Conversion-scripts.md) for more informatio
 
 For full log, please see [Change.md](Change.md)
 
-## 2024/03/15 update to 2.2.0 - QLyCORIS and DoRA
+### 2024/06/xx update to 3.0.0 - Brand New Functional API, Parametrize API and Module API
 
-#### New Algo
+#### The reasons of 3.0.0
 
-* DoRA
-  * Ref: [DoRA: Weight-Decomposed Low-Rank Adaptation](https://github.com/KohakuBlueleaf/LyCORIS)
-* Weight decompose for LoHa and LoKr. (A.K.A DoHa/DoKr)
-  * DoRA/DoHa/DoKr will require smaller Learning rate!
+We reconstruct the whole library with new Class definition and brand new Functional API system.
 
-#### New Features
+We also removed lot of redundant/unused modules.
 
-* Support "bypass" (a.k.a. adapter) mode for LoHa/LoKr/OFT/BOFT
-  * LoHa will require 2xFLOPs since we rebuild full diff weight and then do one more forward.
-  * LoKr, OFT, BOFT should be more efficient than LoHa in bypass mode.
-* Support [bnb 8bit/4bit Linear layer](https://github.com/TimDettmers/bitsandbytes) (a.k.a. QLyCORIS) with LoHa/LoKr/OFT/BOFT.
-  * This will force module to enable bypass mode.
+Since the whole library are changed significantly. We decide to call it 3.0.0 as a new major version.
 
-#### Fixes, slight changes
+#### Major Changes
 
-* Refine some details about code quality. Based on the report from GitRoll. (Thx you gitroll!)
-* Remove redundant calculation in BOFT
-* rank_dropout has been removed from OFT/BOFT temporarily untill we ensure how to apply it.
-* Fix bugs in lokr when `lokr_w1_a` not exist.
-* Fix bugs in conversion scritps.
+* New Module API
+* Add Parametrize API
+* Add Functional API
+  * LoCon/LoHa/LoKr/Diag-OFT/BOFT only.
+* Remove optional deps from install_requires
+* Remove lot of redundant/deprecated modules
+* Better testing
+* HunYuan DiT Support ([PR](https://github.com/kohya-ss/sd-scripts/pull/1378) in kohya-ss/sd-scripts)
+
+#### Full change log
+
+##### New Features
+
+* LyCORIS now have consistent API for different algorithm like `bypass_forward_diff` or `get_diff_weight` method. Developers of other project can utilize these API to do more tricks or integrate LyCORIS into their framework more easily.
+* LyCORIS now have parametrize API which utilize `torch.nn.utils.parametrize.register_parametrization` to directly patch individual parameters. Which can be useful for MHA layer or other tricky modules.
+  * Currently only support 2~5D tensors. And LyCORIS will pretend these weights are weight of Linear/Conv1,2,3D then send it into LyCORIS modules
+  * More native implementation or more detailed control will be added in the future.
+* LyCORIS now have functional API. Developers who prefer functional more than Module things can utilize this feature.
+  * Functional API also allow developers who don't want to introduce new dependencies. Just copy-paste the source code and utilizing it. (with Apache-2 License, directly copy-paste is totally allowed)
+* Add support for Conv1d and Conv3d module on LoCon/LoHa/LoKr/Full/OFT/BOFT/GLoRA (not All algo in LyCORIS support them, you may receive error when apply unsopported algo), support inherited module (for example: `LoRACompatibleConv` or `LoRACompatibleLinear` from [`huggingface/diffusers`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/lora.py))
+* HunYuan DiT support.
+
+##### Improvements, Fixes, Slight Changes
+
+* Drop dependencies related to kohya-ss/sd-scripts:
+  * We now take kohya-ss/sd-scripts as optional dependency
+  * Which means `transformers`, `diffusers` and anything related to kohya are all optional deps now.
+* The definition of dropout and rank_dropout in each algorithm are changed. Since some concept of original rank_dropout in the lora of kohya-ss/sd-script is hard to applied to other algorithm. We can only design the dropout for each module seperatedly.
+* `apply_max_norm` issue are all fixed.
+* DyLoRA, (IA)^3, GLoRA are all rewritten and support Linear/Conv1,2,3d.
+* (IA)^3, GLoRA, Diag-OFT, BOFT are supported in `create_lycoris_from_weights`
+  * `lycoris.kohya.create_network_from_weights` also support them as well.
+* Fix wrong implementation of BOFT.
+* `create_lycoris_from_weights` and `create_network_from_weights` now have correct logging infos.
+* `get_module` and `make_module` are moved into modules' API.
 
 ## Todo list
 
-- [X] Module and Document for using LyCORIS in any other model, Not only SD.
-- [X] Proposition3 in [FedPara](https://arxiv.org/abs/2108.06098)
-  * also need custom backward to save the vram
-- [ ] Low rank + sparse representation
-  - [X] For extraction
-  - [ ] For training
-- [ ] Support more operation, not only linear and conv2d.
-- [X] Configure varying ranks or dimensions for specific modules as needed.
 - [ ] Automatically selecting an algorithm based on the specific rank requirement.
-- [ ] Explore other low-rank representations or parameter-efficient methods to fine-tune either the entire model or specific parts of it.
 - [ ] More experiments for different task, not only diffusion models.
+  - [X] LoKr and LoHa have been proven to be useful for Large Language Model.
+- [ ] Explore other low-rank representations or parameter-efficient methods to fine-tune either the entire model or specific parts of it.
+- [ ] Documentation for whole library.
 
 ## Citation
 

{lycoris_lora-2.3.0.dev11 → lycoris_lora-3.0.0}/lycoris/__init__.py

@@ -1,5 +1,8 @@
+try:
+    from . import kohya
+except:
+    pass
 from . import (
-    kohya,
     modules,
     utils,
 )

{lycoris_lora-2.3.0.dev11 → lycoris_lora-3.0.0}/lycoris/config.py

@@ -6,6 +6,7 @@ PRESET = {
             "ResnetBlock2D",
             "Downsample2D",
             "Upsample2D",
+            "HunYuanDiTBlock",
         ],
         "unet_target_name": [
             "conv_in",
@@ -13,7 +14,12 @@ PRESET = {
             "time_embedding.linear_1",
             "time_embedding.linear_2",
         ],
-        "text_encoder_target_module": ["CLIPAttention", "CLIPMLP"],
+        "text_encoder_target_module": [
+            "CLIPAttention",
+            "CLIPMLP",
+            "MT5Block",
+            "BertLayer",
+        ],
         "text_encoder_target_name": [],
     },
     "full-lin": {
@@ -21,30 +27,47 @@ PRESET = {
         "unet_target_module": [
             "Transformer2DModel",
             "ResnetBlock2D",
+            "HunYuanDiTBlock",
         ],
         "unet_target_name": [
             "time_embedding.linear_1",
             "time_embedding.linear_2",
         ],
-        "text_encoder_target_module": ["CLIPAttention", "CLIPMLP"],
+        "text_encoder_target_module": [
+            "CLIPAttention",
+            "CLIPMLP",
+            "MT5Block",
+            "BertLayer",
+        ],
         "text_encoder_target_name": [],
     },
     "attn-mlp": {
         "enable_conv": False,
         "unet_target_module": [
             "Transformer2DModel",
+            "HunYuanDiTBlock",
         ],
         "unet_target_name": [],
-        "text_encoder_target_module": ["CLIPAttention", "CLIPMLP"],
+        "text_encoder_target_module": [
+            "CLIPAttention",
+            "CLIPMLP",
+            "MT5Block",
+            "BertLayer",
+        ],
         "text_encoder_target_name": [],
     },
     "attn-only": {
         "enable_conv": False,
         "unet_target_module": [
             "CrossAttention",
+            "SelfAttention",
         ],
         "unet_target_name": [],
-        "text_encoder_target_module": ["CLIPAttention"],
+        "text_encoder_target_module": [
+            "CLIPAttention",
+            "BertAttention",
+            "MT5LayerSelfAttention",
+        ],
         "text_encoder_target_name": [],
     },
     "unet-only": {
@@ -54,6 +77,7 @@ PRESET = {
             "ResnetBlock2D",
             "Downsample2D",
             "Upsample2D",
+            "HunYuanDiTBlock",
         ],
         "unet_target_name": [
             "conv_in",
@@ -68,6 +92,7 @@ PRESET = {
         "enable_conv": False,
         "unet_target_module": [
             "Transformer2DModel",
+            "HunYuanDiTBlock",
         ],
         "unet_target_name": [],
         "text_encoder_target_module": [],
@@ -83,4 +108,15 @@ PRESET = {
         "text_encoder_target_module": [],
         "text_encoder_target_name": [],
     },
+    "ia3": {
+        "enable_conv": False,
+        "unet_target_module": [],
+        "unet_target_name": ["to_k", "to_v", "ff.net.2"],
+        "text_encoder_target_module": [],
+        "text_encoder_target_name": ["k_proj", "v_proj", "mlp.fc2"],
+        "name_algo_map": {
+            "mlp.fc2": {"train_on_input": True},
+            "ff.net.2": {"train_on_input": True},
+        },
+    },
 }

lycoris_lora-3.0.0/lycoris/functional/__init__.py

@@ -0,0 +1,9 @@
+from .general import (
+    rebuild_tucker,
+    factorization,
+    power2factorization,
+    FUNC_LIST,
+    tucker_weight,
+    tucker_weight_from_conv,
+    apply_dora_scale,
+)

lycoris_lora-3.0.0/lycoris/functional/boft.py

@@ -0,0 +1,99 @@
+import math
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from einops import rearrange
+
+from .general import power2factorization, FUNC_LIST
+from .diag_oft import get_r
+
+
+def weight_gen(org_weight, max_block_size, boft_m=-1, rescale=False):
+    """### boft_weight_gen
+
+    Args:
+        org_weight (torch.Tensor): the weight tensor
+        max_block_size (int): max block size
+        rescale (bool, optional): whether to rescale the weight. Defaults to False.
+
+    Returns:
+        torch.Tensor: oft_blocks[, rescale_weight]
+    """
+    out_dim, *rest = org_weight.shape
+    block_size, block_num = power2factorization(out_dim, max_block_size)
+    max_boft_m = sum(int(i) for i in f"{block_num-1:b}") + 1
+    if boft_m == -1:
+        boft_m = max_boft_m
+    boft_m = min(boft_m, max_boft_m)
+    oft_blocks = torch.zeros(boft_m, block_num, block_size, block_size)
+    if rescale is not None:
+        return oft_blocks, torch.ones(out_dim, *[1] * len(rest))
+    else:
+        return oft_blocks, None
+
+
+def diff_weight(org_weight, oft_blocks, rescale=None, constraint=None):
+    """### boft_diff_weight
+
+    Args:
+        TODO
+
+    Returns:
+        torch.Tensor: ΔW
+    """
+    m, num, b, _ = oft_blocks.shape
+    r_b = b // 2
+    I = torch.eye(b, device=oft_blocks.device)
+    r = get_r(oft_blocks, I, constraint)
+    inp = org = org_weight.to(dtype=r.dtype)
+
+    for i in range(m):
+        bi = r[i]  # b_num, b_size, b_size
+        inp = rearrange(inp, "(c g k) ... -> (c k g) ...", g=2, k=2**i * r_b)
+        inp = rearrange(inp, "(d b) ... -> d b ...", b=b)
+        inp = torch.einsum("b i j, b j ... -> b i ...", bi, inp)
+        inp = rearrange(inp, "d b ... -> (d b) ...")
+        inp = rearrange(inp, "(c k g) ... -> (c g k) ...", g=2, k=2**i * r_b)
+
+    if rescale is not None:
+        inp = inp * rescale
+
+    return inp - org
+
+
+def bypass_forward_diff(
+    org_out, oft_blocks, rescale=None, constraint=None, need_transpose=False
+):
+    """### boft_bypass_forward_diff
+
+    Args:
+        TODO
+
+    Returns:
+        torch.Tensor: output tensor
+    """
+    m, num, b, _ = oft_blocks.shape
+    r_b = b // 2
+    I = torch.eye(b, device=oft_blocks.device)
+    r = get_r(oft_blocks, I, constraint)
+    inp = org = org_out.to(dtype=r.dtype)
+    if need_transpose:
+        inp = org = inp.transpose(1, -1)
+
+    for i in range(m):
+        bi = r[i]  # b_num, b_size, b_size
+        inp = rearrange(inp, "... (c g k) ->... (c k g)", g=2, k=2**i * r_b)
+        inp = rearrange(inp, "... (d b) -> ... d b", b=b)
+        inp = torch.einsum("b i j, ... b j -> ... b i", bi, inp)
+        inp = rearrange(inp, "... d b -> ... (d b)")
+        inp = rearrange(inp, "... (c k g) -> ... (c g k)", g=2, k=2**i * r_b)
+
+    if rescale is not None:
+        inp = inp * rescale.transpose(0, -1)
+
+    inp = inp - org
+    if need_transpose:
+        inp = inp.transpose(1, -1)
+    return inp

lycoris_lora-3.0.0/lycoris/functional/diag_oft.py

@@ -0,0 +1,103 @@
+import math
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from .general import factorization, FUNC_LIST
+
+
+def get_r(oft_blocks, I=None, constraint=0):
+    if I is None:
+        I = torch.eye(oft_blocks.shape[-1], device=oft_blocks.device)
+    if I.ndim < oft_blocks.ndim:
+        for _ in range(oft_blocks.ndim - I.ndim):
+            I = I.unsqueeze(0)
+    # for Q = -Q^T
+    q = oft_blocks - oft_blocks.transpose(-1, -2)
+    normed_q = q
+    if constraint is not None and constraint > 0:
+        q_norm = torch.norm(q) + 1e-8
+        if q_norm > constraint:
+            normed_q = q * constraint / q_norm
+    # use float() to prevent unsupported type
+    r = (I + normed_q) @ (I - normed_q).float().inverse()
+    return r
+
+
+def weight_gen(org_weight, max_block_size, rescale=False):
+    """### weight_gen
+
+    Args:
+        org_weight (torch.Tensor): the weight tensor
+        max_block_size (int): max block size
+        rescale (bool, optional): whether to rescale the weight. Defaults to False.
+
+    Returns:
+        torch.Tensor: oft_blocks[, rescale_weight]
+    """
+    out_dim, *rest = org_weight.shape
+    block_size, block_num = factorization(out_dim, max_block_size)
+    oft_blocks = torch.zeros(block_num, block_size, block_size)
+    if rescale:
+        return oft_blocks, torch.ones(out_dim, *[1] * len(rest))
+    else:
+        return oft_blocks, None
+
+
+def diff_weight(org_weight, oft_blocks, rescale=None, constraint=None):
+    """### diff_weight
+
+    Args:
+        TODO
+
+    Returns:
+        torch.Tensor: ΔW
+    """
+    I = torch.eye(oft_blocks.shape[1], device=oft_blocks.device)
+    r = get_r(oft_blocks, I, constraint)
+
+    block_num, block_size, _ = oft_blocks.shape
+    _, *shape = org_weight.shape
+    org_weight = org_weight.to(dtype=r.dtype)
+    org_weight = org_weight.view(block_num, block_size, *shape)
+    # Init R=0, so add I on it to ensure the output of step0 is original model output
+    weight = torch.einsum(
+        "k n m, k n ... -> k m ...",
+        r - I,
+        org_weight,
+    ).view(-1, *shape)
+    if rescale is not None:
+        weight = rescale * weight
+        weight = weight + (rescale - 1) * org_weight
+    return weight
+
+
+def bypass_forward_diff(
+    org_out, oft_blocks, rescale=None, constraint=None, need_transpose=False
+):
+    """### bypass_forward_diff
+
+    Args:
+        TODO
+
+    Returns:
+        torch.Tensor: output tensor
+    """
+    block_num, block_size, _ = oft_blocks.shape
+    I = torch.eye(block_size, device=oft_blocks.device)
+    r = get_r(oft_blocks, I, constraint)
+    if need_transpose:
+        org_out = org_out.transpose(1, -1)
+    org_out = org_out.to(dtype=r.dtype)
+    *shape, _ = org_out.shape
+    oft_out = torch.einsum(
+        "k n m, ... k n -> ... k m", r - I, org_out.view(*shape, block_num, block_size)
+    )
+    out = oft_out.view(*shape, -1)
+    if rescale is not None:
+        out = rescale.transpose(-1, 0) * out
+        out = out + (rescale - 1).transpose(-1, 0) * org_out
+    if need_transpose:
+        out = out.transpose(1, -1)
+    return out

lycoris_lora-3.0.0/lycoris/functional/general.py

@@ -0,0 +1,106 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+FUNC_LIST = [None, None, F.linear, F.conv1d, F.conv2d, F.conv3d]
+
+
+def rebuild_tucker(t, wa, wb):
+    rebuild2 = torch.einsum("i j ..., i p, j r -> p r ...", t, wa, wb)
+    return rebuild2
+
+
+def factorization(dimension: int, factor: int = -1) -> tuple[int, int]:
+    """
+    return a tuple of two value of input dimension decomposed by the number closest to factor
+    second value is higher or equal than first value.
+
+    In LoRA with Kroneckor Product, first value is a value for weight scale.
+    second value is a value for weight.
+
+    Because of non-commutative property, A⊗B ≠ B⊗A. Meaning of two matrices is slightly different.
+
+    examples)
+    factor
+        -1               2                4               8               16               ...
+    127 -> 1, 127   127 -> 1, 127    127 -> 1, 127   127 -> 1, 127   127 -> 1, 127
+    128 -> 8, 16    128 -> 2, 64     128 -> 4, 32    128 -> 8, 16    128 -> 8, 16
+    250 -> 10, 25   250 -> 2, 125    250 -> 2, 125   250 -> 5, 50    250 -> 10, 25
+    360 -> 8, 45    360 -> 2, 180    360 -> 4, 90    360 -> 8, 45    360 -> 12, 30
+    512 -> 16, 32   512 -> 2, 256    512 -> 4, 128   512 -> 8, 64    512 -> 16, 32
+    1024 -> 32, 32  1024 -> 2, 512   1024 -> 4, 256  1024 -> 8, 128  1024 -> 16, 64
+    """
+
+    if factor > 0 and (dimension % factor) == 0:
+        m = factor
+        n = dimension // factor
+        if m > n:
+            n, m = m, n
+        return m, n
+    if factor < 0:
+        factor = dimension
+    m, n = 1, dimension
+    length = m + n
+    while m < n:
+        new_m = m + 1
+        while dimension % new_m != 0:
+            new_m += 1
+        new_n = dimension // new_m
+        if new_m + new_n > length or new_m > factor:
+            break
+        else:
+            m, n = new_m, new_n
+    if m > n:
+        n, m = m, n
+    return m, n
+
+
+def power2factorization(dimension: int, factor: int = -1) -> tuple[int, int]:
+    """
+    m = 2k
+    n = 2**p
+    m*n = dim
+    """
+    if factor == -1:
+        factor = dimension
+
+    # Find the first solution and check if it is even doable
+    m = n = 0
+    while m <= factor:
+        m += 2
+        while dimension % m != 0 and m < dimension:
+            m += 2
+        if m > factor:
+            break
+        if sum(int(i) for i in f"{dimension//m:b}") == 1:
+            n = dimension // m
+
+    return dimension // n, n
+
+
+def tucker_weight_from_conv(up, down, mid):
+    up = up.reshape(up.size(0), up.size(1))
+    down = down.reshape(down.size(0), down.size(1))
+    return torch.einsum("m n ..., i m, n j -> i j ...", mid, up, down)
+
+
+def tucker_weight(wa, wb, t):
+    temp = torch.einsum("i j ..., j r -> i r ...", t, wb)
+    return torch.einsum("i j ..., i r -> r j ...", temp, wa)
+
+
+def apply_dora_scale(org_weight, rebuild, dora_scale, scale):
+    dora_norm_dims = org_weight.dim() - 1
+    weight = org_weight + rebuild
+    weight = weight.to(dora_scale.dtype)
+    weight_norm = (
+        weight.transpose(0, 1)
+        .reshape(weight.shape[1], -1)
+        .norm(dim=1, keepdim=True)
+        .reshape(weight.shape[1], *[1] * dora_norm_dims)
+        .transpose(0, 1)
+    )
+    merged_scale1 = weight / weight_norm * dora_scale
+    diff_weight = merged_scale1 - org_weight
+    return org_weight + diff_weight * scale