lycoris-lora 1.8.0.dev6__tar.gz → 1.8.0.dev8__tar.gz

lycoris_lora-1.8.0.dev8/.gitignore

@@ -0,0 +1,128 @@
+# test
+test_model/
+
+# Editors
+.vscode/
+.idea/
+
+# Vagrant
+.vagrant/
+
+# Mac/OSX
+.DS_Store
+
+# Windows
+Thumbs.db
+
+# Source for the following rules: https://raw.githubusercontent.com/github/gitignore/master/Python.gitignore
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json

lycoris_lora-1.8.0.dev8/Algo.md

@@ -0,0 +1,134 @@
+# Algorithm explanation
+
+
+## Basic Idea
+Linear Layer:
+
+$Y = W \cdot X$
+
+finetuning is to get the $W'$
+
+$Y = W \cdot X + W' \cdot X$
+
+and normally $shape(W') = shape(W)$
+
+---
+
+## Conventional Methods
+[Ref](https://arxiv.org/abs/2106.09685)
+
+Linear:
+
+$Y_{out \times batch} = W_{out \times in} \cdot X_{in \times batch}$
+
+$\xrightarrow{} Y_{out \times batch} = W_{out \times in} \cdot X_{in \times batch} + Wa_{out \times dim} \cdot Wb_{dim \times in} \cdot X_{in \times batch}$
+
+
+LoRA for Convolution:
+Consider im2col of matmul first:
+
+![image](https://user-images.githubusercontent.com/59680068/221547963-c821b9fa-2825-4b8d-8192-c3109268417f.png)
+![image](https://user-images.githubusercontent.com/59680068/221547996-4be14700-1392-4859-9e29-e3e669142a09.png)
+$X:[channel, width, height]$
+
+$\xrightarrow{reorder}[c \times kw \times kh, outw \times outh]$
+
+$Kernels: [out, c, kw, kh] \xrightarrow{reshape} [out, c \times kw \times kh]$
+
+$Conv(X, Kernels) = Kernels  \times  X \xrightarrow{reshape} [out, outw, outh]$
+
+and then write down this conventional LoRA for conv layer
+$Conv(in, out, ksize, padding, stride)$
+
+$\xrightarrow{}Conv(dim, out, 1)\circ Conv(in, dim, ksize, padding, stride)$
+
+
+In this method, we can get that
+$W' = Wa \cdot Wb$ with $rank(W') \le dim$
+
+---
+
+## Hadamard Product
+[Ref](https://arxiv.org/abs/2108.06098)
+![image](https://user-images.githubusercontent.com/59680068/223942143-05b5ebff-06c4-4d07-a0eb-037fd6f04e77.png)
+
+
+consider $W' = Wa \odot Wb$, we can get $rank(W') \le rank(Wa) \times rank(Wb)$.
+And then we use conventional method on $Wa$ and $Wb$. Which means it can use 2x dim to get square rank.
+
+**Rank != Information capacity, but they are relative**
+
+based on the experiment result from the paper, it seems like although the rank(Wa) * rank(Wb) is just upper bound, but almost everytime it will produce W' with rank = rank(Wa)*rank(Wb).
+
+### Why custom backward
+with $W' = (Wa_1 \cdot Wa_2) \odot (Wb_1 \cdot Wb_2)$, when you need to calc the backpropogation, you will need $\Delta{W'}$ and $Wa$ to calc $\Delta{Wb}$, also $Wb$ for $\Delta{Wa}$.
+
+With pytorch's autograd, this kind of operation will cache the $Wa$ and $Wb$ for calc the backward, which means it will cache 2x size of weight for backward.
+
+To avoid this terrible situation, I impl a custom backward which will reconstruct $Wa$ and $Wb$ when actually needed, this method saved tons of memory.
+
+### CP-Decomposition
+[Ref](https://arxiv.org/abs/1412.6553)
+
+As mentioned before, the weight shape for convolution layer is $[out, in, kw, kh]$. And we just unfold it to $[out, in \times kw \times kh]$ for decomposition.
+
+But actually there is a method to decomposition any shape ot tensor called cp decomposition.
+
+Using cp-decomposition in Covolution will be something like:
+
+$\tau: [dim, dim, kw, kh]$ <br>
+$x_1: [dim, out]$<br>
+$x_2: [dim, in]$<br>
+$W' = \tau \times_1 x_1 \times_2 x_2$<br>
+$W': [out, in, kw, kh]$
+
+Or write this thing as multiple conv layer:
+
+Conv(in, dim, (1, 1))<br>
+↓<br>
+Conv(dim, dim, (kw, kh), stride, padding)<br>
+↓<br>
+conv(dim, out, (1, 1))<br>
+
+For hadamard product implementation, just use 2 different $W'$ and multiply them together.
+
+---
+
+## Kronercker Product
+
+### Definition
+
+If $W_1$ is an a x b matrix and $W_2$ is a c x d matrix, then the Kronecker Product of two matrices is 
+
+$W' = W_1 \otimes W_2$ and an ac x bd matrix.
+
+In meaning of matrix, $W_2$ becomes weight and $W_1$ becomes weight scale of $W_2$
+
+### About rank
+
+And we can decompose $W_2$ using LoRA with rank, r.
+
+$W_2 = Wa_2 \cdot Wb_2$ then $W' = W_1 \otimes (Wa_2 \cdot Wb_2)$
+
+we can get $rank(W') \le rank(W_1) \times rank(Wa_2 \cdot Wb_2)$ and $rank(W_1) \le min(a, b), rank(Wa_2 \cdot Wb_2) \le r$ 
+
+=> $rank(W') \le min(a, b) \times r$
+
+Remember that $min(a, b) \times r$ is upper bound. $rank(W') = min(a, b) \times r$ does not guarantee. Experiment needs.
+
+### Number of parameters
+
+We decompose matrix, $W' = W_1 \otimes (Wa_2 \cdot Wb_2)$.
+
+(# of parameters) = $(a \times b) + (c \times r + r \times d) = a \times b + r \times (c + d)$, m = ac, n = bd.
+
+and suppose best case $a=c= \sqrt{m}, b=d= \sqrt{n}$
+
+then, (# of parameters) = $\sqrt{mn} + r \times (\sqrt{m} + \sqrt{n})$
+
+We can reduce the number of parameters to order of square root maximally.
+
+---
+
+## Sparse Bias
+Todo...

lycoris_lora-1.8.0.dev8/Change.md

@@ -0,0 +1,25 @@
+# Change Log
+
+## 2023/06/28 update to 1.7.1
+* **rearrange the version format, previous 0.1.7 should be 1.7.0**
+* fix the bug in scale weight norm
+
+## 2023/06/26 Update to 0.1.7
+* Add support for rank_dropout and module_dropout on LoCon/LoHa/LoKr
+* Add support for scale_weight_norms on LoCon/LoHa/LoKr
+* Will support SDXL on 0.1.8 (you can follow the dev branch)
+
+## 2023/06/04 update to 0.1.6
+* add dylora and IA^3 algorithm
+
+## 2023/03/29 Update to 0.1.4
+* cp decomposition is default to disable now
+* add 4 more layer to train (conv_in/out, time_embedding)
+
+## 2023/03/12 Update to 0.1.0
+* Add cp-decomposition implementation for convolution layer
+  * Both LoRA(LoCon) and LoHa can use this more parameter-efficient decomposition
+* Add sparse bias for extracted LoRA
+  * Will add to training in the future (Maybe)
+* Change weight initialization method in LoHa
+  * Use lower std to avoid loss to go high or NaN when using normal lr (like 0.5 in Dadap)

lycoris_lora-1.8.0.dev8/Demo.md

@@ -0,0 +1,53 @@
+# Demo, Example, Comparing
+
+
+## expand LoRA to Convolution Layer
+lora for transformer only
+vs
+lora for whole model
+
+Yog-Sothoth LoRA/LoCon:
+https://civitai.com/models/14878/loconlora-yog-sothoth-depersonalization
+
+LoRA rank=1:
+![00159](https://user-images.githubusercontent.com/59680068/222422792-b37648c3-af2e-4bee-9f82-b14a5d8e5f5d.png)
+LoCon rank=1
+![00164](https://user-images.githubusercontent.com/59680068/222422830-4ec9f550-cdff-4314-b694-1658bf9f1c83.png)
+
+xy grid:
+![image](https://user-images.githubusercontent.com/59680068/222424002-5ce2572c-9102-4e2d-83f2-100bc41ec272.png)
+
+
+With some experiments from community, finetuning whole model can learn "More". But in some case this means it will overfit heavily.
+
+---
+
+## Hadamard product vs Conventional
+We introduced LoRA finetuning with Hadamard Product representation from FedPara.
+And based on this [experiments](https://civitai.com/models/17336/roukin8-character-lohaloconfullckpt-8), LoHa with same size (and dim>2, or rank>4) can get better result in some situation.
+
+![Image](https://i.imgur.com/l3P0TgM.jpg)
+
+This is some comments from the experiments:
+```
+Why LoHa?
+
+The following comparison should be quite convincing.
+
+(Left 4 LoHa with different number of training steps; Right 4 same thing but for LoCon; same seed same training parameters for the two training)
+
+
+In addition to the five characters I also train a bunch of style into the model (what I have always been doing actually).
+
+However, LoRa and LoCon do not combine styles with characters that well (characters are only trained with anime images) and this capacity gets largely improved in LoHa.
+
+Note that the two files have almost the same size (around 30mb). For this I set (linear_dim, conv_dim) to (16,8) for LoCon and (8,4) for LoHa. However with Hadamard product the resulting matrix could now be of rank 8x8=64 for linear layers and 4x4=16 for convolutional layers.
+```
+
+And there is more example about LoHA vs LoCon in same file size. (diff < 200KB)
+![xyz_grid-0330-2023-03-08_d4d1ef62c3_KBlueLeaf_KBlueLeaf-v1 1_4114224331_c2af89d9-2160x2789](https://user-images.githubusercontent.com/59680068/223930255-a1e4b91e-25da-41ce-9dd2-12bd0e976afc.jpg)
+![xyz_grid-0324-2023-03-08_d4d1ef62c3_KBlueLeaf_KBlueLeaf-v1 1_2496183095_a91c0526-2160x2789](https://user-images.githubusercontent.com/59680068/223930265-f1f1d658-3722-46ce-93e6-fdec18a58c19.jpg)
+
+![xyz_grid-0300-2023-03-08_b38775f1cf_download_谈秋-v2_3652865965_ed37959c-2640x2047](https://user-images.githubusercontent.com/59680068/223930346-c32a062c-4b3a-40a6-83e3-853127129b7d.jpg)
+![xyz_grid-0296-2023-03-08_d4d1ef62c3_KBlueLeaf_KBlueLeaf-v1 1_4001890147_ed37959c-2640x2047](https://user-images.githubusercontent.com/59680068/223930351-45653f67-7b0c-42e6-9a81-cc691d2b9dbf.jpg)
+

{lycoris_lora-1.8.0.dev6 → lycoris_lora-1.8.0.dev8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: lycoris_lora
-Version: 1.8.0.dev6
+Version: 1.8.0.dev8
 Summary: Lora beYond Conventional methods, Other Rank adaptation Implementations for Stable diffusion
 Home-page: https://github.com/KohakuBlueleaf/LyCORIS
 Author: Shih-Ying Yeh(KohakuBlueLeaf), Yu-Guan Hsieh, Zhidong Gao

lycoris_lora-1.8.0.dev8/experiments/better_conv.py

@@ -0,0 +1,105 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+class HadaWeightCP(torch.autograd.Function):
+    @staticmethod
+    def forward(ctx, orig_weight, t1, w1a, w1b, t2, w2a, w2b, scale=torch.tensor(1)):
+        ctx.save_for_backward(t1, w1a, w1b, t2, w2a, w2b, scale)
+        
+        temp = torch.einsum('i j k l, j r -> i r k l', t1, w1b)
+        rebuild1 = torch.einsum('i j k l, i r -> r j k l', temp, w1a)
+        
+        temp = torch.einsum('i j k l, j r -> i r k l', t2, w2b)
+        rebuild2 = torch.einsum('i j k l, i r -> r j k l', temp, w2a)
+        
+        return orig_weight + rebuild1*rebuild2*scale
+
+    @staticmethod
+    def backward(ctx, grad_out):
+        (t1, w1a, w1b, t2, w2a, w2b, scale) = ctx.saved_tensors
+        
+        grad_out = grad_out*scale
+        
+        temp = torch.einsum('i j k l, j r -> i r k l', t2, w2b)
+        rebuild = torch.einsum('i j k l, i r -> r j k l', temp, w2a)
+        
+        grad_w = rebuild*grad_out
+        del rebuild
+        
+        grad_w1a = torch.einsum('r j k l, i j k l -> r i', temp, grad_w)
+        grad_temp = torch.einsum('i j k l, i r -> r j k l', grad_w, w1a.T)
+        del grad_w, temp
+        
+        grad_w1b = torch.einsum('i r k l, i j k l -> r j', t1, grad_temp)
+        grad_t1 = torch.einsum('i j k l, j r -> i r k l', grad_temp, w1b.T)
+        del grad_temp
+        
+        temp = torch.einsum('i j k l, j r -> i r k l', t1, w1b)
+        rebuild = torch.einsum('i j k l, i r -> r j k l', temp, w1a)
+        
+        grad_w = rebuild*grad_out
+        del rebuild
+        
+        grad_w2a = torch.einsum('r j k l, i j k l -> r i', temp, grad_w)
+        grad_temp = torch.einsum('i j k l, i r -> r j k l', grad_w, w2a.T)
+        del grad_w, temp
+        
+        grad_w2b = torch.einsum('i r k l, i j k l -> r j', t2, grad_temp)
+        grad_t2 = torch.einsum('i j k l, j r -> i r k l', grad_temp, w2b.T)
+        del grad_temp
+        return grad_out, grad_t1, grad_w1a, grad_w1b, grad_t2, grad_w2a, grad_w2b, None
+
+
+def make_weight_cp(orig_weight, t1, w1a, w1b, t2, w2a, w2b, scale=torch.tensor(0.25)):
+    return HadaWeightCP.apply(orig_weight, t1, w1a, w1b, t2, w2a, w2b, scale)
+
+
+def make_cp(orig_weight, t1, w1a, w1b, t2, w2a, w2b, scale=torch.tensor(0.25)):
+    temp = torch.einsum('i j k l, j r -> i r k l', t1, w1b)
+    rebuild1 = torch.einsum('i j k l, i r -> r j k l', temp, w1a)
+    
+    temp = torch.einsum('i j k l, j r -> i r k l', t2, w2b)
+    rebuild2 = torch.einsum('i j k l, i r -> r j k l', temp, w2a)
+    
+    return orig_weight + rebuild1*rebuild2 * scale
+
+
+KERNEL_SIZE = 3
+STRIDE = 1
+PADDING = 1
+IN_CH = 1280
+OUT_CH = 1208
+LORA_RANK = 4
+SIZE = 32
+
+
+t1 = nn.Parameter(torch.randn(LORA_RANK, LORA_RANK, 3, 3))
+w1a = nn.Parameter(torch.randn(LORA_RANK, OUT_CH))
+w1b = nn.Parameter(torch.randn(LORA_RANK, IN_CH))
+
+t2 = nn.Parameter(torch.randn(LORA_RANK, LORA_RANK, 3, 3))
+w2a = nn.Parameter(torch.randn(LORA_RANK, OUT_CH))
+w2b = nn.Parameter(torch.randn(LORA_RANK, IN_CH))
+
+orig = nn.Parameter(torch.randn(OUT_CH, IN_CH, 3, 3))
+
+test_x = torch.randn(1, 4, 64, 64)
+test_t = torch.randn(1, 4, 64, 64)
+
+
+w1 = make_cp(orig, t1, w1a, w1b, t2, w2a, w2b)
+w2 = make_weight_cp(orig, t1, w1a, w1b, t2, w2a, w2b)
+
+torch.mean(w1).backward()
+grad1 = t1.grad.clone()
+t1.grad = None
+
+torch.mean(w2).backward()
+grad2 = t1.grad.clone()
+
+
+print('MSE Loss: ', F.mse_loss(grad1, grad2))
+print('L1 Loss : ', F.l1_loss(grad1, grad2))
+print('Distance: ', torch.dist(grad1, grad2))

lycoris_lora-1.8.0.dev8/experiments/better_conv_extract.py

@@ -0,0 +1,67 @@
+import os, sys
+sys.path.insert(0, os.getcwd())
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from lycoris.utils import extract_conv
+from pytorch_lightning import seed_everything
+
+
+def make_sparse(t, sparsity=0.95):
+    sparse_t = t.masked_fill(
+        torch.abs(t) < torch.quantile(torch.abs(t), sparsity), 
+        0
+    )
+    return sparse_t
+
+def make_cp(t, x1, x2):
+    x1 = x1.reshape(x1.size(0), -1)
+    x2 = x2.reshape(x2.size(0), -1)
+    
+    # [rank, rank, k, k] * [rank, out]
+    temp = torch.einsum('n m k l, i n -> i m k l', t, x2)
+    
+    # [out, rank, k, k] * [rank, in]
+    result = torch.einsum('i m k l, m j -> i j k l', temp, x1)
+    print(result.shape)
+    return result
+
+
+seed_everything(0)
+
+KERNEL_SIZE = 3
+STRIDE = 1
+PADDING = 1
+IN_CH = 1280
+OUT_CH = 640
+LORA_RANK = 32
+SIZE = 32
+
+conv_orig = nn.Conv2d(IN_CH, OUT_CH, 3, 1, 1, bias=False).cuda()
+conv_orig.weight = nn.Parameter(
+    (torch.randn(OUT_CH, LORA_RANK) 
+     @ torch.randn(LORA_RANK, IN_CH*9) 
+     * 0.05
+     ).reshape(conv_orig.weight.shape).cuda()
+)
+conv_rebd = nn.Conv2d(IN_CH, OUT_CH, 3, 1, 1, bias=False).cuda()
+
+extract_a, extract_b, diff = extract_conv(conv_orig.weight, 'fixed', LORA_RANK, 'cuda')
+out_weight = extract_b.reshape(OUT_CH, -1) @ extract_a.reshape(LORA_RANK, -1)
+out_weight = out_weight.reshape(conv_orig.weight.shape)
+
+extract_a, extract_c, diff = extract_conv(extract_a.transpose(0, 1), 'fixed', LORA_RANK, 'cuda')
+extract_a = extract_a.transpose(0, 1)
+extract_c = extract_c.transpose(0, 1)
+print(extract_a.shape, extract_b.shape, extract_c.shape)
+
+
+conv_rebd.weight = nn.Parameter(make_cp(
+    extract_a, extract_c, extract_b
+))
+print()
+print('without sparse bias')
+print('MSE Loss: ', F.mse_loss(conv_orig.weight, conv_rebd.weight))
+print('L1 Loss : ', F.l1_loss(conv_orig.weight, conv_rebd.weight))
+print('Distance: ', torch.dist(conv_orig.weight, conv_rebd.weight))