mmgp 1.1.0__tar.gz → 1.2.0__tar.gz

mmgp-1.2.0/LICENSE.md

@@ -0,0 +1,2 @@
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007

mmgp-1.2.0/PKG-INFO

@@ -0,0 +1,109 @@
+Metadata-Version: 2.1
+Name: mmgp
+Version: 1.2.0
+Summary: Memory Management for the GPU Poor
+Author-email: deepbeepmeep <deepbeepmeep@yahoo.com>
+License:                     GNU GENERAL PUBLIC LICENSE
+                               Version 3, 29 June 2007
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE.md
+Requires-Dist: torch>=2.1.0
+Requires-Dist: optimum-quanto
+
+
+<p align="center">
+  <H2>Memory Management for the GPU Poor by DeepBeepMeep</H2>	
+</p>
+
+
+This module contains multiples optimisations so that models such as Flux (and derived), Mochi, CogView, HunyuanVideo, ...  can run smoothly on a 24 GB GPU limited card. 
+This a replacement for the accelerate library that should in theory manage offloading, but doesn't work properly with models that are loaded / unloaded several
+times in a pipe (eg VAE).
+
+Requirements:
+- GPU: RTX 3090/ RTX 4090 (24 GB of VRAM)
+- RAM: minimum 48 GB, recommended 64 GB 
+
+## Usage 
+First you need to install the module in your current project with:
+```shell
+pip install mmgp
+```
+
+It is almost plug and play and just needs to be invoked from the main app just after the model pipeline has been created.
+1) First make sure that the pipeline explictly loads the models in the CPU device, for instance:
+```
+  pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16).to("cpu")
+```
+
+2) Once every potential Lora has been loaded and merged, add the following lines:
+
+```
+  from mmgp import offload
+  offload.all(pipe)
+```  
+
+## Options
+The 'transformer' model in the pipe contains usually the video or image generator is quantized on the fly by default to 8 bits. If you want to save time on disk and reduce the loading time, you may want to load directly a prequantized model. In that case you need to set the option *quantizeTransformer* to *False* to turn off on the fly quantization.
+
+You can specify a list of additional models string ids to quantize (for instance the text_encoder) using the optional argument *modelsToQuantize* for instance *modelsToQuantize = ["text_encoder_2"]*.This may be useful if you have less than 48 GB of RAM.
+
+Note that there is little advantage on the GPU / VRAM side to quantize text encoders as their inputs are usually quite light. 
+
+Conversely if you have more than 64GB of RAM you may want to enable RAM pinning with the option *pinInRAM = True*. You will get in return super fast loading / unloading of models
+(this can save significant time if the same pipeline is run multiple times in a row)
+
+In Summary, if you have:
+- Between 32 GB and 48 GB of RAM
+```
+  offload.all(pipe, modelsToQuantize = ["text_encoder_2"]) # for Flux models
+  #OR
+  offload.all(pipe, modelsToQuantize = ["text_encoder"]) # for HunyuanVideo models
+
+```  
+
+- Between 48 GB and 64 GB of RAM
+```
+  offload.all(pipe)
+```  
+- More than 64 GB of RAM
+```
+  offload.all(pipe), pinInRAM = True
+```
+
+## Special
+Sometime there isn't an explicit pipe object as each submodel is loaded separately in the main app. If this is the case, you need to create a dictionary that manually maps all the models.\
+For instance :
+
+
+- for flux derived models: 
+```
+pipe = { "text_encoder": clip, "text_encoder_2": t5, "transformer": model, "vae":ae }
+```
+- for mochi: 
+```
+pipe = { "text_encoder": self.text_encoder, "transformer": self.dit, "vae":self.decoder }
+```
+
+
+Please note that there should be always one model whose Id is 'transformer'. It corresponds to the main image / video model which usually needs to be quantized (this is done on the fly by default when loading the model).
+
+Becareful, lots of models use the T5 XXL as a text encoder. However, quite often their corresponding pipeline configurations point at the official Google T5 XXL repository 
+where there is a huge 40GB model to download and load. It is cumbersorme as it is a 32 bits model and contains the decoder part of T5 that is not used. 
+I suggest you use instead one of the 16 bits encoder only version available around, for instance:
+```
+text_encoder_2 = T5EncoderModel.from_pretrained("black-forest-labs/FLUX.1-dev", subfolder="text_encoder_2", torch_dtype=torch.float16)
+```
+
+Sometime just providing the pipe won't be sufficient as you will need to change the content of the core model: 
+- For instance you may need to disable an existing CPU offload logic that already exists (such as manual calls to move tensors between cuda and the cpu)
+- mmpg to tries to fake the device as being "cuda" but sometimes some code won't be fooled and it will create tensors in the cpu device and this may cause some issues.
+
+You are free to use my module for non commercial use as long you give me proper credits. You may contact me on twitter @deepbeepmeep
+
+Thanks to
+---------
+- Huggingface / accelerate for the hooking examples
+- Huggingface / quanto for their very useful quantizer
+- gau-nernst for his Pinnig RAM samples

{mmgp-1.1.0 → mmgp-1.2.0}/README.md

@@ -12,6 +12,7 @@ Requirements:
 - GPU: RTX 3090/ RTX 4090 (24 GB of VRAM)
 - RAM: minimum 48 GB, recommended 64 GB 
 
+## Usage 
 First you need to install the module in your current project with:
 ```shell
 pip install mmgp
@@ -27,13 +28,38 @@ It is almost plug and play and just needs to be invoked from the main app just a
 
 ```
   from mmgp import offload
-  offload.me(pipe)
+  offload.all(pipe)
 ```  
+
+## Options
 The 'transformer' model in the pipe contains usually the video or image generator is quantized on the fly by default to 8 bits. If you want to save time on disk and reduce the loading time, you may want to load directly a prequantized model. In that case you need to set the option *quantizeTransformer* to *False* to turn off on the fly quantization.
 
-If you have more than 64GB RAM you may want to enable RAM pinning with the option *pinInRAM = True*. You will get in return super fast loading / unloading of models
+You can specify a list of additional models string ids to quantize (for instance the text_encoder) using the optional argument *modelsToQuantize* for instance *modelsToQuantize = ["text_encoder_2"]*.This may be useful if you have less than 48 GB of RAM.
+
+Note that there is little advantage on the GPU / VRAM side to quantize text encoders as their inputs are usually quite light. 
+
+Conversely if you have more than 64GB of RAM you may want to enable RAM pinning with the option *pinInRAM = True*. You will get in return super fast loading / unloading of models
 (this can save significant time if the same pipeline is run multiple times in a row)
 
+In Summary, if you have:
+- Between 32 GB and 48 GB of RAM
+```
+  offload.all(pipe, modelsToQuantize = ["text_encoder_2"]) # for Flux models
+  #OR
+  offload.all(pipe, modelsToQuantize = ["text_encoder"]) # for HunyuanVideo models
+
+```  
+
+- Between 48 GB and 64 GB of RAM
+```
+  offload.all(pipe)
+```  
+- More than 64 GB of RAM
+```
+  offload.all(pipe), pinInRAM = True
+```
+
+## Special
 Sometime there isn't an explicit pipe object as each submodel is loaded separately in the main app. If this is the case, you need to create a dictionary that manually maps all the models.\
 For instance :
 

{mmgp-1.1.0 → mmgp-1.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mmgp"
-version = "1.1.0"
+version = "1.2.0"
 authors = [
   { name = "deepbeepmeep", email = "deepbeepmeep@yahoo.com" },
 ]

mmgp-1.2.0/src/mmgp.egg-info/PKG-INFO

@@ -0,0 +1,109 @@
+Metadata-Version: 2.1
+Name: mmgp
+Version: 1.2.0
+Summary: Memory Management for the GPU Poor
+Author-email: deepbeepmeep <deepbeepmeep@yahoo.com>
+License:                     GNU GENERAL PUBLIC LICENSE
+                               Version 3, 29 June 2007
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE.md
+Requires-Dist: torch>=2.1.0
+Requires-Dist: optimum-quanto
+
+
+<p align="center">
+  <H2>Memory Management for the GPU Poor by DeepBeepMeep</H2>	
+</p>
+
+
+This module contains multiples optimisations so that models such as Flux (and derived), Mochi, CogView, HunyuanVideo, ...  can run smoothly on a 24 GB GPU limited card. 
+This a replacement for the accelerate library that should in theory manage offloading, but doesn't work properly with models that are loaded / unloaded several
+times in a pipe (eg VAE).
+
+Requirements:
+- GPU: RTX 3090/ RTX 4090 (24 GB of VRAM)
+- RAM: minimum 48 GB, recommended 64 GB 
+
+## Usage 
+First you need to install the module in your current project with:
+```shell
+pip install mmgp
+```
+
+It is almost plug and play and just needs to be invoked from the main app just after the model pipeline has been created.
+1) First make sure that the pipeline explictly loads the models in the CPU device, for instance:
+```
+  pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16).to("cpu")
+```
+
+2) Once every potential Lora has been loaded and merged, add the following lines:
+
+```
+  from mmgp import offload
+  offload.all(pipe)
+```  
+
+## Options
+The 'transformer' model in the pipe contains usually the video or image generator is quantized on the fly by default to 8 bits. If you want to save time on disk and reduce the loading time, you may want to load directly a prequantized model. In that case you need to set the option *quantizeTransformer* to *False* to turn off on the fly quantization.
+
+You can specify a list of additional models string ids to quantize (for instance the text_encoder) using the optional argument *modelsToQuantize* for instance *modelsToQuantize = ["text_encoder_2"]*.This may be useful if you have less than 48 GB of RAM.
+
+Note that there is little advantage on the GPU / VRAM side to quantize text encoders as their inputs are usually quite light. 
+
+Conversely if you have more than 64GB of RAM you may want to enable RAM pinning with the option *pinInRAM = True*. You will get in return super fast loading / unloading of models
+(this can save significant time if the same pipeline is run multiple times in a row)
+
+In Summary, if you have:
+- Between 32 GB and 48 GB of RAM
+```
+  offload.all(pipe, modelsToQuantize = ["text_encoder_2"]) # for Flux models
+  #OR
+  offload.all(pipe, modelsToQuantize = ["text_encoder"]) # for HunyuanVideo models
+
+```  
+
+- Between 48 GB and 64 GB of RAM
+```
+  offload.all(pipe)
+```  
+- More than 64 GB of RAM
+```
+  offload.all(pipe), pinInRAM = True
+```
+
+## Special
+Sometime there isn't an explicit pipe object as each submodel is loaded separately in the main app. If this is the case, you need to create a dictionary that manually maps all the models.\
+For instance :
+
+
+- for flux derived models: 
+```
+pipe = { "text_encoder": clip, "text_encoder_2": t5, "transformer": model, "vae":ae }
+```
+- for mochi: 
+```
+pipe = { "text_encoder": self.text_encoder, "transformer": self.dit, "vae":self.decoder }
+```
+
+
+Please note that there should be always one model whose Id is 'transformer'. It corresponds to the main image / video model which usually needs to be quantized (this is done on the fly by default when loading the model).
+
+Becareful, lots of models use the T5 XXL as a text encoder. However, quite often their corresponding pipeline configurations point at the official Google T5 XXL repository 
+where there is a huge 40GB model to download and load. It is cumbersorme as it is a 32 bits model and contains the decoder part of T5 that is not used. 
+I suggest you use instead one of the 16 bits encoder only version available around, for instance:
+```
+text_encoder_2 = T5EncoderModel.from_pretrained("black-forest-labs/FLUX.1-dev", subfolder="text_encoder_2", torch_dtype=torch.float16)
+```
+
+Sometime just providing the pipe won't be sufficient as you will need to change the content of the core model: 
+- For instance you may need to disable an existing CPU offload logic that already exists (such as manual calls to move tensors between cuda and the cpu)
+- mmpg to tries to fake the device as being "cuda" but sometimes some code won't be fooled and it will create tensors in the cpu device and this may cause some issues.
+
+You are free to use my module for non commercial use as long you give me proper credits. You may contact me on twitter @deepbeepmeep
+
+Thanks to
+---------
+- Huggingface / accelerate for the hooking examples
+- Huggingface / quanto for their very useful quantizer
+- gau-nernst for his Pinnig RAM samples

{mmgp-1.1.0 → mmgp-1.2.0}/src/mmgp.py

@@ -13,10 +13,12 @@
 #   for instance: pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16).to("cpu")
 # 2) Once every potential Lora has been loaded and merged, add the following lines:
 #   from mmgp import offload
-#   offload.me(pipe)
-# The 'transformer' model that contains usually the video or image generator is quantized on the fly by default to 8 bits. If you want to save time on disk and reduce the loading time, you may want to load directly a prequantized model. In that case you need to set the option quantizeTransformer to False to turn off on the fly quantization.
-#
-# If you have more than 64GB RAM you may want to enable RAM pinning with the option pinInRAM = True. You will get in return super fast loading / unloading of models
+#   offload.all(pipe)
+# The 'transformer' model that contains usually the video or image generator is quantized on the fly by default to 8 bits so that it can fit into 24 GB of VRAM. 
+# If you want to save time on disk and reduce the loading time, you may want to load directly a prequantized model. In that case you need to set the option quantizeTransformer to False to turn off on the fly quantization.
+# You can specify a list of additional models string ids to quantize (for instance the text_encoder) using the optional argument modelsToQuantize. This may be useful if you have less than 48 GB of RAM.
+# Note that there is little advantage on the GPU / VRAM side to quantize text encoders as their inputs are usually quite light. 
+# Conversely if you have more than 64GB RAM you may want to enable RAM pinning with the option pinInRAM = True. You will get in return super fast loading / unloading of models
 # (this can save significant time if the same pipeline is run multiple times in a row)
 # 
 # Sometime there isn't an explicit pipe object as each submodel is loaded separately in the main app. If this is the case, you need to create a dictionary that manually maps all the models.
@@ -263,7 +265,7 @@ class offload:
   
 
     @classmethod
-    def all(cls, pipe_or_dict_of_modules, quantizeTransformer = True, pinInRAM = False,  verbose = True):
+    def all(cls, pipe_or_dict_of_modules, quantizeTransformer = True, pinInRAM = False,  verbose = True, modelsToQuantize = None ):
         self = cls()
         self.verbose = verbose
         self.pinned_modules_data = {}
@@ -284,8 +286,12 @@ class offload:
         
         models = {k: v for k, v in pipe_or_dict_of_modules.items() if isinstance(v, torch.nn.Module)}
 
+        modelsToQuantize =  modelsToQuantize if modelsToQuantize is not None else []
+        if not isinstance(modelsToQuantize, list):
+            modelsToQuantize = [modelsToQuantize]
         if quantizeTransformer:
-            self.models_to_quantize = ["transformer"]
+            modelsToQuantize.append("transformer")
+        self.models_to_quantize = modelsToQuantize
  #       del  models["transformer"] # to test everything but the transformer that has a much longer loading
  #       models = { 'transformer': pipe_or_dict_of_modules["transformer"]} # to test only the transformer
         for model_id in models: