mmgp 1.0.4__tar.gz → 1.0.5__tar.gz

{mmgp-1.0.4 → mmgp-1.0.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: mmgp
-Version: 1.0.4
+Version: 1.0.5
 Summary: Memory Management for the GPU Poor
 Author-email: deepbeepmeep <deepbeepmeep@yahoo.com>
 License:                     GNU GENERAL PUBLIC LICENSE
@@ -685,38 +685,48 @@ Requires-Dist: optimum-quanto
 
 **------------------ Memory Management for the GPU Poor by DeepBeepMeep ------------------**
 
-This module contains multiples optimisations so that models such as Flux (and derived), Mochi, CogView, HunyuanVideo, ...  run smoothly on a 24 GB GPU limited card 
-This a replacement for the accelerate library that should in theory manage offloading, but doesn't work properly with models that are loaded / unloaded several times in a pipe (eg VAE) 
+This module contains multiples optimisations so that models such as Flux (and derived), Mochi, CogView, HunyuanVideo, ...  can run smoothly on a 24 GB GPU limited card. 
+This a replacement for the accelerate library that should in theory manage offloading, but doesn't work properly with models that are loaded / unloaded several
+times in a pipe (eg VAE).
 
 Requirements:
-- GPU: RTX 3090/ RTX 4090
+- GPU: RTX 3090/ RTX 4090 (24 GB of VRAM)
 - RAM: minimum 48 GB, recommended 64 GB 
 
 It is almost plug and play and just needs to be invoked from the main app just after the model pipeline has been created.
 1) First make sure that the pipeline explictly loads the models in the CPU device 
   for instance: pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16).to("cpu")
 2) Once every potential Lora has been loaded and merged, add the following lines:
-  from mmgp import offload
-  offload.me(pipe)
-If you don't have enough RAM you may disable RAM pinning but model switching option pinInRAM= False
-Sometime there isn't an explicit pipe object as each submodel is loaded separately in the main app. If this is the case, you need to create a dictionary that manually maps all the models^.
+
+  *from mmgp import offload*
+  *offload.me(pipe)*
+  
+The 'transformer' model that contains usually the video or image generator is quantized on the fly by default to 8 bits. If you want to save time on disk and reduce the loading time, you may want to load directly a prequantized model. In that case you need to set the option *quantizeTransformer* to *False* to turn off on the fly quantization.
+
+If you have more than 64GB RAM you may want to enable RAM pinning with the option *pinInRAM = True*. You will get in return super fast loading / unloading of models
+(this can save significant time if the same pipeline is run multiple times in a row)
+
+Sometime there isn't an explicit pipe object as each submodel is loaded separately in the main app. If this is the case, you need to create a dictionary that manually maps all the models.
 
 For instance :
-for flux derived models: pipe = { "text_encoder": clip, "text_encoder_2": t5, "transformer": model, "vae":ae }
-for mochi: pipe = { "text_encoder": self.text_encoder, "transformer": self.dit, "vae":self.decoder }
+for flux derived models: *pipe = { "text_encoder": clip, "text_encoder_2": t5, "transformer": model, "vae":ae }*
+for mochi: *pipe = { "text_encoder": self.text_encoder, "transformer": self.dit, "vae":self.decoder }*
 
-Please note that there should be always one model whose Id is 'transformer'. It is corresponds to the main image / video model which usually needs to be quantized (this is done by default)
+Please note that there should be always one model whose Id is 'transformer'. It corresponds to the main image / video model which usually needs to be quantized (this is done on the fly by default when loading the model).
 
-Becareful, lots of models uses the T5 XXL as a text encoder. However, quite often their corresponding pipeline configuratons points at the official Google T5 XXL repository 
+Becareful, lots of models use the T5 XXL as a text encoder. However, quite often their corresponding pipeline configurations point at the official Google T5 XXL repository 
 where there is a huge 40GB model to download and load. It is cumbersorme as it is a 32 bits model and contains the decoder part of T5 that is not used. 
 I suggest you use instead one of the 16 bits encoder only version available around, for instance:
-text_encoder_2 = T5EncoderModel.from_pretrained("black-forest-labs/FLUX.1-dev", subfolder="text_encoder_2", torch_dtype=torch.float16)
+*text_encoder_2 = T5EncoderModel.from_pretrained("black-forest-labs/FLUX.1-dev", subfolder="text_encoder_2", torch_dtype=torch.float16)*
 
-You are free to use my code for non commercial use as long you give me proper credits. You may contact me on twitter @deepbeepmeep
+Sometime just providing the pipe won't be sufficient as you will need to change the content of the core model: 
+- For instance you may need to disable an existing CPU offload logic that already exists (such as manual calls to move tensors between cuda and the cpu)
+- mmpg to tries to fake the device as being "cuda" but sometimes some code won't be fooled and it will create tensors in the cpu device and this may cause some issues.
 
+You are free to use my module for non commercial use as long you give me proper credits. You may contact me on twitter @deepbeepmeep
 
-Thanks  
--------
+Thanks to
+---------
 Huggingface / accelerate for the hooking examples
 Huggingface / quanto for their very useful quantizer
-gau-nernst for his Pinnig RAM examples
+gau-nernst for his Pinnig RAM samples

mmgp-1.0.5/README.md

@@ -0,0 +1,47 @@
+**------------------ Memory Management for the GPU Poor by DeepBeepMeep ------------------**
+
+This module contains multiples optimisations so that models such as Flux (and derived), Mochi, CogView, HunyuanVideo, ...  can run smoothly on a 24 GB GPU limited card. 
+This a replacement for the accelerate library that should in theory manage offloading, but doesn't work properly with models that are loaded / unloaded several
+times in a pipe (eg VAE).
+
+Requirements:
+- GPU: RTX 3090/ RTX 4090 (24 GB of VRAM)
+- RAM: minimum 48 GB, recommended 64 GB 
+
+It is almost plug and play and just needs to be invoked from the main app just after the model pipeline has been created.
+1) First make sure that the pipeline explictly loads the models in the CPU device 
+  for instance: pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16).to("cpu")
+2) Once every potential Lora has been loaded and merged, add the following lines:
+
+  *from mmgp import offload*
+  *offload.me(pipe)*
+  
+The 'transformer' model that contains usually the video or image generator is quantized on the fly by default to 8 bits. If you want to save time on disk and reduce the loading time, you may want to load directly a prequantized model. In that case you need to set the option *quantizeTransformer* to *False* to turn off on the fly quantization.
+
+If you have more than 64GB RAM you may want to enable RAM pinning with the option *pinInRAM = True*. You will get in return super fast loading / unloading of models
+(this can save significant time if the same pipeline is run multiple times in a row)
+
+Sometime there isn't an explicit pipe object as each submodel is loaded separately in the main app. If this is the case, you need to create a dictionary that manually maps all the models.
+
+For instance :
+for flux derived models: *pipe = { "text_encoder": clip, "text_encoder_2": t5, "transformer": model, "vae":ae }*
+for mochi: *pipe = { "text_encoder": self.text_encoder, "transformer": self.dit, "vae":self.decoder }*
+
+Please note that there should be always one model whose Id is 'transformer'. It corresponds to the main image / video model which usually needs to be quantized (this is done on the fly by default when loading the model).
+
+Becareful, lots of models use the T5 XXL as a text encoder. However, quite often their corresponding pipeline configurations point at the official Google T5 XXL repository 
+where there is a huge 40GB model to download and load. It is cumbersorme as it is a 32 bits model and contains the decoder part of T5 that is not used. 
+I suggest you use instead one of the 16 bits encoder only version available around, for instance:
+*text_encoder_2 = T5EncoderModel.from_pretrained("black-forest-labs/FLUX.1-dev", subfolder="text_encoder_2", torch_dtype=torch.float16)*
+
+Sometime just providing the pipe won't be sufficient as you will need to change the content of the core model: 
+- For instance you may need to disable an existing CPU offload logic that already exists (such as manual calls to move tensors between cuda and the cpu)
+- mmpg to tries to fake the device as being "cuda" but sometimes some code won't be fooled and it will create tensors in the cpu device and this may cause some issues.
+
+You are free to use my module for non commercial use as long you give me proper credits. You may contact me on twitter @deepbeepmeep
+
+Thanks to
+---------
+Huggingface / accelerate for the hooking examples
+Huggingface / quanto for their very useful quantizer
+gau-nernst for his Pinnig RAM samples

{mmgp-1.0.4 → mmgp-1.0.5}/pyproject.toml

@@ -70,5 +70,5 @@ sort_first = [
 [tool.setuptools_scm]
 write_to = "src/_version.py"
 parentdir_prefix_version = "mmgp-"
-fallback_version = "1.0.4"
+fallback_version = "1.0.5"
 version_scheme = "post-release"

{mmgp-1.0.4 → mmgp-1.0.5}/src/_version.py

@@ -12,5 +12,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '1.0.4'
-__version_tuple__ = version_tuple = (1, 0, 4)
+__version__ = version = '1.0.5'
+__version_tuple__ = version_tuple = (1, 0, 5)

{mmgp-1.0.4 → mmgp-1.0.5}/src/mmgp.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: mmgp
-Version: 1.0.4
+Version: 1.0.5
 Summary: Memory Management for the GPU Poor
 Author-email: deepbeepmeep <deepbeepmeep@yahoo.com>
 License:                     GNU GENERAL PUBLIC LICENSE
@@ -685,38 +685,48 @@ Requires-Dist: optimum-quanto
 
 **------------------ Memory Management for the GPU Poor by DeepBeepMeep ------------------**
 
-This module contains multiples optimisations so that models such as Flux (and derived), Mochi, CogView, HunyuanVideo, ...  run smoothly on a 24 GB GPU limited card 
-This a replacement for the accelerate library that should in theory manage offloading, but doesn't work properly with models that are loaded / unloaded several times in a pipe (eg VAE) 
+This module contains multiples optimisations so that models such as Flux (and derived), Mochi, CogView, HunyuanVideo, ...  can run smoothly on a 24 GB GPU limited card. 
+This a replacement for the accelerate library that should in theory manage offloading, but doesn't work properly with models that are loaded / unloaded several
+times in a pipe (eg VAE).
 
 Requirements:
-- GPU: RTX 3090/ RTX 4090
+- GPU: RTX 3090/ RTX 4090 (24 GB of VRAM)
 - RAM: minimum 48 GB, recommended 64 GB 
 
 It is almost plug and play and just needs to be invoked from the main app just after the model pipeline has been created.
 1) First make sure that the pipeline explictly loads the models in the CPU device 
   for instance: pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16).to("cpu")
 2) Once every potential Lora has been loaded and merged, add the following lines:
-  from mmgp import offload
-  offload.me(pipe)
-If you don't have enough RAM you may disable RAM pinning but model switching option pinInRAM= False
-Sometime there isn't an explicit pipe object as each submodel is loaded separately in the main app. If this is the case, you need to create a dictionary that manually maps all the models^.
+
+  *from mmgp import offload*
+  *offload.me(pipe)*
+  
+The 'transformer' model that contains usually the video or image generator is quantized on the fly by default to 8 bits. If you want to save time on disk and reduce the loading time, you may want to load directly a prequantized model. In that case you need to set the option *quantizeTransformer* to *False* to turn off on the fly quantization.
+
+If you have more than 64GB RAM you may want to enable RAM pinning with the option *pinInRAM = True*. You will get in return super fast loading / unloading of models
+(this can save significant time if the same pipeline is run multiple times in a row)
+
+Sometime there isn't an explicit pipe object as each submodel is loaded separately in the main app. If this is the case, you need to create a dictionary that manually maps all the models.
 
 For instance :
-for flux derived models: pipe = { "text_encoder": clip, "text_encoder_2": t5, "transformer": model, "vae":ae }
-for mochi: pipe = { "text_encoder": self.text_encoder, "transformer": self.dit, "vae":self.decoder }
+for flux derived models: *pipe = { "text_encoder": clip, "text_encoder_2": t5, "transformer": model, "vae":ae }*
+for mochi: *pipe = { "text_encoder": self.text_encoder, "transformer": self.dit, "vae":self.decoder }*
 
-Please note that there should be always one model whose Id is 'transformer'. It is corresponds to the main image / video model which usually needs to be quantized (this is done by default)
+Please note that there should be always one model whose Id is 'transformer'. It corresponds to the main image / video model which usually needs to be quantized (this is done on the fly by default when loading the model).
 
-Becareful, lots of models uses the T5 XXL as a text encoder. However, quite often their corresponding pipeline configuratons points at the official Google T5 XXL repository 
+Becareful, lots of models use the T5 XXL as a text encoder. However, quite often their corresponding pipeline configurations point at the official Google T5 XXL repository 
 where there is a huge 40GB model to download and load. It is cumbersorme as it is a 32 bits model and contains the decoder part of T5 that is not used. 
 I suggest you use instead one of the 16 bits encoder only version available around, for instance:
-text_encoder_2 = T5EncoderModel.from_pretrained("black-forest-labs/FLUX.1-dev", subfolder="text_encoder_2", torch_dtype=torch.float16)
+*text_encoder_2 = T5EncoderModel.from_pretrained("black-forest-labs/FLUX.1-dev", subfolder="text_encoder_2", torch_dtype=torch.float16)*
 
-You are free to use my code for non commercial use as long you give me proper credits. You may contact me on twitter @deepbeepmeep
+Sometime just providing the pipe won't be sufficient as you will need to change the content of the core model: 
+- For instance you may need to disable an existing CPU offload logic that already exists (such as manual calls to move tensors between cuda and the cpu)
+- mmpg to tries to fake the device as being "cuda" but sometimes some code won't be fooled and it will create tensors in the cpu device and this may cause some issues.
 
+You are free to use my module for non commercial use as long you give me proper credits. You may contact me on twitter @deepbeepmeep
 
-Thanks  
--------
+Thanks to
+---------
 Huggingface / accelerate for the hooking examples
 Huggingface / quanto for their very useful quantizer
-gau-nernst for his Pinnig RAM examples
+gau-nernst for his Pinnig RAM samples

{mmgp-1.0.4 → mmgp-1.0.5}/src/mmgp.py

@@ -1,10 +1,11 @@
-# ------------------ Memory Management for the GPU Poor by DeepBeepMeep ------------------
+# ------------------ Memory Management for the GPU Poor by DeepBeepMeep (mmgp)------------------
 #
-# This module contains multiples optimisations so that models such as Flux (and derived), Mochi, CogView, HunyuanVideo, ...  run smoothly on a 24 GB GPU limited card 
-# This a replacement for the accelerate library that should in theory manage offloading, but doesn't work properly with models that are loaded / unloaded several times in a pipe (eg VAE) 
+# This module contains multiples optimisations so that models such as Flux (and derived), Mochi, CogView, HunyuanVideo, ...  can run smoothly on a 24 GB GPU limited card. 
+# This a replacement for the accelerate library that should in theory manage offloading, but doesn't work properly with models that are loaded / unloaded several
+# times in a pipe (eg VAE).
 #
 # Requirements:
-# - GPU: RTX 3090/ RTX 4090
+# - GPU: RTX 3090/ RTX 4090 (24 GB of VRAM)
 # - RAM: minimum 48 GB, recommended 64 GB 
 #
 # It is almost plug and play and just needs to be invoked from the main app just after the model pipeline has been created.
@@ -13,27 +14,35 @@
 # 2) Once every potential Lora has been loaded and merged, add the following lines:
 #   from mmgp import offload
 #   offload.me(pipe)
-# If you don't have enough RAM you may disable RAM pinning but model switching option pinInRAM= False
-# Sometime there isn't an explicit pipe object as each submodel is loaded separately in the main app. If this is the case, you need to create a dictionary that manually maps all the models^.
+# The 'transformer' model that contains usually the video or image generator is quantized on the fly by default to 8 bits. If you want to save time on disk and reduce the loading time, you may want to load directly a prequantized model. In that case you need to set the option quantizeTransformer to False to turn off on the fly quantization.
+#
+# If you have more than 64GB RAM you may want to enable RAM pinning with the option pinInRAM = True. You will get in return super fast loading / unloading of models
+# (this can save significant time if the same pipeline is run multiple times in a row)
+# 
+# Sometime there isn't an explicit pipe object as each submodel is loaded separately in the main app. If this is the case, you need to create a dictionary that manually maps all the models.
 #
 # For instance :
 # for flux derived models: pipe = { "text_encoder": clip, "text_encoder_2": t5, "transformer": model, "vae":ae }
 # for mochi: pipe = { "text_encoder": self.text_encoder, "transformer": self.dit, "vae":self.decoder }
 #
-# Please note that there should be always one model whose Id is 'transformer'. It is corresponds to the main image / video model which usually needs to be quantized (this is done by default)
+# Please note that there should be always one model whose Id is 'transformer'. It corresponds to the main image / video model which usually needs to be quantized (this is done on the fly by default when loading the model)
 # 
-# Becareful, lots of models uses the T5 XXL as a text encoder. However, quite often their corresponding pipeline configuratons points at the official Google T5 XXL repository 
+# Becareful, lots of models use the T5 XXL as a text encoder. However, quite often their corresponding pipeline configurations point at the official Google T5 XXL repository 
 # where there is a huge 40GB model to download and load. It is cumbersorme as it is a 32 bits model and contains the decoder part of T5 that is not used. 
 # I suggest you use instead one of the 16 bits encoder only version available around, for instance:
 # text_encoder_2 = T5EncoderModel.from_pretrained("black-forest-labs/FLUX.1-dev", subfolder="text_encoder_2", torch_dtype=torch.float16)
 #
-# You are free to use my code for non commercial use as long you give me proper credits. You may contact me on twitter @deepbeepmeep
+# Sometime just providing the pipe won't be sufficient as you will need to change the content of the core model: 
+# - For instance you may need to disable an existing CPU offload logic that already exists (such as manual calls to move tensors between cuda and the cpu)
+# - mmpg to tries to fake the device as being "cuda" but sometimes some code won't be fooled and it will create tensors in the cpu device and this may cause some issues.
+# 
+# You are free to use my module for non commercial use as long you give me proper credits. You may contact me on twitter @deepbeepmeep
 #
-# Thanks 
-# -------
+# Thanks to
+# ---------
 # Huggingface / accelerate for the hooking examples
 # Huggingface / quanto for their very useful quantizer
-# gau-nernst for his Pinnig RAM examples
+# gau-nernst for his Pinnig RAM samples
 
 
 #
@@ -45,8 +54,6 @@ import functools
 from optimum.quanto import freeze, qfloat8, qint8, quantize, QModuleMixin, QTensor
 
 
-# config Dimension X (CogVideo derived ) : Quantization False: because Lora applied later
-
 
 cotenants_map = { 
                              "text_encoder": ["vae", "text_encoder_2"],

mmgp-1.0.4/README.md

@@ -1,37 +0,0 @@
-**------------------ Memory Management for the GPU Poor by DeepBeepMeep ------------------**
-
-This module contains multiples optimisations so that models such as Flux (and derived), Mochi, CogView, HunyuanVideo, ...  run smoothly on a 24 GB GPU limited card 
-This a replacement for the accelerate library that should in theory manage offloading, but doesn't work properly with models that are loaded / unloaded several times in a pipe (eg VAE) 
-
-Requirements:
-- GPU: RTX 3090/ RTX 4090
-- RAM: minimum 48 GB, recommended 64 GB 
-
-It is almost plug and play and just needs to be invoked from the main app just after the model pipeline has been created.
-1) First make sure that the pipeline explictly loads the models in the CPU device 
-  for instance: pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16).to("cpu")
-2) Once every potential Lora has been loaded and merged, add the following lines:
-  from mmgp import offload
-  offload.me(pipe)
-If you don't have enough RAM you may disable RAM pinning but model switching option pinInRAM= False
-Sometime there isn't an explicit pipe object as each submodel is loaded separately in the main app. If this is the case, you need to create a dictionary that manually maps all the models^.
-
-For instance :
-for flux derived models: pipe = { "text_encoder": clip, "text_encoder_2": t5, "transformer": model, "vae":ae }
-for mochi: pipe = { "text_encoder": self.text_encoder, "transformer": self.dit, "vae":self.decoder }
-
-Please note that there should be always one model whose Id is 'transformer'. It is corresponds to the main image / video model which usually needs to be quantized (this is done by default)
-
-Becareful, lots of models uses the T5 XXL as a text encoder. However, quite often their corresponding pipeline configuratons points at the official Google T5 XXL repository 
-where there is a huge 40GB model to download and load. It is cumbersorme as it is a 32 bits model and contains the decoder part of T5 that is not used. 
-I suggest you use instead one of the 16 bits encoder only version available around, for instance:
-text_encoder_2 = T5EncoderModel.from_pretrained("black-forest-labs/FLUX.1-dev", subfolder="text_encoder_2", torch_dtype=torch.float16)
-
-You are free to use my code for non commercial use as long you give me proper credits. You may contact me on twitter @deepbeepmeep
-
-
-Thanks  
--------
-Huggingface / accelerate for the hooking examples
-Huggingface / quanto for their very useful quantizer
-gau-nernst for his Pinnig RAM examples