libcom 0.1.2__tar.gz → 0.1.3__tar.gz

{libcom-0.1.2 → libcom-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.1
 Name: libcom
-Version: 0.1.2
+Version: 0.1.3
 Summary: Image Composition Toolbox
 Home-page: https://github.com/bcmi/libcom
 Author: BCMI Lab
@@ -13,38 +13,6 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: einops
-Requires-Dist: huggingface_hub
-Requires-Dist: imageio
-Requires-Dist: torch==2.6.0
-Requires-Dist: torchvision==0.21.0
-Requires-Dist: diffusers==0.34.0
-Requires-Dist: mmdet==3.2.0
-Requires-Dist: mmpose==1.2.0
-Requires-Dist: peft
-Requires-Dist: sentencepiece
-Requires-Dist: protobuf
-Requires-Dist: modelscope
-Requires-Dist: omegaconf
-Requires-Dist: opencv_python
-Requires-Dist: opencv_python_headless
-Requires-Dist: pytorch_lightning
-Requires-Dist: transformers
-Requires-Dist: tqdm
-Requires-Dist: open_clip_torch
-Requires-Dist: taming-transformers-rom1504
-Requires-Dist: scikit-learn
-Dynamic: author
-Dynamic: author-email
-Dynamic: classifier
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: keywords
-Dynamic: license
-Dynamic: license-file
-Dynamic: requires-dist
-Dynamic: summary
 
 <div align="center">
 </br>
@@ -86,24 +54,25 @@ Welcome to follow WeChat public account ["Newly AIGCer"](https://www.ustcnewly.c
 - **OPAScoreModel** [[OPA]](https://github.com/bcmi/Object-Placement-Assessment-Dataset-OPA) evaluates the rationality of foreground object placement in a composite image.
 - **FOPAHeatMapModel** [[FOPA]](https://github.com/bcmi/FOPA-Fast-Object-Placement-Assessment) can predict the rationality scores for all locations/scales given a background-foreground pair, and output the composite image with optimal location/scale.  
 - **color_transfer** adjusts the foreground color to match the background using traditional color transfer method.
-- **ImageHarmonizationModel** [[CDTNet]](https://github.com/bcmi/CDTNet-High-Resolution-Image-Harmonization)  [[PCTNet]](https://github.com/rakutentech/PCT-Net-Image-Harmonization) adjusts the foreground illumination to be compatible the background given photorealistic background and photorealistic foreground.
+- **ImageHarmonizationModel** [[PCTNet]](https://github.com/rakutentech/PCT-Net-Image-Harmonization) adjusts the foreground illumination to be compatible the background given photorealistic background and photorealistic foreground.
 - **PainterlyHarmonizationModel** [[PHDNet]](https://github.com/bcmi/PHDNet-Painterly-Image-Harmonization)  [[PHDiffusion]](https://github.com/bcmi/PHDiffusion-Painterly-Image-Harmonization) adjusts the foreground style to be compatible with the background given artistic background and photorealistic foreground.
 - **HarmonyScoreModel** [[BargainNet]](https://github.com/bcmi/BargainNet-Image-Harmonization) evaluates the harmony level between foreground and background in a composite image.
 - **InharmoniousLocalizationModel** [[MadisNet]](https://github.com/bcmi/MadisNet-Inharmonious-Region-Localization) localizes the inharmonious region in a synthetic image.
 - **FOSScoreModel** [[DiscoFOS]](https://github.com/bcmi/Foreground-Object-Search-Dataset-FOSD) evaluates the compatibility between foreground and background in a composite image in terms of geometry and semantics.
-- **ShadowGenerationModel** [[GPSDiffusion]](https://github.com/bcmi/GPSDiffusion-Object-Shadow-Generation) generates plausible shadow for the inserted object in a composite image. This model is unstable and you can pick the most satisfactory one from multiple generated results. 
-- **ControlComModel** [[ControlCom]](https://github.com/bcmi/ControlCom-Image-Composition) is a generative image composition model which unifies image blending and image harmonization. The pose and view of foreground stay unchanged. **Note that in the provided foreground image, the foreground object's length and width should fully extend to the edges of the image (see our example), otherwise the performance would be severely affected.**
-- **MureObjectStitchModel** [[MureObjectStitch]](https://github.com/bcmi/MureObjectStitch-Image-Composition) is another generative image composition model which can adjust the pose and view of foreground. It supports multiple reference images of one foreground object. If you have a few images containing the foreground object, we suggest finetuning MureObjectStitch using these images for better detail preservation. **Note that in the provided foreground image, the foreground object's length and width should fully extend to the edges of the image (see our example), otherwise the performance would be severely affected.**
+- **ShadowGenerationModel** [[GPSDiffusion]](https://github.com/bcmi/GPSDiffusion-Object-Shadow-Generation) generates plausible shadow for the inserted object in a composite image. 
+- **ReflectionGenerationModel** generates plausible reflection for the inserted object in a composite image. 
+- **KontextBlendingHarmonizationModel** [[FluxKontext]](https://github.com/black-forest-labs/flux) is a generative image composition model which inserts foreground into the specified bounding box in the background. The pose and view of foreground stay unchanged. The "blending" mode does not adjust the foreground illumination, while the "harmonization" mode adjusts the foreground illumination to make the composite image harmonious. 
+- **InsertAnythingModel** [[InsertAnything]](https://github.com/song-wensong/insert-anything) is a generative image composition model which inserts foreground into the specified bounding box in the background. The model has reasonable ability to adjust the pose and view of foreground according to the background. 
 
 
 **For the detailed method descriptions, code examples, visualization results, and performance comments, please refer to our [[documents]](https://libcom.readthedocs.io/en/latest/).** If the model performance is not satisfactory, you can finetune the pretrained model on your own dataset using the source repository and replace the original model. 
 
 ## Requirements
 
-The main branch is built on the Linux system with **Python 3.8** and **PyTorch>=1.10.1**. For other dependencies, please refer to [[conda_env]](https://github.com/bcmi/libcom/blob/main/requirements/libcom.yaml) and [[runtime_dependencies]](https://github.com/bcmi/libcom/blob/main/requirements/runtime.txt).
+The main branch is built on the Linux system with **Python 3.10** and **PyTorch>=2.6**. For other dependencies, please refer to [[requirements]](https://github.com/bcmi/libcom/blob/main/requirements.txt).
 
 ## Get Started
-Please refer to [[Installation]](https://github.com/bcmi/libcom/blob/test/docs/get_started.md) for installation instructions and [[documents]](https://libcom.readthedocs.io/en/latest/) for user guidance.
+Please refer to [[Installation]](https://github.com/bcmi/libcom/blob/main/docs/get_started.md) for installation instructions and [[documents]](https://libcom.readthedocs.io/en/latest/) for user guidance.
 
 ## Contributors
 - Institution: [Brain-like Computing and Machine Intelligence (BCMI) Lab](https://bcmi.sjtu.edu.cn/).
@@ -128,3 +97,4 @@ If you use our toolbox, please cite our survey paper using the following BibTeX
   year={2021}
 }
 ```
+

{libcom-0.1.2 → libcom-0.1.3}/README.md

@@ -38,24 +38,25 @@ Welcome to follow WeChat public account ["Newly AIGCer"](https://www.ustcnewly.c
 - **OPAScoreModel** [[OPA]](https://github.com/bcmi/Object-Placement-Assessment-Dataset-OPA) evaluates the rationality of foreground object placement in a composite image.
 - **FOPAHeatMapModel** [[FOPA]](https://github.com/bcmi/FOPA-Fast-Object-Placement-Assessment) can predict the rationality scores for all locations/scales given a background-foreground pair, and output the composite image with optimal location/scale.  
 - **color_transfer** adjusts the foreground color to match the background using traditional color transfer method.
-- **ImageHarmonizationModel** [[CDTNet]](https://github.com/bcmi/CDTNet-High-Resolution-Image-Harmonization)  [[PCTNet]](https://github.com/rakutentech/PCT-Net-Image-Harmonization) adjusts the foreground illumination to be compatible the background given photorealistic background and photorealistic foreground.
+- **ImageHarmonizationModel** [[PCTNet]](https://github.com/rakutentech/PCT-Net-Image-Harmonization) adjusts the foreground illumination to be compatible the background given photorealistic background and photorealistic foreground.
 - **PainterlyHarmonizationModel** [[PHDNet]](https://github.com/bcmi/PHDNet-Painterly-Image-Harmonization)  [[PHDiffusion]](https://github.com/bcmi/PHDiffusion-Painterly-Image-Harmonization) adjusts the foreground style to be compatible with the background given artistic background and photorealistic foreground.
 - **HarmonyScoreModel** [[BargainNet]](https://github.com/bcmi/BargainNet-Image-Harmonization) evaluates the harmony level between foreground and background in a composite image.
 - **InharmoniousLocalizationModel** [[MadisNet]](https://github.com/bcmi/MadisNet-Inharmonious-Region-Localization) localizes the inharmonious region in a synthetic image.
 - **FOSScoreModel** [[DiscoFOS]](https://github.com/bcmi/Foreground-Object-Search-Dataset-FOSD) evaluates the compatibility between foreground and background in a composite image in terms of geometry and semantics.
-- **ShadowGenerationModel** [[GPSDiffusion]](https://github.com/bcmi/GPSDiffusion-Object-Shadow-Generation) generates plausible shadow for the inserted object in a composite image. This model is unstable and you can pick the most satisfactory one from multiple generated results. 
-- **ControlComModel** [[ControlCom]](https://github.com/bcmi/ControlCom-Image-Composition) is a generative image composition model which unifies image blending and image harmonization. The pose and view of foreground stay unchanged. **Note that in the provided foreground image, the foreground object's length and width should fully extend to the edges of the image (see our example), otherwise the performance would be severely affected.**
-- **MureObjectStitchModel** [[MureObjectStitch]](https://github.com/bcmi/MureObjectStitch-Image-Composition) is another generative image composition model which can adjust the pose and view of foreground. It supports multiple reference images of one foreground object. If you have a few images containing the foreground object, we suggest finetuning MureObjectStitch using these images for better detail preservation. **Note that in the provided foreground image, the foreground object's length and width should fully extend to the edges of the image (see our example), otherwise the performance would be severely affected.**
+- **ShadowGenerationModel** [[GPSDiffusion]](https://github.com/bcmi/GPSDiffusion-Object-Shadow-Generation) generates plausible shadow for the inserted object in a composite image. 
+- **ReflectionGenerationModel** generates plausible reflection for the inserted object in a composite image. 
+- **KontextBlendingHarmonizationModel** [[FluxKontext]](https://github.com/black-forest-labs/flux) is a generative image composition model which inserts foreground into the specified bounding box in the background. The pose and view of foreground stay unchanged. The "blending" mode does not adjust the foreground illumination, while the "harmonization" mode adjusts the foreground illumination to make the composite image harmonious. 
+- **InsertAnythingModel** [[InsertAnything]](https://github.com/song-wensong/insert-anything) is a generative image composition model which inserts foreground into the specified bounding box in the background. The model has reasonable ability to adjust the pose and view of foreground according to the background. 
 
 
 **For the detailed method descriptions, code examples, visualization results, and performance comments, please refer to our [[documents]](https://libcom.readthedocs.io/en/latest/).** If the model performance is not satisfactory, you can finetune the pretrained model on your own dataset using the source repository and replace the original model. 
 
 ## Requirements
 
-The main branch is built on the Linux system with **Python 3.8** and **PyTorch>=1.10.1**. For other dependencies, please refer to [[conda_env]](https://github.com/bcmi/libcom/blob/main/requirements/libcom.yaml) and [[runtime_dependencies]](https://github.com/bcmi/libcom/blob/main/requirements/runtime.txt).
+The main branch is built on the Linux system with **Python 3.10** and **PyTorch>=2.6**. For other dependencies, please refer to [[requirements]](https://github.com/bcmi/libcom/blob/main/requirements.txt).
 
 ## Get Started
-Please refer to [[Installation]](https://github.com/bcmi/libcom/blob/test/docs/get_started.md) for installation instructions and [[documents]](https://libcom.readthedocs.io/en/latest/) for user guidance.
+Please refer to [[Installation]](https://github.com/bcmi/libcom/blob/main/docs/get_started.md) for installation instructions and [[documents]](https://libcom.readthedocs.io/en/latest/) for user guidance.
 
 ## Contributors
 - Institution: [Brain-like Computing and Machine Intelligence (BCMI) Lab](https://bcmi.sjtu.edu.cn/).
@@ -80,3 +81,4 @@ If you use our toolbox, please cite our survey paper using the following BibTeX
   year={2021}
 }
 ```
+

{libcom-0.1.2 → libcom-0.1.3}/libcom/insert_anything/insert_anything.py

@@ -38,31 +38,29 @@ class InsertAnythingModel:
         >>> from libcom import InsertAnythingModel
         >>> from libcom.utils.process_image import make_image_grid, draw_bbox_on_image
         >>> import cv2
-        >>> import os
-        >>> net    = InsertAnythingModel(device=0)
-        >>> sample_list = ['000000000003', '000000000004']
-        >>> sample_dir  = './tests/insertanything/'
-        >>> bbox_list   = [[623, 1297, 1159, 1564], [363, 205, 476, 276]]
-        >>> for i, sample in enumerate(sample_list):
-        >>>     bg_img = sample_dir + f'background/{sample}.jpg'
-        >>>     fg_img_path = sample_dir + f'foreground/{sample}/'
-        >>>     fg_mask_path = sample_dir + f'foreground_mask/{sample}/'
-        >>>     fg_img_list = [os.path.join(fg_img_path, f) for f in os.listdir(fg_img_path)]
-        >>>     fg_mask_list = [os.path.join(fg_mask_path, f) for f in os.listdir(fg_mask_path)]
-        >>>     bbox   = bbox_list[i]
-        >>>     comp, show_fg_img = net(bg_img, fg_img_list, fg_mask_list, bbox, sample_steps=25, num_samples=3)
-        >>>     bg_img   = draw_bbox_on_image(bg_img, bbox)
-        >>>     grid_img = make_image_grid([bg_img, show_fg_img] + [comp[i] for i in range(len(comp))])
+
+        >>> net = InsertAnythingModel(device=0)
+        >>> img_names = ["000000049931.png", "000000460450.png", "6c5601278dcb5e6d_m09728_f5cd2891_17.png"]
+        >>> bboxes = [[168, 137, 488, 413], [134, 158, 399, 511], [130, 91, 392, 271]]
+        >>> test_dir  = 'tests/controllable_composition/'
+
+        >>> for i in range(len(img_names)):
+        >>>     bg_img  = test_dir + 'background/' + img_names[i]
+        >>>     fg_img  = test_dir + 'foreground/' + img_names[i]
+        >>>     bbox    = bboxes[i]
+        >>>     mask    = test_dir + 'foreground_mask/' + img_names[i]
+        >>>     comp    = net(bg_img, fg_img, mask, bbox, num_samples=2)
+        >>>     bg_img  = draw_bbox_on_image(bg_img, bbox)
+        >>>     grid_img = make_image_grid([bg_img, fg_img, comp[0]])
         >>>     cv2.imwrite(f'../docs/_static/image/insertanything_result{i+1}.jpg', grid_img)
 
     Expected result:
 
-    .. image:: _static/image/insertanything_result1.jpg
+    .. image:: _static/image/insert_anything_result1.jpg
         :scale: 21 %
-    .. image:: _static/image/insertanything_result2.jpg
+    .. image:: _static/image/insert_anything_result2.jpg
         :scale: 21 %
 
-
     """
     def __init__(self, device=0, model_type='insertanything', **kwargs):
         assert model_type in model_set, f'Not implementation for {model_type}'

libcom-0.1.3/libcom/insert_anything/source/utils.py

@@ -0,0 +1,144 @@
+import numpy as np
+import cv2
+import math
+
+def f(r, T=0.6, beta=0.1):
+    return np.where(r < T, beta + (1 - beta) / T * r, 1)
+
+# Get the bounding box of the mask
+def get_bbox_from_mask(mask):
+    h,w = mask.shape[0],mask.shape[1]
+
+    if mask.sum() < 10:
+        return 0,h,0,w
+    rows = np.any(mask,axis=1)
+    cols = np.any(mask,axis=0)
+    y1,y2 = np.where(rows)[0][[0,-1]]
+    x1,x2 = np.where(cols)[0][[0,-1]]
+    return (y1,y2,x1,x2)
+
+# Expand the bounding box
+def expand_bbox(mask, yyxx, ratio, min_crop=0):
+    y1,y2,x1,x2 = yyxx
+    H,W = mask.shape[0], mask.shape[1]
+
+    yyxx_area = (y2-y1+1) * (x2-x1+1)
+    r1 = yyxx_area / (H * W)
+    r2 = f(r1)
+    ratio = math.sqrt(r2 / r1)
+
+    xc, yc = 0.5 * (x1 + x2), 0.5 * (y1 + y2)
+    h = ratio * (y2-y1+1)
+    w = ratio * (x2-x1+1)
+    h = max(h,min_crop)
+    w = max(w,min_crop)
+
+    x1 = int(xc - w * 0.5)
+    x2 = int(xc + w * 0.5)
+    y1 = int(yc - h * 0.5)
+    y2 = int(yc + h * 0.5)
+
+    x1 = max(0,x1)
+    x2 = min(W,x2)
+    y1 = max(0,y1)
+    y2 = min(H,y2)
+    return (y1,y2,x1,x2)
+
+# Pad the image to a square shape
+def pad_to_square(image, pad_value = 255, random = False):
+    H,W = image.shape[0], image.shape[1]
+    if H == W:
+        return image
+
+    padd = abs(H - W)
+    if random:
+        padd_1 = int(np.random.randint(0,padd))
+    else:
+        padd_1 = int(padd / 2)
+    padd_2 = padd - padd_1
+
+    if len(image.shape) == 2: 
+        if H > W:
+            pad_param = ((0, 0), (padd_1, padd_2))
+        else:
+            pad_param = ((padd_1, padd_2), (0, 0))
+    elif len(image.shape) == 3: 
+        if H > W:
+            pad_param = ((0, 0), (padd_1, padd_2), (0, 0))
+        else:
+            pad_param = ((padd_1, padd_2), (0, 0), (0, 0))
+
+    image = np.pad(image, pad_param, 'constant', constant_values=pad_value)
+
+    return image
+
+# Expand the image and mask
+def expand_image_mask(image, mask, ratio=1.4):
+    h,w = image.shape[0], image.shape[1]
+    H,W = int(h * ratio), int(w * ratio) 
+    h1 = int((H - h) // 2)
+    h2 = H - h - h1
+    w1 = int((W -w) // 2)
+    w2 = W -w - w1
+
+    pad_param_image = ((h1,h2),(w1,w2),(0,0))
+    pad_param_mask = ((h1,h2),(w1,w2))
+    image = np.pad(image, pad_param_image, 'constant', constant_values=255)
+    mask = np.pad(mask, pad_param_mask, 'constant', constant_values=0)
+    return image, mask
+
+# Convert the bounding box to a square shape
+def box2squre(image, box):
+    H,W = image.shape[0], image.shape[1]
+    y1,y2,x1,x2 = box
+    cx = (x1 + x2) // 2
+    cy = (y1 + y2) // 2
+    h,w = y2-y1, x2-x1
+
+    if h >= w:
+        x1 = cx - h//2
+        x2 = cx + h//2
+    else:
+        y1 = cy - w//2
+        y2 = cy + w//2
+    x1 = max(0,x1)
+    x2 = min(W,x2)
+    y1 = max(0,y1)
+    y2 = min(H,y2)
+    return (y1,y2,x1,x2)
+
+# Crop the predicted image back to the original image
+def crop_back( pred, tar_image,  extra_sizes, tar_box_yyxx_crop):
+    H1, W1, H2, W2 = extra_sizes
+    y1,y2,x1,x2 = tar_box_yyxx_crop    
+    pred = cv2.resize(pred, (W2, H2))
+    m = 2 # maigin_pixel
+
+    if W1 == H1:
+        if m != 0:
+            tar_image[y1+m :y2-m, x1+m:x2-m, :] =  pred[m:-m, m:-m]
+        else:
+            tar_image[y1 :y2, x1:x2, :] =  pred[:, :]
+        return tar_image
+
+    if W1 < W2:
+        pad1 = int((W2 - W1) / 2)
+        pad2 = W2 - W1 - pad1
+        pred = pred[:,pad1: -pad2, :]
+    else:
+        pad1 = int((H2 - H1) / 2)
+        pad2 = H2 - H1 - pad1
+        pred = pred[pad1: -pad2, :, :]
+
+    gen_image = tar_image.copy()
+    if m != 0:
+        gen_image[y1+m :y2-m, x1+m:x2-m, :] =  pred[m:-m, m:-m]
+    else:
+        gen_image[y1 :y2, x1:x2, :] =  pred[:, :]
+    
+    return gen_image
+
+def bbox2mask(bbox, mask_w, mask_h):
+    mask = np.zeros((mask_h, mask_w), dtype=np.uint8)
+    mask[bbox[1]: bbox[3], bbox[0]: bbox[2]] = 1
+    return mask

{libcom-0.1.2 → libcom-0.1.3}/libcom/kontext_blending_harmonization/kontext_blending_harmonization.py

@@ -25,16 +25,17 @@ class KontextBlendingHarmonizationModel:
         >>> from libcom import KontextBlendingHarmonizationModel
         >>> from libcom.utils.process_image import make_image_grid, draw_bbox_on_image
         >>> import cv2
-        >>> img_names = ['6c5601278dcb5e6d_m09728_f5cd2891_17.png', '000000460450.png']
-        >>> bboxes    = [[130, 91, 392, 271], [134, 158, 399, 511]] # x1,y1,x2,y2
-        >>> test_dir  = '../tests/controllable_composition/'
-        >>> net     = KontextBlendingHarmonizationModel(device=0, model_type="Kontext_blend")
+
+        >>> net = KontextBlendingHarmonizationModel(device=0, model_type="Kontext_blend")
+        >>> img_names = ["000000049931.png", "000000460450.png", "6c5601278dcb5e6d_m09728_f5cd2891_17.png"]
+        >>> bboxes = [[168, 137, 488, 413], [134, 158, 399, 511], [130, 91, 392, 271]]
+        >>> test_dir  = 'tests/controllable_composition/'
+
         >>> for i in range(len(img_names)):
         >>>     bg_img  = test_dir + 'background/' + img_names[i]
         >>>     fg_img  = test_dir + 'foreground/' + img_names[i]
         >>>     bbox    = bboxes[i]
         >>>     mask    = test_dir + 'foreground_mask/' + img_names[i]
-
         >>>     comp    = net(bg_img, fg_img, bbox, mask)
         >>>     bg_img  = draw_bbox_on_image(bg_img, bbox)
         >>>     grid_img = make_image_grid([bg_img, fg_img, comp[0]])
@@ -43,10 +44,10 @@ class KontextBlendingHarmonizationModel:
     Expected result:
 
     .. image:: _static/image/kontext_result1.jpg
-        :scale: 38 %
+        :scale: 21 %
     .. image:: _static/image/kontext_result2.jpg
-        :scale: 38 %
-            
+        :scale: 21 %
+ 
     """
 
     def __init__(self, device=0, model_type='Kontext_blend', **kwargs):