libcom 0.0.1.post10__tar.gz → 0.0.2.post1__tar.gz

{libcom-0.0.1.post10 → libcom-0.0.2.post1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: libcom
-Version: 0.0.1.post10
+Version: 0.0.2.post1
 Summary: Image Composition Toolbox
 Home-page: https://github.com/bcmi/libcom
 Author: BCMI Lab
@@ -36,7 +36,7 @@ License-File: LICENSE
 
 
 ## Introduction
-**_libcom_ (the library of image composition) is an image composition toolbox.** The goal of image composition is inserting one foreground into a background image to get a realistic composite image, by addressing the inconsistencies (appearance, geometry, and semantic inconsistency) between foreground and background. Generally speaking, image composition could be used to combine the visual elements from different images.
+**_libcom_ (the library of image composition) is an image composition toolbox.** The goal of image composition (object insertion) is inserting one foreground into a background image to get a realistic composite image, by addressing the inconsistencies (appearance, geometry, and semantic inconsistency) between foreground and background. Generally speaking, image composition could be used to combine the visual elements from different images.
 <div align="center">
 </br>
 <img src="https://raw.githubusercontent.com/bcmi/libcom/main/resources/image_composition_task.gif" width="600" />
@@ -57,7 +57,8 @@ License-File: LICENSE
 - **HarmonyScoreModel** evaluates the harmony level between foreground and background in a composite image.
 - **InharmoniousLocalizationModel** localizes the inharmonious region in a synthetic image.
 - **FOSScoreModel** evaluates the compatibility between foreground and background in a composite image in terms of geometry and semantics.
-- **ControlComModel** is a generative image composition model, which unifies image blending and image harmonization in one diffusion model. 
+- **ControlComModel** is a generative image composition model, which unifies image blending, image harmonization, view synthesis, and generative composition within a diffusion model. 
+- **ObjectStitchModel** is another generative image composition model that aims to generate a composite image from a pair of background and foreground, with non-object pixels filled with black.
 - **ShadowGenerationModel** generates plausible shadow for the inserted object in a composite image. 
 
 **For the detailed method descriptions, code examples, visualization results, and performance comments, please refer to our [[documents]](https://libcom.readthedocs.io/en/latest/).**

{libcom-0.0.1.post10 → libcom-0.0.2.post1}/README.md

@@ -17,7 +17,7 @@
 
 
 ## Introduction
-**_libcom_ (the library of image composition) is an image composition toolbox.** The goal of image composition is inserting one foreground into a background image to get a realistic composite image, by addressing the inconsistencies (appearance, geometry, and semantic inconsistency) between foreground and background. Generally speaking, image composition could be used to combine the visual elements from different images.
+**_libcom_ (the library of image composition) is an image composition toolbox.** The goal of image composition (object insertion) is inserting one foreground into a background image to get a realistic composite image, by addressing the inconsistencies (appearance, geometry, and semantic inconsistency) between foreground and background. Generally speaking, image composition could be used to combine the visual elements from different images.
 <div align="center">
 </br>
 <img src="https://raw.githubusercontent.com/bcmi/libcom/main/resources/image_composition_task.gif" width="600" />
@@ -38,7 +38,8 @@
 - **HarmonyScoreModel** evaluates the harmony level between foreground and background in a composite image.
 - **InharmoniousLocalizationModel** localizes the inharmonious region in a synthetic image.
 - **FOSScoreModel** evaluates the compatibility between foreground and background in a composite image in terms of geometry and semantics.
-- **ControlComModel** is a generative image composition model, which unifies image blending and image harmonization in one diffusion model. 
+- **ControlComModel** is a generative image composition model, which unifies image blending, image harmonization, view synthesis, and generative composition within a diffusion model. 
+- **ObjectStitchModel** is another generative image composition model that aims to generate a composite image from a pair of background and foreground, with non-object pixels filled with black.
 - **ShadowGenerationModel** generates plausible shadow for the inserted object in a composite image. 
 
 **For the detailed method descriptions, code examples, visualization results, and performance comments, please refer to our [[documents]](https://libcom.readthedocs.io/en/latest/).**

{libcom-0.0.1.post10 → libcom-0.0.2.post1}/libcom/__init__.py

@@ -9,11 +9,12 @@ from .fopa_heat_map import *
 from .fos_score import *
 from .controllable_composition import *
 from .shadow_generation import *
+from .objectstitch import *
 
 __all__ = [
     'color_transfer', 'get_composite_image', 'OPAScoreModel', 
     'HarmonyScoreModel', 'InharmoniousLocalizationModel',
     'ImageHarmonizationModel', 'PainterlyHarmonizationModel',
     'FOPAHeatMapModel', 'FOSScoreModel', 'ControlComModel',
-    'ShadowGenerationModel'
+    'ShadowGenerationModel', 'ObjectStitchModel'
 ]

{libcom-0.0.1.post10 → libcom-0.0.2.post1}/libcom/controllable_composition/controllable_composition.py

@@ -29,8 +29,9 @@ from .source.ControlCom.ldm.models.diffusion.plms import PLMSSampler
 from .source.ControlCom.ldm.data.open_images_control import get_tensor, get_tensor_clip, get_bbox_tensor, bbox2mask, tensor2numpy
 
 cur_dir   = os.path.dirname(os.path.abspath(__file__))
-model_set = ['ControlCom'] 
-task_set  = ['blending', 'harmonization'] # 'viewsynthesis', 'composition'
+model_dir = os.environ.get('LIBCOM_MODEL_DIR',cur_dir)
+model_set = ['ControlCom', 'ControlCom_blend_harm', 'ControlCom_view_comp'] 
+task_set  = ['blending', 'harmonization', 'viewsynthesis', 'composition']
 
 class ControlComModel:
     """
@@ -38,7 +39,7 @@ class ControlComModel:
 
     Args:
         device (str | torch.device): gpu id
-        model_type (str): predefined model type
+        model_type (str): predefined model type. "ControlCom" refers to the version trained on all four tasks comprehensively, covering a wide range of domains or objectives. "ControlCom_blend_harm" fine-tunes the "full" version specifically for image blending and harmonization tasks. "ControlCom_view_comp" fine-tunes the "full" version to excel in view synthesis and generative composition tasks.
         kwargs (dict): sampler='ddim' (default) or 'plms', other parameters for building model
     
     Examples:
@@ -53,7 +54,7 @@ class ControlComModel:
         >>>     fg_img  = test_dir + 'foreground/' + img_names[i]
         >>>     bbox    = bboxes[i]
         >>>     mask    = test_dir + 'foreground_mask/' + img_names[i]
-        >>>     net     = ControlComModel(device=0)
+        >>>     net     = ControlComModel(device=0, model_type="ControlCom")
         >>>     comp    = net(bg_img, fg_img, bbox, mask, task=['blending', 'harmonization'])
         >>>     bg_img  = draw_bbox_on_image(bg_img, bbox)
         >>>     grid_img = make_image_grid([bg_img, fg_img, comp[0], comp[1]])
@@ -73,7 +74,7 @@ class ControlComModel:
         self.model_type = model_type
         self.option = kwargs
         
-        weight_path = os.path.join(cur_dir, 'pretrained_models', 'ControlCom.pth')
+        weight_path = os.path.join(cur_dir, 'pretrained_models', f'{self.model_type}.pth')
         download_pretrained_model(weight_path)
         
         self.device = check_gpu_device(device)
@@ -84,7 +85,7 @@ class ControlComModel:
         pl_sd  = torch.load(weight_path, map_location="cpu")
         sd     = pl_sd["state_dict"]
         config = OmegaConf.load(os.path.join(cur_dir, 'source/ControlCom/configs/controlcom.yaml'))
-        clip_path = os.path.join(cur_dir, '../shared_pretrained_models', 'openai-clip-vit-large-patch14')
+        clip_path = os.path.join(model_dir, '../shared_pretrained_models', 'openai-clip-vit-large-patch14')
         download_entire_folder(clip_path)
         config.model.params.cond_stage_config.params.version = clip_path
         model  = instantiate_from_config(config.model)
@@ -181,6 +182,10 @@ class ControlComModel:
                 indicator.append([0,0])
             elif t == 'harmonization':
                 indicator.append([1,0])
+            elif t == 'viewsynthesis':
+                indicator.append([0,1])
+            else:
+                indicator.append([1,1])
         return indicator
                 
     @torch.no_grad()

{libcom-0.0.1.post10 → libcom-0.0.2.post1}/libcom/controllable_composition/source/ControlCom/configs/controlcom.yaml

@@ -1,6 +1,6 @@
 model:
   base_learning_rate: 1.0e-05
-  target: ldm.models.diffusion.ddpm.LatentDiffusion
+  target: libcom.controllable_composition.source.ControlCom.ldm.models.diffusion.ddpm.LatentDiffusion
   params:
     linear_start: 0.00085
     linear_end: 0.012
@@ -41,7 +41,7 @@ model:
     use_guidance: true
     local_uncond: same
     scheduler_config:
-      target: ldm.lr_scheduler.LambdaLinearScheduler
+      target: libcom.controllable_composition.source.ControlCom.ldm.lr_scheduler.LambdaLinearScheduler
       params:
         warm_up_steps:
         - 1000
@@ -54,7 +54,7 @@ model:
         f_min:
         - 1.0
     unet_config:
-      target: ldm.modules.diffusionmodules.openaimodel.UNetModel
+      target: libcom.controllable_composition.source.ControlCom.ldm.modules.diffusionmodules.openaimodel.UNetModel
       params:
         image_size: 32
         in_channels: 11
@@ -78,7 +78,7 @@ model:
         legacy: false
         add_conv_in_front_of_unet: false
         local_encoder_config:
-          conditioning_key: ldm.modules.local_module.LocalRefineBlock
+          conditioning_key: libcom.controllable_composition.source.ControlCom.ldm.modules.local_module.LocalRefineBlock
           add_position_emb: false
           roi_size: 16
           context_dim: 1024
@@ -89,7 +89,7 @@ model:
           add_in_decoder: true
           add_before_crossattn: false
     first_stage_config:
-      target: ldm.models.autoencoder.AutoencoderKL
+      target: libcom.controllable_composition.source.ControlCom.ldm.models.autoencoder.AutoencoderKL
       params:
         embed_dim: 4
         monitor: val/rec_loss
@@ -111,7 +111,7 @@ model:
         lossconfig:
           target: torch.nn.Identity
     cond_stage_config:
-      target: ldm.modules.encoders.modules.FrozenCLIPImageEmbedder
+      target: libcom.controllable_composition.source.ControlCom.ldm.modules.encoders.modules.FrozenCLIPImageEmbedder
       params:
         version: openai-clip-vit-large-patch14
         local_hidden_index: 12

libcom-0.0.2.post1/libcom/controllable_composition/source/ControlCom/ldm/data/datamodule.py

@@ -0,0 +1,128 @@
+import argparse, os, sys, datetime, glob, importlib, csv
+import numpy as np
+import time
+import torch
+import torchvision
+import pytorch_lightning as pl
+
+from packaging import version
+from omegaconf import OmegaConf
+from torch.utils.data import random_split, DataLoader, Dataset, Subset
+from functools import partial
+from PIL import Image
+
+from pytorch_lightning import seed_everything
+from pytorch_lightning.trainer import Trainer
+from pytorch_lightning.callbacks import ModelCheckpoint, Callback, LearningRateMonitor
+from pytorch_lightning.utilities.rank_zero import rank_zero_only
+from pytorch_lightning.utilities import rank_zero_info
+
+from libcom.controllable_composition.source.ControlCom.ldm.data.base import Txt2ImgIterableBaseDataset
+from libcom.controllable_composition.source.ControlCom.ldm.util import instantiate_from_config
+import socket
+from pytorch_lightning.plugins.environments import ClusterEnvironment,SLURMEnvironment
+
+class WrappedDataset(Dataset):
+    """Wraps an arbitrary object with __len__ and __getitem__ into a pytorch dataset"""
+
+    def __init__(self, dataset):
+        self.data = dataset
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        return self.data[idx]
+    
+def worker_init_fn(_):
+    worker_info = torch.utils.data.get_worker_info()
+
+    dataset = worker_info.dataset
+    worker_id = worker_info.id
+
+    if isinstance(dataset, Txt2ImgIterableBaseDataset):
+        split_size = dataset.num_records // worker_info.num_workers
+        # reset num_records to the true number to retain reliable length information
+        dataset.sample_ids = dataset.valid_ids[worker_id * split_size:(worker_id + 1) * split_size]
+        current_id = np.random.choice(len(np.random.get_state()[1]), 1)
+        return np.random.seed(np.random.get_state()[1][current_id] + worker_id)
+    else:
+        return np.random.seed(np.random.get_state()[1][0] + worker_id)
+
+class DataModuleFromConfig(pl.LightningDataModule):
+    def __init__(self, batch_size, num_workers=None, train=None, validation=None, test=None, predict=None,
+                 wrap=False, shuffle_test_loader=False, use_worker_init_fn=False,
+                 shuffle_val_dataloader=False):
+        super().__init__()
+        self.batch_size = batch_size
+        self.dataset_configs = dict()
+        self.num_workers = num_workers if num_workers is not None else batch_size
+        self.use_worker_init_fn = use_worker_init_fn
+        if train is not None:
+            self.dataset_configs["train"] = train
+            self.train_dataloader = self._train_dataloader
+        if validation is not None:
+            self.dataset_configs["validation"] = validation
+            self.val_dataloader = partial(self._val_dataloader, shuffle=shuffle_val_dataloader)
+        if test is not None:
+            self.dataset_configs["test"] = test
+            self.test_dataloader = partial(self._test_dataloader, shuffle=shuffle_test_loader)
+        if predict is not None:
+            self.dataset_configs["predict"] = predict
+            self.predict_dataloader = self._predict_dataloader
+        self.wrap = wrap
+
+    def prepare_data(self):
+        for data_cfg in self.dataset_configs.values():
+            instantiate_from_config(data_cfg)
+
+    def setup(self, stage=None):
+        self.datasets = dict(
+            (k, instantiate_from_config(self.dataset_configs[k]))
+            for k in self.dataset_configs)
+        if self.wrap:
+            for k in self.datasets:
+                self.datasets[k] = WrappedDataset(self.datasets[k])
+
+    def _train_dataloader(self):
+        is_iterable_dataset = isinstance(self.datasets['train'], Txt2ImgIterableBaseDataset)
+        if is_iterable_dataset or self.use_worker_init_fn:
+            init_fn = worker_init_fn
+        else:
+            init_fn = None
+        return DataLoader(self.datasets["train"], batch_size=self.batch_size,
+                          num_workers=self.num_workers, shuffle=False if is_iterable_dataset else True,
+                          worker_init_fn=init_fn)
+
+    def _val_dataloader(self, shuffle=False):
+        if isinstance(self.datasets['validation'], Txt2ImgIterableBaseDataset) or self.use_worker_init_fn:
+            init_fn = worker_init_fn
+        else:
+            init_fn = None
+        return DataLoader(self.datasets["validation"],
+                          batch_size=self.batch_size,
+                          num_workers=self.num_workers,
+                          worker_init_fn=init_fn,
+                          shuffle=shuffle)
+
+    def _test_dataloader(self, shuffle=False):
+        is_iterable_dataset = isinstance(self.datasets['train'], Txt2ImgIterableBaseDataset)
+        if is_iterable_dataset or self.use_worker_init_fn:
+            init_fn = worker_init_fn
+        else:
+            init_fn = None
+
+        # do not shuffle dataloader for iterable dataset
+        shuffle = shuffle and (not is_iterable_dataset)
+
+        return DataLoader(self.datasets["test"], batch_size=self.batch_size,
+                          num_workers=self.num_workers, worker_init_fn=init_fn, shuffle=shuffle)
+
+    def _predict_dataloader(self, shuffle=False):
+        if isinstance(self.datasets['predict'], Txt2ImgIterableBaseDataset) or self.use_worker_init_fn:
+            init_fn = worker_init_fn
+        else:
+            init_fn = None
+        return DataLoader(self.datasets["predict"], batch_size=self.batch_size,
+                          num_workers=self.num_workers, worker_init_fn=init_fn)
+    

libcom-0.0.2.post1/libcom/controllable_composition/source/ControlCom/ldm/models/diffusion/classifier.py

@@ -0,0 +1,267 @@
+import os
+import torch
+import pytorch_lightning as pl
+from omegaconf import OmegaConf
+from torch.nn import functional as F
+from torch.optim import AdamW
+from torch.optim.lr_scheduler import LambdaLR
+from copy import deepcopy
+from einops import rearrange
+from glob import glob
+from natsort import natsorted
+
+from libcom.controllable_composition.source.ControlCom.ldm.modules.diffusionmodules.openaimodel import EncoderUNetModel, UNetModel
+from libcom.controllable_composition.source.ControlCom.ldm.util import log_txt_as_img, default, ismap, instantiate_from_config
+
+__models__ = {
+    'class_label': EncoderUNetModel,
+    'segmentation': UNetModel
+}
+
+
+def disabled_train(self, mode=True):
+    """Overwrite model.train with this function to make sure train/eval mode
+    does not change anymore."""
+    return self
+
+
+class NoisyLatentImageClassifier(pl.LightningModule):
+
+    def __init__(self,
+                 diffusion_path,
+                 num_classes,
+                 ckpt_path=None,
+                 pool='attention',
+                 label_key=None,
+                 diffusion_ckpt_path=None,
+                 scheduler_config=None,
+                 weight_decay=1.e-2,
+                 log_steps=10,
+                 monitor='val/loss',
+                 *args,
+                 **kwargs):
+        super().__init__(*args, **kwargs)
+        self.num_classes = num_classes
+        # get latest config of diffusion model
+        diffusion_config = natsorted(glob(os.path.join(diffusion_path, 'configs', '*-project.yaml')))[-1]
+        self.diffusion_config = OmegaConf.load(diffusion_config).model
+        self.diffusion_config.params.ckpt_path = diffusion_ckpt_path
+        self.load_diffusion()
+
+        self.monitor = monitor
+        self.numd = self.diffusion_model.first_stage_model.encoder.num_resolutions - 1
+        self.log_time_interval = self.diffusion_model.num_timesteps // log_steps
+        self.log_steps = log_steps
+
+        self.label_key = label_key if not hasattr(self.diffusion_model, 'cond_stage_key') \
+            else self.diffusion_model.cond_stage_key
+
+        assert self.label_key is not None, 'label_key neither in diffusion model nor in model.params'
+
+        if self.label_key not in __models__:
+            raise NotImplementedError()
+
+        self.load_classifier(ckpt_path, pool)
+
+        self.scheduler_config = scheduler_config
+        self.use_scheduler = self.scheduler_config is not None
+        self.weight_decay = weight_decay
+
+    def init_from_ckpt(self, path, ignore_keys=list(), only_model=False):
+        sd = torch.load(path, map_location="cpu")
+        if "state_dict" in list(sd.keys()):
+            sd = sd["state_dict"]
+        keys = list(sd.keys())
+        for k in keys:
+            for ik in ignore_keys:
+                if k.startswith(ik):
+                    print("Deleting key {} from state_dict.".format(k))
+                    del sd[k]
+        missing, unexpected = self.load_state_dict(sd, strict=False) if not only_model else self.model.load_state_dict(
+            sd, strict=False)
+        print(f"Restored from {path} with {len(missing)} missing and {len(unexpected)} unexpected keys")
+        if len(missing) > 0:
+            print(f"Missing Keys: {missing}")
+        if len(unexpected) > 0:
+            print(f"Unexpected Keys: {unexpected}")
+
+    def load_diffusion(self):
+        model = instantiate_from_config(self.diffusion_config)
+        self.diffusion_model = model.eval()
+        self.diffusion_model.train = disabled_train
+        for param in self.diffusion_model.parameters():
+            param.requires_grad = False
+
+    def load_classifier(self, ckpt_path, pool):
+        model_config = deepcopy(self.diffusion_config.params.unet_config.params)
+        model_config.in_channels = self.diffusion_config.params.unet_config.params.out_channels
+        model_config.out_channels = self.num_classes
+        if self.label_key == 'class_label':
+            model_config.pool = pool
+
+        self.model = __models__[self.label_key](**model_config)
+        if ckpt_path is not None:
+            print('#####################################################################')
+            print(f'load from ckpt "{ckpt_path}"')
+            print('#####################################################################')
+            self.init_from_ckpt(ckpt_path)
+
+    @torch.no_grad()
+    def get_x_noisy(self, x, t, noise=None):
+        noise = default(noise, lambda: torch.randn_like(x))
+        continuous_sqrt_alpha_cumprod = None
+        if self.diffusion_model.use_continuous_noise:
+            continuous_sqrt_alpha_cumprod = self.diffusion_model.sample_continuous_noise_level(x.shape[0], t + 1)
+            # todo: make sure t+1 is correct here
+
+        return self.diffusion_model.q_sample(x_start=x, t=t, noise=noise,
+                                             continuous_sqrt_alpha_cumprod=continuous_sqrt_alpha_cumprod)
+
+    def forward(self, x_noisy, t, *args, **kwargs):
+        return self.model(x_noisy, t)
+
+    @torch.no_grad()
+    def get_input(self, batch, k):
+        x = batch[k]
+        if len(x.shape) == 3:
+            x = x[..., None]
+        x = rearrange(x, 'b h w c -> b c h w')
+        x = x.to(memory_format=torch.contiguous_format).float()
+        return x
+
+    @torch.no_grad()
+    def get_conditioning(self, batch, k=None):
+        if k is None:
+            k = self.label_key
+        assert k is not None, 'Needs to provide label key'
+
+        targets = batch[k].to(self.device)
+
+        if self.label_key == 'segmentation':
+            targets = rearrange(targets, 'b h w c -> b c h w')
+            for down in range(self.numd):
+                h, w = targets.shape[-2:]
+                targets = F.interpolate(targets, size=(h // 2, w // 2), mode='nearest')
+
+            # targets = rearrange(targets,'b c h w -> b h w c')
+
+        return targets
+
+    def compute_top_k(self, logits, labels, k, reduction="mean"):
+        _, top_ks = torch.topk(logits, k, dim=1)
+        if reduction == "mean":
+            return (top_ks == labels[:, None]).float().sum(dim=-1).mean().item()
+        elif reduction == "none":
+            return (top_ks == labels[:, None]).float().sum(dim=-1)
+
+    def on_train_epoch_start(self):
+        # save some memory
+        self.diffusion_model.model.to('cpu')
+
+    @torch.no_grad()
+    def write_logs(self, loss, logits, targets):
+        log_prefix = 'train' if self.training else 'val'
+        log = {}
+        log[f"{log_prefix}/loss"] = loss.mean()
+        log[f"{log_prefix}/acc@1"] = self.compute_top_k(
+            logits, targets, k=1, reduction="mean"
+        )
+        log[f"{log_prefix}/acc@5"] = self.compute_top_k(
+            logits, targets, k=5, reduction="mean"
+        )
+
+        self.log_dict(log, prog_bar=False, logger=True, on_step=self.training, on_epoch=True)
+        self.log('loss', log[f"{log_prefix}/loss"], prog_bar=True, logger=False)
+        self.log('global_step', self.global_step, logger=False, on_epoch=False, prog_bar=True)
+        lr = self.optimizers().param_groups[0]['lr']
+        self.log('lr_abs', lr, on_step=True, logger=True, on_epoch=False, prog_bar=True)
+
+    def shared_step(self, batch, t=None):
+        x, *_ = self.diffusion_model.get_input(batch, k=self.diffusion_model.first_stage_key)
+        targets = self.get_conditioning(batch)
+        if targets.dim() == 4:
+            targets = targets.argmax(dim=1)
+        if t is None:
+            t = torch.randint(0, self.diffusion_model.num_timesteps, (x.shape[0],), device=self.device).long()
+        else:
+            t = torch.full(size=(x.shape[0],), fill_value=t, device=self.device).long()
+        x_noisy = self.get_x_noisy(x, t)
+        logits = self(x_noisy, t)
+
+        loss = F.cross_entropy(logits, targets, reduction='none')
+
+        self.write_logs(loss.detach(), logits.detach(), targets.detach())
+
+        loss = loss.mean()
+        return loss, logits, x_noisy, targets
+
+    def training_step(self, batch, batch_idx):
+        loss, *_ = self.shared_step(batch)
+        return loss
+
+    def reset_noise_accs(self):
+        self.noisy_acc = {t: {'acc@1': [], 'acc@5': []} for t in
+                          range(0, self.diffusion_model.num_timesteps, self.diffusion_model.log_every_t)}
+
+    def on_validation_start(self):
+        self.reset_noise_accs()
+
+    @torch.no_grad()
+    def validation_step(self, batch, batch_idx):
+        loss, *_ = self.shared_step(batch)
+
+        for t in self.noisy_acc:
+            _, logits, _, targets = self.shared_step(batch, t)
+            self.noisy_acc[t]['acc@1'].append(self.compute_top_k(logits, targets, k=1, reduction='mean'))
+            self.noisy_acc[t]['acc@5'].append(self.compute_top_k(logits, targets, k=5, reduction='mean'))
+
+        return loss
+
+    def configure_optimizers(self):
+        optimizer = AdamW(self.model.parameters(), lr=self.learning_rate, weight_decay=self.weight_decay)
+
+        if self.use_scheduler:
+            scheduler = instantiate_from_config(self.scheduler_config)
+
+            print("Setting up LambdaLR scheduler...")
+            scheduler = [
+                {
+                    'scheduler': LambdaLR(optimizer, lr_lambda=scheduler.schedule),
+                    'interval': 'step',
+                    'frequency': 1
+                }]
+            return [optimizer], scheduler
+
+        return optimizer
+
+    @torch.no_grad()
+    def log_images(self, batch, N=8, *args, **kwargs):
+        log = dict()
+        x = self.get_input(batch, self.diffusion_model.first_stage_key)
+        log['inputs'] = x
+
+        y = self.get_conditioning(batch)
+
+        if self.label_key == 'class_label':
+            y = log_txt_as_img((x.shape[2], x.shape[3]), batch["human_label"])
+            log['labels'] = y
+
+        if ismap(y):
+            log['labels'] = self.diffusion_model.to_rgb(y)
+
+            for step in range(self.log_steps):
+                current_time = step * self.log_time_interval
+
+                _, logits, x_noisy, _ = self.shared_step(batch, t=current_time)
+
+                log[f'inputs@t{current_time}'] = x_noisy
+
+                pred = F.one_hot(logits.argmax(dim=1), num_classes=self.num_classes)
+                pred = rearrange(pred, 'b h w c -> b c h w')
+
+                log[f'pred@t{current_time}'] = self.diffusion_model.to_rgb(pred)
+
+        for key in log:
+            log[key] = log[key][:N]
+
+        return log

{libcom-0.0.1.post10 → libcom-0.0.2.post1}/libcom/controllable_composition/source/ControlCom/ldm/models/diffusion/ddpm.py

@@ -80,7 +80,6 @@ class DDPM(pl.LightningModule):
         super().__init__()
         assert parameterization in ["eps", "x0"], 'currently only supporting "eps" and "x0"'
         self.parameterization = parameterization
-        # print(f"{self.__class__.__name__}: Running in {self.parameterization}-prediction mode")
         self.cond_stage_model = None
         self.clip_denoised = clip_denoised
         self.log_every_t = log_every_t
@@ -1636,7 +1635,7 @@ def log_local(save_dir,
 
 if __name__ == '__main__':
     from omegaconf import OmegaConf
-    from ldm.util import instantiate_from_config
+    from libcom.controllable_composition.source.ControlCom.ldm.util import instantiate_from_config
     import os, torchvision
     from PIL import Image
     import shutil

{libcom-0.0.1.post10 → libcom-0.0.2.post1}/libcom/controllable_composition/source/ControlCom/ldm/modules/attention.py

@@ -8,7 +8,7 @@ from einops import rearrange, repeat
 import os, sys
 proj_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 sys.path.insert(0, proj_dir)
-from ldm.modules.diffusionmodules.util import checkpoint
+from libcom.controllable_composition.source.ControlCom.ldm.modules.diffusionmodules.util import checkpoint
 from torchvision.ops import roi_align
 
 def exists(val):

{libcom-0.0.1.post10 → libcom-0.0.2.post1}/libcom/controllable_composition/source/ControlCom/ldm/modules/diffusionmodules/openaimodel.py

@@ -11,7 +11,7 @@ import torch.nn.functional as F
 import os, sys
 proj_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
 sys.path.insert(0, proj_dir)
-from ldm.modules.diffusionmodules.util import (
+from libcom.controllable_composition.source.ControlCom.ldm.modules.diffusionmodules.util import (
     checkpoint,
     conv_nd,
     linear,
@@ -1141,7 +1141,7 @@ if __name__ == '__main__':
     import torch
     device = torch.device("cuda:0")
     from omegaconf import OmegaConf
-    from ldm.util import instantiate_from_config
+    from libcom.controllable_composition.source.ControlCom.ldm.util import instantiate_from_config
     cfg_path = os.path.join(proj_dir, 'configs/finetune_paint.yaml')
     configs  = OmegaConf.load(cfg_path).model.params
     model  = instantiate_from_config(configs.unet_config).to(device)

{libcom-0.0.1.post10 → libcom-0.0.2.post1}/libcom/controllable_composition/source/ControlCom/ldm/modules/encoders/modules.py

@@ -1,12 +1,11 @@
 import torch
 import torch.nn as nn
 from functools import partial
-from einops import rearrange, repeat
-from transformers import CLIPTokenizer, CLIPTextModel,CLIPVisionModel,CLIPModel,CLIPVisionModelWithProjection
+from transformers import CLIPVisionModel
 import os,sys
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))))
-from ldm.modules.x_transformer import Encoder, TransformerWrapper  # TODO: can we directly rely on lucidrains code and simply add this as a reuirement? --> test
-from ldm.modules.encoders.xf import LayerNorm, Transformer
+from libcom.controllable_composition.source.ControlCom.ldm.modules.x_transformer import Encoder, TransformerWrapper  # TODO: can we directly rely on lucidrains code and simply add this as a reuirement? --> test
+from libcom.controllable_composition.source.ControlCom.ldm.modules.encoders.xf import LayerNorm, Transformer
 import math
 import torch.nn.functional as F
 
@@ -217,7 +216,7 @@ class FrozenCLIPImageEmbedder(AbstractEncoder):
 
 
 if __name__ == "__main__":
-    from ldm.util import count_params
+    from libcom.controllable_composition.source.ControlCom.ldm.util import count_params
     device = torch.device("cuda:0")
     model = FrozenCLIPImageEmbedder().to(device)
     count_params(model, verbose=True)

{libcom-0.0.1.post10 → libcom-0.0.2.post1}/libcom/controllable_composition/source/ControlCom/ldm/modules/local_module.py

@@ -6,8 +6,8 @@ import torch.nn.functional as F
 from torch import nn, einsum
 from einops import rearrange, repeat
 import os, sys
-from ldm.modules.diffusionmodules.util import checkpoint
-from ldm.modules.attention import CrossAttention, zero_module, Normalize
+from libcom.controllable_composition.source.ControlCom.ldm.modules.diffusionmodules.util import checkpoint
+from libcom.controllable_composition.source.ControlCom.ldm.modules.attention import CrossAttention, zero_module, Normalize
 from torchvision.ops import roi_align
 
 class FDN(nn.Module):

libcom-0.0.2.post1/libcom/controllable_composition/source/ControlCom/ldm/modules/losses/__init__.py

@@ -0,0 +1 @@
+from libcom.controllable_composition.source.ControlCom.ldm.modules.losses.contperceptual import LPIPSWithDiscriminator

{libcom-0.0.1.post10 → libcom-0.0.2.post1}/libcom/controllable_composition/source/ControlCom/ldm/util.py

@@ -90,8 +90,6 @@ def get_obj_from_str(string, reload=False):
     if reload:
         module_imp = importlib.import_module(module)
         importlib.reload(module_imp)
-    if 'ldm' in module:
-        module = '.source.ControlCom.' + module
     return getattr(importlib.import_module(module, package='libcom.controllable_composition'), cls)