linmult 1.1.0__tar.gz

linmult-1.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 fodorad
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

linmult-1.1.0/PKG-INFO

@@ -0,0 +1,7 @@
+Metadata-Version: 2.1
+Name: linmult
+Version: 1.1.0
+Summary: General-purpose Multimodal Transformer with Linear Complexity Attention Mechanism
+Author: fodorad
+License: MIT
+License-File: LICENSE

linmult-1.1.0/README.md

@@ -0,0 +1,143 @@
+# LinMulT
+
+General-purpose Multimodal Transformer with Linear Complexity Attention Mechanism.
+
+# Setup
+
+### Environment
+* Python 3.10+
+* PyTorch and cuDNN 1.13.1+cu117
+
+### Install package with pip+git
+```
+pip install -U git+https://github.com/fodorad/LinMulT.git
+```
+
+### Install package from repository root
+```
+git clone https://github.com/fodorad/LinMulT
+cd LinMulT
+pip install -e .
+pip install -U -r requirements.txt
+```
+
+# Quick start
+### Example 1:
+Simple transformer encoder with linear attention.
+The forward pass is performed using an input sequence.
+```
+import torch
+from linmult import LinT
+
+# input shape: (batch_size, time_dimension, feature_dimension)
+x = torch.rand((32, 15, 1024), device='cuda')
+model = LinT(input_modality_channels=1024, output_dim=5).cuda()
+y_pred_seq = model(x)
+
+# output shape: (batch_size, time_dimension, output_dimension)
+assert y_pred_seq.size() == torch.Size([32, 15, 5])
+```
+
+### Example 2:
+Multimodal Transformer with Linear Attention.
+The forward pass is performed using 2 input sequences. Both input sequences have the same time dimension.
+```
+import torch
+from linmult import LinMulT
+
+# input shape: (batch_size, time_dimension, feature_dimension)
+x_1 = torch.rand((32, 15, 1024), device='cuda')
+x_2 = torch.rand((32, 15, 160), device='cuda')
+model = LinMulT(input_modality_channels=[1024, 160], output_dim=5).cuda()
+y_pred_cls, y_pred_seq = model([x_1, x_2])
+
+# 1. output shape: (batch_size, output_dimension)
+assert y_pred_cls.size() == torch.Size([32, 5])
+
+# 2. output shape: (batch_size, time_dimension, output_dimension)
+assert y_pred_seq.size() == torch.Size([32, 15, 5])
+```
+
+### Example 3:
+Multimodal Transformer with Linear Attention. The forward pass is performed using 3 input sequences with different time dimensions.
+```
+import torch
+from linmult import LinMulT
+
+# input shape: (batch_size, time_dimension, feature_dimension)
+x_1 = torch.rand((16, 1500, 25), device='cuda')
+x_2 = torch.rand((16, 450, 35), device='cuda')
+x_3 = torch.rand((16, 120, 768), device='cuda')
+model = LinMulT(input_modality_channels=[25, 35, 768],
+                output_dim=5,
+                add_time_collapse=True,
+                add_self_attention_fusion=False).cuda()
+y_pred_cls = model([x_1, x_2, x_3])
+
+# output shape: (batch_size, output_dimension)
+assert y_pred_cls.size() == torch.Size([16, 5])
+```
+
+# Run tests
+```
+python -m unittest
+```
+# Similar projects using LinMulT
+
+### (2023) BlinkLinMulT
+LinMulT is trained for blink presence detection and eye state recognition tasks.
+Our results demonstrate comparable or superior performance compared to state-of-the-art models on 2 tasks, using 7 public benchmark databases.
+* paper: BlinkLinMulT: Transformer-based Eye Blink Detection (accepted, available soon)
+* code: https://github.com/fodorad/BlinkLinMulT
+
+### (2022) PersonalityLinMulT
+LinMulT is trained for Big Five personality trait estimation using the First Impressions V2 dataset and sentiment estimation using the MOSI and MOSEI datasets.
+* paper: Multimodal Sentiment and Personality Perception Under Speech: A Comparison of Transformer-based Architectures ([pdf](https://proceedings.mlr.press/v173/fodor22a/fodor22a.pdf), [website](https://proceedings.mlr.press/v173/fodor22a.html))
+* code: https://github.com/fodorad/PersonalityLinMulT
+
+
+# Citation - BibTex
+If you found our research helpful or influential please consider citing:
+
+### (2023) LinMulT for blink presence detection and eye state recognition:
+```
+@article{blinklinmult-fodor23,
+  title = {BlinkLinMulT: Transformer-based Eye Blink Detection},
+  author = {Fodor, {\'A}d{\'a}m and Fenech, Kristian and L{\H{o}}rincz, Andr{\'a}s},
+  journal = {...}
+  pages = {1--19},
+  year = {2023}
+}
+```
+
+### (2022) LinMulT for personality trait and sentiment estimation:
+```
+@InProceedings{pmlr-v173-fodor22a,
+  title = {Multimodal Sentiment and Personality Perception Under Speech: A Comparison of Transformer-based Architectures},
+  author = {Fodor, {\'A}d{\'a}m and Saboundji, Rachid R. and Jacques Junior, Julio C. S. and Escalera, Sergio and Gallardo-Pujol, David and L{\H{o}}rincz, Andr{\'a}s},
+  booktitle = {Understanding Social Behavior in Dyadic and Small Group Interactions},
+  pages = {218--241},
+  year = {2022},
+  editor = {Palmero, Cristina and Jacques Junior, Julio C. S. and Clapés, Albert and Guyon, Isabelle and Tu, Wei-Wei and Moeslund, Thomas B. and Escalera, Sergio},
+  volume = {173},
+  series = {Proceedings of Machine Learning Research},
+  month = {16 Oct},
+  publisher = {PMLR},
+  pdf = {https://proceedings.mlr.press/v173/fodor22a/fodor22a.pdf},
+  url = {https://proceedings.mlr.press/v173/fodor22a.html}
+}
+```
+
+# Acknowledgement
+The code is inspired by the following two materials:
+
+### Multimodal Transformer:
+* paper: Multimodal Transformer for Unaligned Multimodal Language Sequences ([1906.00295](https://arxiv.org/pdf/1906.00295.pdf))
+* code: https://github.com/yaohungt/Multimodal-Transformer
+
+### Linear Attention:
+* paper: Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention ([2006.16236](https://arxiv.org/pdf/2006.16236.pdf))
+* code: https://github.com/idiap/fast-transformers
+
+# Contact
+* Ádám Fodor (foauaai@inf.elte.hu)

linmult-1.1.0/linmult/__init__.py

@@ -0,0 +1,2 @@
+from linmult.models.LinT import LinT
+from linmult.models.LinMulT import LinMulT

linmult-1.1.0/linmult/models/LinMulT.py

@@ -0,0 +1,238 @@
+##########################################################
+#                                                        #
+#   Code is inspired by the following repositories:      #
+#   https://github.com/yaohungt/Multimodal-Transformer   #
+#                                                        #
+##########################################################
+import logging
+from typing import Iterable
+import torch
+from torch import nn
+import torch.nn.functional as F
+from linmult.models.transformer import TransformerEncoder
+
+logging.basicConfig(level=logging.INFO,
+                    format="%(asctime)s %(levelname)s %(message)s",
+                    datefmt="%Y-%m-%d %H:%M:%S")
+
+
+class LinMulT(nn.Module):
+
+    def __init__(self,
+                 input_modality_channels: Iterable[int],
+                 output_dim: int,
+                 projected_modality_dim: int | list = 40, # d
+                 number_of_heads: int = 8,
+                 number_of_layers: int = 4, # D
+                 embedding_dropout: float = 0.1,
+                 cross_attention_dropout: float = 0.1,
+                 self_attention_dropout: float = 0.0,
+                 relu_dropout: float = 0.1,
+                 residual_dropout: float = 0.1,
+                 output_dropout: float = 0.1,
+                 attention_mask: bool = True,
+                 add_time_collapse: bool = False,
+                 add_self_attention_fusion: bool = True,
+                 add_projection_fusion: bool = True,
+                 aggregation: str = 'meanpooling'):
+        """Construct a MulT model with linear attention mechanism.
+
+        Args:
+            aggregation (str | None): aggregation applied to the output sequence to get output_cls.
+                None - when add_time_collapse is True, aggregation is not used at all.
+                last - last timestep is used. Original MulT implementation.
+                cls - classification token is used.
+                meanpooling - mean is calculated over the T time dimension.
+                maxpooling - max is calculated over the T time dimension.
+        """
+        super().__init__()
+
+        if aggregation not in {None, 'last', 'cls', 'meanpooling', 'maxpooling'}:
+            raise Exception(f'Invalid aggregation {aggregation}.')
+
+        if add_time_collapse and add_self_attention_fusion:
+            raise Exception(f'These arguments cannot be True at the same time: {{add_time_collapse, add_self_attention_fusion}}')
+
+        self.input_modality_channels = input_modality_channels
+        self.output_dim = output_dim
+        self.number_of_modalities = len(self.input_modality_channels)
+
+        if isinstance(projected_modality_dim, int):
+            self.projected_modality_dim = [projected_modality_dim] * self.number_of_modalities
+        else: # list
+            if len(projected_modality_dim) != self.number_of_modalities:
+                raise Exception('Length of projected_modality_dim should be the number of modalities.')
+            self.projected_modality_dim = projected_modality_dim
+
+        self.number_of_heads = number_of_heads
+        self.number_of_layers = number_of_layers
+        self.embedding_dropout = embedding_dropout
+        self.cross_attention_dropout = cross_attention_dropout
+        self.self_attention_dropout = self_attention_dropout
+        self.relu_dropout = relu_dropout
+        self.residual_dropout = residual_dropout
+        self.output_dropout = output_dropout
+        self.attention_mask = attention_mask
+        self.add_time_collapse = add_time_collapse
+        self.add_self_attention_fusion = add_self_attention_fusion
+        self.add_projection_fusion = add_projection_fusion
+        self.aggregation = aggregation if not add_time_collapse else None
+        combined_dim = (self.number_of_modalities - 1) * torch.tensor(self.projected_modality_dim).sum()
+
+        # 1. Temporal Convolutional Layers
+        self.projectors = nn.ModuleList([
+            nn.Conv1d(input_modality_channels, projected_modality_dim, kernel_size=1, padding=0, bias=False)
+            for input_modality_channels, projected_modality_dim
+            in zip(self.input_modality_channels, self.projected_modality_dim)
+        ])
+
+        # 2. Crossmodal Attention Transformers
+        # e.g.: a, v, t modalities correspond to 0, 1, 2 indices
+        # Q -> a, K and V -> v, t:  v t - 1 2
+        # Q -> v, K and V -> a, t:  a t - 0 2
+        # Q -> t, K and V -> a, v:  a v - 0 1
+        self.modality_indices = range(self.number_of_modalities)
+        self.crossmodal_transformers = nn.ModuleList([])
+        for target_index in self.modality_indices: # e.g. target_index = 0
+            input_indices = [ind for ind in self.modality_indices if ind != target_index] # e.g. input_indices = [1, 2]
+            self.crossmodal_transformers.append(
+                nn.ModuleList([
+                    self.create_transformer(modality_index=input_index, attention_type='cross')
+                    for input_index in input_indices
+                ])
+            )
+
+        # 3. Self Attention Transformers
+        self.self_attention_transformers = nn.ModuleList([
+            self.create_transformer(modality_index=target_index, attention_type='self', layers=3)
+            for target_index in self.modality_indices
+        ])
+
+        # 4. Self Attention Fusion Transformer
+        if self.add_self_attention_fusion:
+            self.self_attention_fusion_transformer = self.create_fusion_transformer()
+
+        if self.add_projection_fusion:
+            self.projection_1 = nn.Linear(combined_dim, combined_dim)
+            self.projection_2 = nn.Linear(combined_dim, combined_dim)
+
+        # 5. Sequence Head & Aggregation
+        self.out_layer = nn.Linear(combined_dim, self.output_dim) # (B, T, output_dim) or (B, output_dim)
+
+    def create_transformer(self, modality_index, attention_type: str, layers=-1):
+        if attention_type == 'cross': # Crossmodal Attention Transformer
+            embedding_dim = self.projected_modality_dim[modality_index]
+            attention_dropout = self.cross_attention_dropout
+        else: # Self Attention Transformer
+            embedding_dim = (self.number_of_modalities - 1) * self.projected_modality_dim[modality_index]
+            attention_dropout = self.self_attention_dropout
+
+        return TransformerEncoder(embedding_dim=embedding_dim,
+                                  number_of_heads=self.number_of_heads,
+                                  number_of_layers=max(self.number_of_layers, layers),
+                                  attention_dropout=attention_dropout,
+                                  relu_dropout=self.relu_dropout,
+                                  residual_dropout=self.residual_dropout,
+                                  embedding_dropout=self.embedding_dropout,
+                                  attention_mask=self.attention_mask)
+
+    def create_fusion_transformer(self, layers=-1):
+        return TransformerEncoder(embedding_dim=self.number_of_modalities * self.projected_modality_dim[0],
+                                  number_of_heads=self.number_of_heads,
+                                  number_of_layers=max(self.number_of_layers, layers),
+                                  attention_dropout=self.self_attention_dropout,
+                                  relu_dropout=self.relu_dropout,
+                                  residual_dropout=self.residual_dropout,
+                                  embedding_dropout=self.self_attention_dropout,
+                                  attention_mask=self.attention_mask)
+
+    def forward(self, inputs: list[torch.Tensor]) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]:
+        """Inference with Multimodal Transformer.
+
+        Args:
+            inputs (list[torch.Tensor]): input tensors of shape (B, T, F)
+
+        Returns:
+            (torch.Tensor | tuple[torch.Tensor, torch.Tensor]): tensor of shape (B, F) and/or (B, T, F)
+        """
+        # transpose and add embedding dropout
+        inp = []  # x_a, x_v, x_t
+        for input in inputs:
+            input_T = input.transpose(1, 2) # (B, T, F) -> (B, F, T)
+            if self.embedding_dropout > 0:
+                inp.append(F.dropout(input_T, p=self.embedding_dropout, training=self.training))
+            else:
+                inp.append(input_T)
+        logging.debug(f'input sizes: {[tuple(i.size()) for i in inp]}')
+
+        # temporal convolution projection of input tensors
+        proj_x_mod = [self.projectors[i](input).permute(0, 2, 1) for i, input in enumerate(inp)]
+        logging.debug(f'projected input sizes: {[tuple(i.size()) for i in proj_x_mod]}')
+
+        if self.aggregation == 'cls':
+            # add cls token to every input as the first timestamp
+            # (projected_dim,) -> (1, 1, projected_dim) -> (batch_size, 1, projected_dim)
+            cls_tokens = [
+                torch.zeros((proj_x_mod[i].shape[0], 1, proj_x_mod[i].shape[-1]), device=proj_x_mod[i].device)
+                for _ in range(self.number_of_modalities)
+            ]
+
+            proj_x_mod = [
+                torch.cat((cls_token, projected_representation), dim=1)
+                for projected_representation, cls_token in zip(proj_x_mod, cls_tokens)
+            ] # (B, T, F) -> (B, T+1, F)
+
+        # cross-modal transformers
+        hidden_representations = []
+        for target_index in range(self.number_of_modalities): # e.g. target_index == 0
+            input_indices = [ind for ind in self.modality_indices if ind != target_index]  # e.g. input_indices = [1, 2]
+            cross_modal_hidden = []
+            for i, input_index in enumerate(input_indices):
+                # AVT: (V,T) --> A
+                logging.debug(f"Query: {[f'modality_{m}' for m in self.modality_indices][target_index]} with shape {tuple(proj_x_mod[target_index].size())} " + \
+                              f"--> Keys, Values: {[f'modality_{m}' for m in self.modality_indices][input_index]} with shape {tuple(proj_x_mod[input_index].size())}")
+                cross_modal_hidden.append(
+                    self.crossmodal_transformers[target_index][i](
+                        proj_x_mod[target_index], proj_x_mod[input_index], proj_x_mod[input_index])
+                ) # Q, K, V
+            logging.debug(f"num of crossmodal transformers: {len(cross_modal_hidden)}, tensor shapes: {[tuple(elem.size()) for elem in cross_modal_hidden]}")
+
+            # self-attention transformer
+            cross_modal_hidden = torch.cat(cross_modal_hidden, dim=2) # within branch
+            self_hidden = self.self_attention_transformers[target_index](cross_modal_hidden)
+            hidden_representations.append(self_hidden) # (B, T, F) or (B, T+1, F)
+        logging.debug(f"last hidden representations with shapes: {[tuple(elem.size()) for elem in hidden_representations]}")
+
+        if self.add_time_collapse:
+            hidden_representation = torch.cat([hidden_representation[:,-1,:] for hidden_representation in hidden_representations], dim=-1) # [(B, T, F), ...] -> (B, combined_dim)
+        else:
+            hidden_representation = torch.cat(hidden_representations, dim=-1) # [(B, T, F), ...] -> (B, T, combined_dim)
+
+        if self.add_self_attention_fusion:
+            hidden_representation = self.self_attention_fusion_transformer(hidden_representation)
+
+        if self.add_projection_fusion:
+            hidden_representation = self.projection_2(F.dropout(F.relu(self.projection_1(hidden_representation)), p=self.output_dropout, training=self.training)) \
+                + hidden_representation # (B, T, combined_dim) or (B, combined_dim)
+
+        if self.add_time_collapse:
+            output_cls = self.out_layer(hidden_representation)
+            return output_cls
+        else:
+            match self.aggregation:
+                case 'last':
+                    output_cls = self.out_layer(hidden_representation[:, -1, :]) # (B, combined_dim)
+                case 'cls':
+                    output_cls = self.out_layer(hidden_representation[:, 0, :]) # (B, T+1, combined_dim) -> (B, combined_dim)
+                    hidden_representation = hidden_representation[:, 1:, :] # (B, T+1, combined_dim) -> (B, T, combined_dim)
+                case 'maxpooling':
+                    output_cls = self.out_layer(torch.max(hidden_representation, dim=1)) # (B, T, combined_dim) -> (B, combined_dim)
+                case _: # 'meanpooling'
+                    output_cls = self.out_layer(torch.mean(hidden_representation, dim=1)) # (B, T, combined_dim) -> (B, combined_dim)
+
+            # output_cls head: sequence -> aggregation -> dense -> summarized logits
+            # output_seq head: sequence -> time-distributed dense -> sequence-wise logits
+            output_seq = self.out_layer(hidden_representation)
+            logging.debug(f"output output_cls shape: {tuple(output_cls.size())}")
+            logging.debug(f"output output_seq shape: {tuple(output_seq.size())}")
+            return output_cls, output_seq

linmult-1.1.0/linmult/models/LinT.py

@@ -0,0 +1,75 @@
+import torch
+from torch import nn
+import torch.nn.functional as F
+from linmult.models.transformer import TransformerEncoder
+
+
+class LinT(nn.Module):
+
+    def __init__(self,
+                 input_modality_channels: int,
+                 output_dim: int,
+                 projected_modality_dim: int | list = 40, # d
+                 number_of_heads: int = 8,
+                 number_of_layers: int = 4, # D
+                 embedding_dropout: float = 0.1,
+                 cross_attention_dropout: float = 0.1,
+                 self_attention_dropout: float = 0.0,
+                 relu_dropout: float = 0.1,
+                 residual_dropout: float = 0.1,
+                 output_dropout: float = 0.1,
+                 attention_mask: bool = True):
+        super().__init__()
+        self.input_modality_channels = input_modality_channels
+        self.output_dim = output_dim
+        self.projected_modality_dim = projected_modality_dim
+        self.number_of_heads = number_of_heads
+        self.number_of_layers = number_of_layers
+        self.embedding_dropout = embedding_dropout
+        self.cross_attention_dropout = cross_attention_dropout
+        self.self_attention_dropout = self_attention_dropout
+        self.relu_dropout = relu_dropout
+        self.residual_dropout = residual_dropout
+        self.output_dropout = output_dropout
+        self.attention_mask = attention_mask
+
+        # 1. Temporal convolutional layers
+        self.projector = nn.Conv1d(input_modality_channels,
+                                   projected_modality_dim,
+                                   kernel_size=1,
+                                   padding=0,
+                                   bias=False)
+
+        # 2. Self Attention Linear Transformer
+        self.self_attention_transformer = TransformerEncoder(
+            embedding_dim=self.projected_modality_dim,
+            number_of_heads=self.number_of_heads,
+            number_of_layers=self.number_of_layers,
+            attention_dropout=self.self_attention_dropout,
+            relu_dropout=self.relu_dropout,
+            residual_dropout=self.residual_dropout,
+            embedding_dropout=self.self_attention_dropout,
+            attention_mask=self.attention_mask)
+
+        # 3. Projection layer
+        self.out_layer = nn.Linear(self.projected_modality_dim, self.output_dim)
+
+    def forward(self, input: torch.Tensor | list[torch.Tensor]) -> torch.Tensor:
+        """input tensor of shape (B, T, F)"""
+
+        if isinstance(input, list):
+            if len(input) == 1:
+                input = input[0]
+            else:
+                raise Exception(f'A single tensor is expected got instead {len(input)}.')
+
+        input = input.transpose(1, 2)
+
+        if self.embedding_dropout > 0:
+            input = F.dropout(input, p=self.embedding_dropout, training=self.training)
+
+        proj_x = self.projector(input)
+        proj_x = proj_x.permute(0, 2, 1)
+        hidden_representation = self.self_attention_transformer(proj_x)
+        output_seq = self.out_layer(hidden_representation)
+        return output_seq