x-transformers 1.27.4__tar.gz → 1.27.6__tar.gz

{x-transformers-1.27.4/x_transformers.egg-info → x-transformers-1.27.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: x-transformers
-Version: 1.27.4
+Version: 1.27.6
 Summary: X-Transformers - Pytorch
 Home-page: https://github.com/lucidrains/x-transformers
 Author: Phil Wang

{x-transformers-1.27.4 → x-transformers-1.27.6}/README.md

@@ -1171,6 +1171,8 @@ This flavor of attention also has <a href="https://arxiv.org/abs/2111.05498">a c
 
 Update: I have discovered a way to remove the learned temperature altogether, by grouping the feature dimension and doing l2-normalization on each group. This allows the queries and keys to have a similarity that is upper bounded by the number of groups. A group size of 8 or 16 was sufficient in my tests. Decided to name this technique "Grouped QK Normalization". The drawback is that I believe an attention head dimension 32 is too small to use this tactic (a dimension often used in vision)
 
+Update 2: Tero Karras has successfully used cosine sim attention in <a href="https://arxiv.org/abs/2312.02696">a new paper</a>.
+
 You can use it as follows
 
 ```python

{x-transformers-1.27.4 → x-transformers-1.27.6}/setup.py

@@ -3,7 +3,7 @@ from setuptools import setup, find_packages
 setup(
   name = 'x-transformers',
   packages = find_packages(exclude=['examples']),
-  version = '1.27.4',
+  version = '1.27.6',
   license='MIT',
   description = 'X-Transformers - Pytorch',
   author = 'Phil Wang',

{x-transformers-1.27.4 → x-transformers-1.27.6}/x_transformers/x_transformers.py

@@ -435,7 +435,7 @@ class RotaryEmbedding(nn.Module):
 
     @autocast(enabled = False)
     def forward(self, t):
-        device = self.inv_freq.device
+        device, seq_len = self.inv_freq.device, t.shape[-1]
 
         t = t.type_as(self.inv_freq)
 
@@ -1329,6 +1329,9 @@ class AttentionLayers(nn.Module):
             if exists(pre_norm):
                 x = pre_norm(x)
 
+                if exists(layer_mem):
+                    layer_mem = pre_norm(layer_mem)
+
             if layer_type == 'a':
                 out, inter = block(x, mask = mask, context_mask = self_attn_kv_mask, attn_mask = attn_mask, rel_pos = self.rel_pos, rotary_pos_emb = rotary_pos_emb, prev_attn = prev_attn, cache = next(iter_attn_cache, None), mem = layer_mem, mem_mask = layer_mem_mask, return_intermediates = True)
             elif layer_type == 'c':

{x-transformers-1.27.4 → x-transformers-1.27.6/x_transformers.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: x-transformers
-Version: 1.27.4
+Version: 1.27.6
 Summary: X-Transformers - Pytorch
 Home-page: https://github.com/lucidrains/x-transformers
 Author: Phil Wang