tokenizerchanger 0.0.1__tar.gz

TokenizerChanger-0.0.1/PKG-INFO

@@ -0,0 +1,92 @@
+Metadata-Version: 2.1
+Name: TokenizerChanger
+Version: 0.0.1
+Summary: Library for manipulating the existing tokenizer.
+Home-page: https://github.com/1kkiRen/Tokenizer-Changer
+Author: 1kkiren
+Author-email: 1kkiren@mail.ru
+Project-URL: GitHub, https://github.com/1kkiRen/Tokenizer-Changer
+Keywords: tokenizer deletion tokens
+Classifier: Programming Language :: Python :: 3.10
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+
+# Tokens-Deletion
+Python script for manipulating the existing tokenizer.
+
+The solution was tested on Llama3-8B tokenizer.
+
+-----
+# Usage:
+
+```python
+changer = TokenizerChanger(tokenizer)
+```
+Create the object of `TokenizerChanger` class that requires an existing tokenizer that will be changed, e.g. `PreTrainedTokenizerFast` class from рџ¤— Tokenizers library.
+
+## Deletion:
+```python
+changer.delete_k_least_frequent_tokens(k=1000)
+changer.delete_k_least_frequent_tokens(k=1000, exclude=list_of_tokens)
+```
+Deletes k most frequent tokens. The `exclude` argument stands for tokens that will be ignored during the deletion of least frequent tokens.
+
+```python
+changer.delete_unwanted_tokens(list_of_unwanted_tokens)
+```
+Deletes all tokens from `list_of_unwanted_tokens` from the tokenizer.
+
+```python
+changer.delete_tokens(list_of_unwanted_tokens)
+```
+Now, you can delete exactly the list of unwanted tokens, in contrast to the `delete_unwanted_tokens` function, which deletes all tokens from the list and tokens that contain unwanted tokens as a substring.
+
+```python
+changer.delete_overlaps(vocab)
+```
+Finds and deletes all intersections of the `tokenizer`'s vocabulary and the `vocab` variable from the `tokenizer`. Notice that `vocab` should be a `dict` variable.
+
+```python
+changer.delete_inappropriate_merges(vocab)
+```
+Deletes all merges from `tokenizer` which contradict the `vocab` variable. Notice that `vocab` should be a `list[str]` variable.
+
+
+## Addition:
+The idea of creating such functions arose due to the fact that the built-in functions do not add tokens/merges properly, when some tokens are deleted. That is why you can get more tokens after encoding the same text, even if the necessary tokens have been added.
+
+```python
+changer.add_tokens(list_of_tokens)
+```
+Adds the tokens from the list. The indexes will be filled automatically.
+
+```python
+changer.add_merges(list_of_merges)
+```
+Adds the merges from the list.
+
+
+## "Get" functions:
+```python
+changer.get_overlapping_tokens(vocab)
+```
+Returns the intersection between the `tokenizer`'s vocabulary and the `vocab` variable. Notice that `vocab` should be a `dict` variable.
+
+```python
+changer.get_overlapping_megres(merges)
+```
+Returns the intersection between the `tokenizer`'s merges and the `merges` variable. Notice that `merges` should be a `list` variable.
+
+
+## Saving:
+```python
+changer.save_tokenizer(path)
+```
+Saves the current state of the changed tokenizer. Additionally, it saves tokenizer configs into `path` folder (`./updated_tokenizer` by default).
+
+```python
+tokenizer = ch.updated_tokenizer()
+```
+Return the changed tokenizer.

TokenizerChanger-0.0.1/README.md

@@ -0,0 +1,77 @@
+# Tokens-Deletion
+Python script for manipulating the existing tokenizer.
+
+The solution was tested on Llama3-8B tokenizer.
+
+-----
+# Usage:
+
+```python
+changer = TokenizerChanger(tokenizer)
+```
+Create the object of `TokenizerChanger` class that requires an existing tokenizer that will be changed, e.g. `PreTrainedTokenizerFast` class from 🤗 Tokenizers library.
+
+## Deletion:
+```python
+changer.delete_k_least_frequent_tokens(k=1000)
+changer.delete_k_least_frequent_tokens(k=1000, exclude=list_of_tokens)
+```
+Deletes k most frequent tokens. The `exclude` argument stands for tokens that will be ignored during the deletion of least frequent tokens.
+
+```python
+changer.delete_unwanted_tokens(list_of_unwanted_tokens)
+```
+Deletes all tokens from `list_of_unwanted_tokens` from the tokenizer.
+
+```python
+changer.delete_tokens(list_of_unwanted_tokens)
+```
+Now, you can delete exactly the list of unwanted tokens, in contrast to the `delete_unwanted_tokens` function, which deletes all tokens from the list and tokens that contain unwanted tokens as a substring.
+
+```python
+changer.delete_overlaps(vocab)
+```
+Finds and deletes all intersections of the `tokenizer`'s vocabulary and the `vocab` variable from the `tokenizer`. Notice that `vocab` should be a `dict` variable.
+
+```python
+changer.delete_inappropriate_merges(vocab)
+```
+Deletes all merges from `tokenizer` which contradict the `vocab` variable. Notice that `vocab` should be a `list[str]` variable.
+
+
+## Addition:
+The idea of creating such functions arose due to the fact that the built-in functions do not add tokens/merges properly, when some tokens are deleted. That is why you can get more tokens after encoding the same text, even if the necessary tokens have been added.
+
+```python
+changer.add_tokens(list_of_tokens)
+```
+Adds the tokens from the list. The indexes will be filled automatically.
+
+```python
+changer.add_merges(list_of_merges)
+```
+Adds the merges from the list.
+
+
+## "Get" functions:
+```python
+changer.get_overlapping_tokens(vocab)
+```
+Returns the intersection between the `tokenizer`'s vocabulary and the `vocab` variable. Notice that `vocab` should be a `dict` variable.
+
+```python
+changer.get_overlapping_megres(merges)
+```
+Returns the intersection between the `tokenizer`'s merges and the `merges` variable. Notice that `merges` should be a `list` variable.
+
+
+## Saving:
+```python
+changer.save_tokenizer(path)
+```
+Saves the current state of the changed tokenizer. Additionally, it saves tokenizer configs into `path` folder (`./updated_tokenizer` by default).
+
+```python
+tokenizer = ch.updated_tokenizer()
+```
+Return the changed tokenizer.

TokenizerChanger-0.0.1/TokenizerChanger/__init__.py

@@ -0,0 +1,7 @@
+"""
+TokenizerChanger library v0.0.1
+
+The Apache 2.0 License Copyright © Dmitrii Kuzmin
+"""
+
+from .tokenizer_changer import *

TokenizerChanger-0.0.1/TokenizerChanger/tokenizer_changer.py

@@ -0,0 +1,159 @@
+import json
+from tqdm import tqdm
+from tokenizers import models
+from transformers import PreTrainedTokenizer
+
+
+class TokenizerChanger:
+    def __init__(self, tokenizer: PreTrainedTokenizer):
+        self.tokenizer: PreTrainedTokenizer = tokenizer
+        self.unwanted_tokens = []
+        self.none_types = []
+        self.target_changes = 0
+        self.model_state = json.loads(
+            tokenizer.backend_tokenizer.model.__getstate__())
+
+    def delete_tokens(self, unwanted_tokens: list[str] = None):
+        self.unwanted_tokens = list(set(unwanted_tokens)) if unwanted_tokens else list(
+            set(self.unwanted_tokens))
+        for token in tqdm(self.unwanted_tokens, desc="Deleting unwanted words"):
+            del self.model_state["vocab"][token]
+
+    def find_least_tokens(self, k_least: int, exclude: list[str] = []):
+        self.unwanted_tokens = []
+        for k, v in tqdm(dict(reversed(list(self.model_state["vocab"].items()))).items(), desc="Finding unwanted tokens"):
+            if len(self.unwanted_tokens) >= k_least:
+                break
+            if k not in exclude:
+                self.unwanted_tokens.append(k)
+
+    def find_tokens(self, unwanted_tokens: list[str]):
+        for token in self.model_state["vocab"]:
+            for unwanted_token in unwanted_tokens:
+                if unwanted_token in token:
+                    self.unwanted_tokens.append(token)
+
+    def delete_merges(self, unwanted_tokens: list[str] = None):
+        processed_merges = [(''.join(merge).replace(' ', ''), merge)
+                            for merge in self.model_state["merges"]]
+
+        unwanted_merges_set = set()
+
+        self.unwanted_tokens = list(set(unwanted_tokens)) if unwanted_tokens else list(
+            set(self.unwanted_tokens))
+
+        for processed_merge, original_merge in tqdm(processed_merges, desc="Finding unwanted merges"):
+            if any(token in processed_merge for token in self.unwanted_tokens):
+                unwanted_merges_set.add(original_merge)
+
+        self.model_state["merges"] = [merge for merge in tqdm(
+            self.model_state["merges"], desc="Deleting unwanted merges") if merge not in unwanted_merges_set]
+
+    def find_token_id_gap(self):
+        reversed_vocab_values = list(
+            reversed(self.model_state['vocab'].values()))
+        last_gap = 0
+        for i in range(len(self.model_state['vocab']) - 1):
+            if reversed_vocab_values[i] - reversed_vocab_values[i + 1] > 1:
+                last_gap = reversed_vocab_values[i + 1]
+
+        return last_gap
+
+    def add_tokens(self, tokens: list[str]):
+        i = 1
+        border_id = self.find_token_id_gap()
+        for token in tqdm(tokens, desc="Adding tokens"):
+            if token not in self.model_state["vocab"]:
+                while border_id + i in self.model_state['vocab'].values():
+                    i += 1
+                self.model_state["vocab"][token] = border_id + i
+                i += 1
+
+    def add_merges(self, merges: list[str]):
+        for merge in tqdm(self.model_state["merges"], desc="Adding merges"):
+            merges.append(merge)
+
+        self.model_state["merges"] = list(set(merges))
+
+    def delete_inappropriate_merges(self, vocab: list[str]):
+        processed_merges = [(''.join(merge).replace(' ', ''), merge)
+                            for merge in self.model_state["merges"]]
+
+        unwanted_merges_set = set()
+
+        for processed_merge, original_merge in tqdm(processed_merges, desc="Finding unwanted merges"):
+            if not all(token in vocab for token in [processed_merge, original_merge[0], original_merge[1]]):
+                unwanted_merges_set.add(original_merge)
+
+        self.model_state["merges"] = [merge for merge in tqdm(
+            self.model_state["merges"], desc="Deleting unwanted merges") if merge not in unwanted_merges_set]
+
+    def get_overlapping_tokens(self, vocab: dict):
+        overlapping_tokens = []
+        for token in tqdm(vocab.keys(), desc="Finding overlapping tokens"):
+            if token in self.model_state["vocab"].keys():
+                overlapping_tokens.append(token)
+        return overlapping_tokens
+
+    def get_overlapping_megres(self, merges: list):
+        overlapping_merges = []
+
+        processed_merges_new_tokenizer = [(''.join(merge).replace(' ', ''), merge)
+                                          for merge in self.model_state["merges"]]
+
+        processed_merges_old_tokenizer = [(''.join(merge).replace(' ', ''), merge)
+                                          for merge in merges]
+
+        for merge in tqdm(processed_merges_new_tokenizer, desc="Finding overlapping merges"):
+            if any(merge in processed_merge for processed_merge in processed_merges_old_tokenizer):
+                overlapping_merges.append(merge)
+
+        return overlapping_merges
+
+    def format_merges(self):
+        for i in tqdm(range(len(self.model_state["merges"])), desc="Formating merges"):
+            if type(self.model_state["merges"][i]) != tuple:
+                self.model_state["merges"][i] = tuple(
+                    map(str, self.model_state["merges"][i].split()))
+
+    def delete_none_types(self):
+        for k, v in self.model_state.items():
+            if v == None:
+                self.none_types.append(k)
+
+        for k in self.none_types:
+            del self.model_state[k]
+
+    def delete_k_least_frequent_tokens(self, k: int, exclude: list[str] = []):
+        self.find_least_tokens(k, exclude)
+        self.delete_tokens()
+        self.delete_merges()
+
+    def delete_unwanted_tokens(self, unwanted_tokens: list):
+        self.find_tokens(unwanted_tokens)
+        self.delete_tokens()
+        self.delete_merges()
+
+    def delete_overlaps(self, vocab: dict):
+        overlaps = list(set(self.get_overlapping_tokens(vocab)))
+        self.delete_tokens(unwanted_tokens=overlaps)
+        self.delete_merges()
+
+    def save_tokenizer(self, path: str = "updated_tokenizer"):
+        self.format_merges()
+        self.delete_none_types()
+
+        model_class = getattr(
+            models, self.model_state.pop("type")
+        )
+
+        self.tokenizer.backend_tokenizer.model = model_class(
+            **self.model_state)
+
+        self.model_state = json.loads(
+            self.tokenizer.backend_tokenizer.model.__getstate__())
+
+        self.tokenizer.save_pretrained(path)
+
+    def updated_tokenizer(self) -> PreTrainedTokenizer:
+        return self.tokenizer

TokenizerChanger-0.0.1/TokenizerChanger.egg-info/PKG-INFO

@@ -0,0 +1,92 @@
+Metadata-Version: 2.1
+Name: TokenizerChanger
+Version: 0.0.1
+Summary: Library for manipulating the existing tokenizer.
+Home-page: https://github.com/1kkiRen/Tokenizer-Changer
+Author: 1kkiren
+Author-email: 1kkiren@mail.ru
+Project-URL: GitHub, https://github.com/1kkiRen/Tokenizer-Changer
+Keywords: tokenizer deletion tokens
+Classifier: Programming Language :: Python :: 3.10
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+
+# Tokens-Deletion
+Python script for manipulating the existing tokenizer.
+
+The solution was tested on Llama3-8B tokenizer.
+
+-----
+# Usage:
+
+```python
+changer = TokenizerChanger(tokenizer)
+```
+Create the object of `TokenizerChanger` class that requires an existing tokenizer that will be changed, e.g. `PreTrainedTokenizerFast` class from рџ¤— Tokenizers library.
+
+## Deletion:
+```python
+changer.delete_k_least_frequent_tokens(k=1000)
+changer.delete_k_least_frequent_tokens(k=1000, exclude=list_of_tokens)
+```
+Deletes k most frequent tokens. The `exclude` argument stands for tokens that will be ignored during the deletion of least frequent tokens.
+
+```python
+changer.delete_unwanted_tokens(list_of_unwanted_tokens)
+```
+Deletes all tokens from `list_of_unwanted_tokens` from the tokenizer.
+
+```python
+changer.delete_tokens(list_of_unwanted_tokens)
+```
+Now, you can delete exactly the list of unwanted tokens, in contrast to the `delete_unwanted_tokens` function, which deletes all tokens from the list and tokens that contain unwanted tokens as a substring.
+
+```python
+changer.delete_overlaps(vocab)
+```
+Finds and deletes all intersections of the `tokenizer`'s vocabulary and the `vocab` variable from the `tokenizer`. Notice that `vocab` should be a `dict` variable.
+
+```python
+changer.delete_inappropriate_merges(vocab)
+```
+Deletes all merges from `tokenizer` which contradict the `vocab` variable. Notice that `vocab` should be a `list[str]` variable.
+
+
+## Addition:
+The idea of creating such functions arose due to the fact that the built-in functions do not add tokens/merges properly, when some tokens are deleted. That is why you can get more tokens after encoding the same text, even if the necessary tokens have been added.
+
+```python
+changer.add_tokens(list_of_tokens)
+```
+Adds the tokens from the list. The indexes will be filled automatically.
+
+```python
+changer.add_merges(list_of_merges)
+```
+Adds the merges from the list.
+
+
+## "Get" functions:
+```python
+changer.get_overlapping_tokens(vocab)
+```
+Returns the intersection between the `tokenizer`'s vocabulary and the `vocab` variable. Notice that `vocab` should be a `dict` variable.
+
+```python
+changer.get_overlapping_megres(merges)
+```
+Returns the intersection between the `tokenizer`'s merges and the `merges` variable. Notice that `merges` should be a `list` variable.
+
+
+## Saving:
+```python
+changer.save_tokenizer(path)
+```
+Saves the current state of the changed tokenizer. Additionally, it saves tokenizer configs into `path` folder (`./updated_tokenizer` by default).
+
+```python
+tokenizer = ch.updated_tokenizer()
+```
+Return the changed tokenizer.

TokenizerChanger-0.0.1/TokenizerChanger.egg-info/SOURCES.txt

@@ -0,0 +1,10 @@
+README.md
+setup.cfg
+setup.py
+TokenizerChanger/__init__.py
+TokenizerChanger/tokenizer_changer.py
+TokenizerChanger.egg-info/PKG-INFO
+TokenizerChanger.egg-info/SOURCES.txt
+TokenizerChanger.egg-info/dependency_links.txt
+TokenizerChanger.egg-info/requires.txt
+TokenizerChanger.egg-info/top_level.txt

TokenizerChanger-0.0.1/TokenizerChanger.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

TokenizerChanger-0.0.1/TokenizerChanger.egg-info/requires.txt

@@ -0,0 +1,3 @@
+tokenizers>=0.19.1
+tqdm>=4.66.4
+transformers>=4.41.2

TokenizerChanger-0.0.1/TokenizerChanger.egg-info/top_level.txt

@@ -0,0 +1 @@
+TokenizerChanger

TokenizerChanger-0.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

TokenizerChanger-0.0.1/setup.py

@@ -0,0 +1,34 @@
+from setuptools import setup, find_packages
+
+
+def readme():
+  with open('README.md', 'r') as f:
+    return f.read()
+
+
+setup(
+  name='TokenizerChanger',
+  version='0.0.1',
+  author='1kkiren',
+  author_email='1kkiren@mail.ru',
+  description='Library for manipulating the existing tokenizer.',
+  long_description=readme(),
+  long_description_content_type='text/markdown',
+  url='https://github.com/1kkiRen/Tokenizer-Changer',
+  packages=find_packages(),
+  install_requires=[
+    'tokenizers>=0.19.1',
+    'tqdm>=4.66.4',
+    'transformers>=4.41.2'
+    ],
+  classifiers=[
+    'Programming Language :: Python :: 3.10',
+    'License :: OSI Approved :: Apache Software License',
+    'Operating System :: OS Independent'
+  ],
+  keywords='tokenizer deletion tokens ',
+  project_urls={
+    'GitHub': 'https://github.com/1kkiRen/Tokenizer-Changer'
+  },
+  python_requires='>=3.9'
+)