tokenicer 0.0.1.dev0__tar.gz → 0.0.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,3 +1,5 @@
1
+ Copyright 2025 ModelCloud.ai
2
+ Copyright 2025 qubitium@modelcloud.ai
1
3
  Apache License
2
4
  Version 2.0, January 2004
3
5
  http://www.apache.org/licenses/
@@ -0,0 +1,2 @@
1
+ include requirements.txt
2
+ prune tests/
@@ -0,0 +1,97 @@
1
+ Metadata-Version: 2.2
2
+ Name: tokenicer
3
+ Version: 0.0.2
4
+ Summary: A (nicer) tokenizer you want to use for model `inference` and `training`: with all known peventable `gotchas` normalized or auto-fixed.
5
+ Home-page: https://github.com/ModelCloud/Tokenicer
6
+ Author: ModelCloud
7
+ Author-email: qubitium@modelcloud.ai
8
+ Classifier: Programming Language :: Python :: 3
9
+ Classifier: License :: OSI Approved :: Apache Software License
10
+ Classifier: Operating System :: OS Independent
11
+ Requires-Python: >=3
12
+ Description-Content-Type: text/markdown
13
+ License-File: LICENSE
14
+ Requires-Dist: transformers
15
+ Dynamic: author
16
+ Dynamic: author-email
17
+ Dynamic: classifier
18
+ Dynamic: description
19
+ Dynamic: description-content-type
20
+ Dynamic: home-page
21
+ Dynamic: requires-dist
22
+ Dynamic: requires-python
23
+ Dynamic: summary
24
+
25
+ <h1 align="center">Toke(n)icer</h1>
26
+ <p align="center">A (nicer) tokenizer you want to use for model inference and training: with all known peventable gotchas normalized or auto-fixed.</p>
27
+ <p align="center">
28
+ <a href="https://github.com/ModelCloud/Tokenicer/releases" style="text-decoration:none;"><img alt="GitHub release" src="https://img.shields.io/github/release/ModelCloud/Tokenicer.svg"></a>
29
+ <a href="https://pypi.org/project/tokenicer/" style="text-decoration:none;"><img alt="PyPI - Version" src="https://img.shields.io/pypi/v/tokenicer"></a>
30
+ <a href="https://pepy.tech/projects/tokenicer" style="text-decoration:none;"><img src="https://static.pepy.tech/badge/tokenicer" alt="PyPI Downloads"></a>
31
+ <a href="https://github.com/ModelCloud/tokenicer/blob/main/LICENSE"><img src="https://img.shields.io/pypi/l/tokenicer"></a>
32
+ <a href="https://huggingface.co/modelcloud/"><img src="https://img.shields.io/badge/🤗%20Hugging%20Face-ModelCloud-%23ff8811.svg"></a>
33
+ </p>
34
+
35
+ ## News
36
+ * 02/10/2025 [0.0.2](https://github.com/ModelCloud/Tokenicer/releases/tag/v0.0.2): 🤗 Initial release!
37
+
38
+ ## Features:
39
+
40
+ * Compatible with all HF `Transformers` recognized tokenizers
41
+ * Auto-fix `models` not setting `padding_token`
42
+ * Auto-Fix `models` released with wrong `padding_token`: many `models` incorrectly use `eos_token` as `pad_token` which leads to subtle and hidden errors in post-training and inference when `batching` is used which is almost always.
43
+ * Zero external dependency outside of `Transformers`
44
+
45
+ ## Upcoming Features:
46
+
47
+ * Add `automatic` tokenizer validation to `model` `training` and subsequent `inference` so that not only tokenizer config but actual `decode`/`encode` are 100% re-validated on model load. Often the case, `inference` and `training` engines modifies the traditional tokenizers causing subtle and inaccurate output when `inference` performed on a platform that is disjointed from the `trainer`.
48
+
49
+ ## Install
50
+
51
+ ### PIP/UV
52
+
53
+ ```bash
54
+ pip install -v tokenicer
55
+ uv pip install -v tokenicer
56
+ ```
57
+
58
+ ### Install from source
59
+
60
+ ```bash
61
+ # clone repo
62
+ git clone https://github.com/ModelCloud/Tokencier.git && cd Tokenicer
63
+
64
+ # compile
65
+ pip install -v .
66
+ ```
67
+
68
+ ## Usage
69
+
70
+ * Replace all calls to `AutoTokenizer.from_pretrained()` with `Tokenizer.load()`: args are 100% compatible with `AutoTokenizer`
71
+
72
+ ```py
73
+ # Replace `AutoTokenizer.from_pretrained()`
74
+ # from tokenizer import AutoTokenizer
75
+ # tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-0.5B-Instruct')
76
+
77
+ # With `Tokenicer.load()`
78
+ from tokenicer import Tokenicer
79
+ tokenizer = Tokenicer.load('Qwen/Qwen2.5-0.5B-Instruct')
80
+
81
+ # That's it! Toke(n)icer has auto-fixed Qwen2.5-0.5B-Instruct's incorrect `pad_token`.
82
+ # Now this this model can be `trained` and `inferenced` correctly with `batch` and `masks`.
83
+ print(f"pad_token: `{tokenizer.pad_token}`")
84
+ ```
85
+
86
+ ## Citation
87
+
88
+ ```
89
+ @misc{gptqmodel,
90
+ author = {ModelCloud.ai and qubitium@modelcloud.ai},
91
+ title = {Toke(n)icer},
92
+ year = {2025},
93
+ publisher = {GitHub},
94
+ journal = {GitHub repository},
95
+ howpublished = {\url{https://github.com/modelcloud/tokenicer}},
96
+ note = {Contact: qubitium@modelcloud.ai}
97
+ }
@@ -0,0 +1,73 @@
1
+ <h1 align="center">Toke(n)icer</h1>
2
+ <p align="center">A (nicer) tokenizer you want to use for model inference and training: with all known peventable gotchas normalized or auto-fixed.</p>
3
+ <p align="center">
4
+ <a href="https://github.com/ModelCloud/Tokenicer/releases" style="text-decoration:none;"><img alt="GitHub release" src="https://img.shields.io/github/release/ModelCloud/Tokenicer.svg"></a>
5
+ <a href="https://pypi.org/project/tokenicer/" style="text-decoration:none;"><img alt="PyPI - Version" src="https://img.shields.io/pypi/v/tokenicer"></a>
6
+ <a href="https://pepy.tech/projects/tokenicer" style="text-decoration:none;"><img src="https://static.pepy.tech/badge/tokenicer" alt="PyPI Downloads"></a>
7
+ <a href="https://github.com/ModelCloud/tokenicer/blob/main/LICENSE"><img src="https://img.shields.io/pypi/l/tokenicer"></a>
8
+ <a href="https://huggingface.co/modelcloud/"><img src="https://img.shields.io/badge/🤗%20Hugging%20Face-ModelCloud-%23ff8811.svg"></a>
9
+ </p>
10
+
11
+ ## News
12
+ * 02/10/2025 [0.0.2](https://github.com/ModelCloud/Tokenicer/releases/tag/v0.0.2): 🤗 Initial release!
13
+
14
+ ## Features:
15
+
16
+ * Compatible with all HF `Transformers` recognized tokenizers
17
+ * Auto-fix `models` not setting `padding_token`
18
+ * Auto-Fix `models` released with wrong `padding_token`: many `models` incorrectly use `eos_token` as `pad_token` which leads to subtle and hidden errors in post-training and inference when `batching` is used which is almost always.
19
+ * Zero external dependency outside of `Transformers`
20
+
21
+ ## Upcoming Features:
22
+
23
+ * Add `automatic` tokenizer validation to `model` `training` and subsequent `inference` so that not only tokenizer config but actual `decode`/`encode` are 100% re-validated on model load. Often the case, `inference` and `training` engines modifies the traditional tokenizers causing subtle and inaccurate output when `inference` performed on a platform that is disjointed from the `trainer`.
24
+
25
+ ## Install
26
+
27
+ ### PIP/UV
28
+
29
+ ```bash
30
+ pip install -v tokenicer
31
+ uv pip install -v tokenicer
32
+ ```
33
+
34
+ ### Install from source
35
+
36
+ ```bash
37
+ # clone repo
38
+ git clone https://github.com/ModelCloud/Tokencier.git && cd Tokenicer
39
+
40
+ # compile
41
+ pip install -v .
42
+ ```
43
+
44
+ ## Usage
45
+
46
+ * Replace all calls to `AutoTokenizer.from_pretrained()` with `Tokenizer.load()`: args are 100% compatible with `AutoTokenizer`
47
+
48
+ ```py
49
+ # Replace `AutoTokenizer.from_pretrained()`
50
+ # from tokenizer import AutoTokenizer
51
+ # tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-0.5B-Instruct')
52
+
53
+ # With `Tokenicer.load()`
54
+ from tokenicer import Tokenicer
55
+ tokenizer = Tokenicer.load('Qwen/Qwen2.5-0.5B-Instruct')
56
+
57
+ # That's it! Toke(n)icer has auto-fixed Qwen2.5-0.5B-Instruct's incorrect `pad_token`.
58
+ # Now this this model can be `trained` and `inferenced` correctly with `batch` and `masks`.
59
+ print(f"pad_token: `{tokenizer.pad_token}`")
60
+ ```
61
+
62
+ ## Citation
63
+
64
+ ```
65
+ @misc{gptqmodel,
66
+ author = {ModelCloud.ai and qubitium@modelcloud.ai},
67
+ title = {Toke(n)icer},
68
+ year = {2025},
69
+ publisher = {GitHub},
70
+ journal = {GitHub repository},
71
+ howpublished = {\url{https://github.com/modelcloud/tokenicer}},
72
+ note = {Contact: qubitium@modelcloud.ai}
73
+ }
@@ -0,0 +1 @@
1
+ transformers
@@ -0,0 +1,43 @@
1
+ # Copyright 2025 ModelCloud.ai
2
+ # Copyright 2025 qubitium@modelcloud.ai
3
+ # Contact: qubitium@modelcloud.ai, x.com/qubitium
4
+ #
5
+ # Licensed under the Apache License, Version 2.0 (the "License");
6
+ # you may not use this file except in compliance with the License.
7
+ # You may obtain a copy of the License at
8
+ #
9
+ # http://www.apache.org/licenses/LICENSE-2.0
10
+ #
11
+ # Unless required by applicable law or agreed to in writing, software
12
+ # distributed under the License is distributed on an "AS IS" BASIS,
13
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
+ # See the License for the specific language governing permissions and
15
+ # limitations under the License.
16
+
17
+ from setuptools import setup, find_packages
18
+ from pathlib import Path
19
+
20
+ __version__ = "0.0.2"
21
+
22
+ with open("requirements.txt") as f:
23
+ requirements = f.read().splitlines()
24
+
25
+ setup(
26
+ name="tokenicer",
27
+ version=__version__,
28
+ author="ModelCloud",
29
+ author_email="qubitium@modelcloud.ai",
30
+ description="A (nicer) tokenizer you want to use for model `inference` and `training`: with all known peventable `gotchas` normalized or auto-fixed.",
31
+ long_description=(Path(__file__).parent / "README.md").read_text(encoding="UTF-8"),
32
+ long_description_content_type="text/markdown",
33
+ url="https://github.com/ModelCloud/Tokenicer",
34
+ packages=find_packages(),
35
+ install_requires=requirements,
36
+ platform=["linux", "windows", "darwin"],
37
+ classifiers=[
38
+ "Programming Language :: Python :: 3",
39
+ "License :: OSI Approved :: Apache Software License",
40
+ "Operating System :: OS Independent",
41
+ ],
42
+ python_requires=">=3",
43
+ )
@@ -0,0 +1,19 @@
1
+ # Copyright 2025 ModelCloud.ai
2
+ # Copyright 2025 qubitium@modelcloud.ai
3
+ # Contact: qubitium@modelcloud.ai, x.com/qubitium
4
+ #
5
+ # Licensed under the Apache License, Version 2.0 (the "License");
6
+ # you may not use this file except in compliance with the License.
7
+ # You may obtain a copy of the License at
8
+ #
9
+ # http://www.apache.org/licenses/LICENSE-2.0
10
+ #
11
+ # Unless required by applicable law or agreed to in writing, software
12
+ # distributed under the License is distributed on an "AS IS" BASIS,
13
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
+ # See the License for the specific language governing permissions and
15
+ # limitations under the License.
16
+
17
+ from .tokenicer import Tokenicer
18
+
19
+ __all__ = ["Tokenicer"]
@@ -0,0 +1,35 @@
1
+ # Copyright 2025 ModelCloud.ai
2
+ # Copyright 2025 qubitium@modelcloud.ai
3
+ # Contact: qubitium@modelcloud.ai, x.com/qubitium
4
+ #
5
+ # Licensed under the Apache License, Version 2.0 (the "License");
6
+ # you may not use this file except in compliance with the License.
7
+ # You may obtain a copy of the License at
8
+ #
9
+ # http://www.apache.org/licenses/LICENSE-2.0
10
+ #
11
+ # Unless required by applicable law or agreed to in writing, software
12
+ # distributed under the License is distributed on an "AS IS" BASIS,
13
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
+ # See the License for the specific language governing permissions and
15
+ # limitations under the License.
16
+
17
+ from collections import namedtuple
18
+
19
+ DEFAULT_PAD_TOKENS = [
20
+ "<|finetune_right_pad_id|>",
21
+ "<|pad|>",
22
+ "<pad>",
23
+ "<|unk|>",
24
+ "<unk>"
25
+ ]
26
+
27
+ TOKEN_TUPLE = namedtuple("TokenTuple", ["token", "token_id"])
28
+
29
+ MODEL_PAD_TOKEN_MAP = {
30
+ "llama": TOKEN_TUPLE(token='<|finetune_right_pad_id|>', token_id=128004),
31
+ "qwen2_5_vl": TOKEN_TUPLE(token='<|vision_pad|>', token_id=151654),
32
+ "qwen2_vl": TOKEN_TUPLE(token='<|vision_pad|>', token_id=151654),
33
+ "qwen2": TOKEN_TUPLE(token='<|fim_pad|>', token_id=151662),
34
+ "deepseek_v3": TOKEN_TUPLE(token='<|▁pad▁|>', token_id=2),
35
+ }
@@ -0,0 +1,145 @@
1
+ # Copyright 2025 ModelCloud.ai
2
+ # Copyright 2025 qubitium@modelcloud.ai
3
+ # Contact: qubitium@modelcloud.ai, x.com/qubitium
4
+ #
5
+ # Licensed under the Apache License, Version 2.0 (the "License");
6
+ # you may not use this file except in compliance with the License.
7
+ # You may obtain a copy of the License at
8
+ #
9
+ # http://www.apache.org/licenses/LICENSE-2.0
10
+ #
11
+ # Unless required by applicable law or agreed to in writing, software
12
+ # distributed under the License is distributed on an "AS IS" BASIS,
13
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
+ # See the License for the specific language governing permissions and
15
+ # limitations under the License.
16
+
17
+ import logging
18
+ from typing import Union, List, Optional
19
+ from transformers import PreTrainedTokenizerBase, PreTrainedModel, AutoTokenizer
20
+ from .util import candidate_id, config_path, auto_config
21
+ from .const import DEFAULT_PAD_TOKENS, MODEL_PAD_TOKEN_MAP
22
+
23
+ logger = logging.getLogger(__name__)
24
+ logging.basicConfig(level=logging.INFO)
25
+
26
+
27
+ class Tokenicer:
28
+ tokenizer: Union[str, PreTrainedTokenizerBase] = None
29
+ model_config = None
30
+
31
+ @classmethod
32
+ def load(cls, pretrained_model_name_or_path: Union[str, PreTrainedTokenizerBase], **kwargs):
33
+ if pretrained_model_name_or_path is None:
34
+ raise ValueError("`pretrained_model_name_or_path` cannot be `None`.")
35
+
36
+ trust_remote_code = kwargs.get('trust_remote_code', False)
37
+
38
+ tokenicer = cls()
39
+
40
+ path = None
41
+ if isinstance(pretrained_model_name_or_path, PreTrainedTokenizerBase):
42
+ tokenizer = pretrained_model_name_or_path
43
+ tokenicer.tokenizer = tokenizer
44
+ path = config_path(tokenizer)
45
+ elif isinstance(pretrained_model_name_or_path, str):
46
+ tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path, **kwargs)
47
+ if isinstance(tokenizer, PreTrainedTokenizerBase):
48
+ tokenicer.tokenizer = tokenizer
49
+ path = pretrained_model_name_or_path
50
+ else:
51
+ ValueError(
52
+ f"Failed to initialize `tokenizer`: please ensure that the `pretrained_model_name_or_path` parameter is set correctly.")
53
+ else:
54
+ raise ValueError(
55
+ f"Unsupported `pretrained_model_name_or_path` type: Expected `str` or `PreTrainedTokenizerBase`, actual = `{type(pretrained_model_name_or_path)}`.")
56
+
57
+ tokenicer.model_config = auto_config(path, trust_remote_code)
58
+
59
+ if tokenicer.model_config is None:
60
+ logger.warning(
61
+ f"Auto model config retrieval from `pretrained_model_name_or_path` failed. "
62
+ f"Please pass a valid `model_or_path` argument to `auto_assign_pad_token()`.",
63
+ )
64
+
65
+ tokenicer.auto_fix_pad_token()
66
+
67
+ return tokenicer
68
+
69
+ def auto_fix_pad_token(
70
+ self,
71
+ model_or_path: Optional[Union[str, PreTrainedModel]] = None,
72
+ pad_tokens: Optional[List[Union[str, int]]] = None,
73
+ ):
74
+ model_config = None
75
+ if model_or_path is not None:
76
+ if isinstance(model_or_path, str):
77
+ model_config = auto_config(model_or_path, self.tokenizer.trust_remote_code)
78
+ elif isinstance(model_or_path, PreTrainedModel):
79
+ model_config = getattr(model_or_path, "config", None)
80
+ else:
81
+ raise ValueError(
82
+ f"Unsupported `model_or_path` type: Expected `str` or `PreTrainedModel`, actual = `{type(model_or_path)}`.")
83
+
84
+ if model_config is None:
85
+ raise ValueError("Can not retrieve config from the provided `model_or_path`.")
86
+ else:
87
+ if self.model_config is not None:
88
+ model_config = self.model_config
89
+ else:
90
+ raise ValueError(
91
+ f"Auto model config retrieval from `pretrained_model_name_or_path` failed. "
92
+ f"Please pass a valid `model_or_path` argument to `auto_assign_pad_token()`.",
93
+ )
94
+
95
+ pad_token_id = model_config.pad_token_id
96
+
97
+ if pad_token_id is None or pad_token_id in [model_config.bos_token_id, model_config.eos_token_id]:
98
+ pad_token_id = self._auto_map_pad_token(model_config=model_config, pad_tokens=pad_tokens)
99
+ if pad_token_id is None:
100
+ raise ValueError(
101
+ "Model tokenizer requires fixing but we are unnable to auto-fix `pad_token`. Please consult model docks manually pass a `pad_token` to `load()`."
102
+ )
103
+
104
+ self.tokenizer.pad_token_id = pad_token_id
105
+ self.tokenizer.pad_token = self.tokenizer.decode([pad_token_id])
106
+
107
+ logger.info(f"Auto fixed pad_token_id={pad_token_id} (token='{self.tokenizer.pad_token}').")
108
+
109
+ def _auto_map_pad_token(self, model_config, pad_tokens) -> Optional[int]:
110
+ pad_token_id = None
111
+
112
+ vocab = self.tokenizer.get_vocab()
113
+
114
+ # Prioritize matching of pad token entered by the user
115
+ if pad_tokens is not None:
116
+ pad_token_id = candidate_id(pad_tokens, vocab)
117
+
118
+ # Match MODEL_PAD_TOKEN_MAP to get pad token
119
+ if pad_token_id is None and MODEL_PAD_TOKEN_MAP.get(model_config.model_type, None) is not None:
120
+ token_tuple = MODEL_PAD_TOKEN_MAP.get(model_config.model_type)
121
+ pad_token = token_tuple.token
122
+ token_id = vocab.get(pad_token, None)
123
+ if token_id is not None and token_id == token_tuple.token_id:
124
+ pad_token_id = token_id
125
+
126
+ # Match DEFAULT_PAD_TOKENS to get pad token
127
+ if pad_token_id is None:
128
+ pad_token_id = candidate_id(DEFAULT_PAD_TOKENS, vocab)
129
+
130
+ # Use eos_token as pad token
131
+ if pad_token_id is None:
132
+ if isinstance(model_config.eos_token_id, list) and model_config.eos_token_id:
133
+ pad_token_id = model_config.eos_token_id[0]
134
+ else:
135
+ pad_token_id = model_config.eos_token_id
136
+ return pad_token_id
137
+
138
+ def __getattr__(self, name):
139
+ if hasattr(self.tokenizer, name):
140
+ return getattr(self.tokenizer, name)
141
+ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
142
+
143
+ def __call__(self, data, **kwargs):
144
+ return self.tokenizer(data, **kwargs)
145
+
@@ -0,0 +1,49 @@
1
+ # Copyright 2025 ModelCloud.ai
2
+ # Copyright 2025 qubitium@modelcloud.ai
3
+ # Contact: qubitium@modelcloud.ai, x.com/qubitium
4
+ #
5
+ # Licensed under the Apache License, Version 2.0 (the "License");
6
+ # you may not use this file except in compliance with the License.
7
+ # You may obtain a copy of the License at
8
+ #
9
+ # http://www.apache.org/licenses/LICENSE-2.0
10
+ #
11
+ # Unless required by applicable law or agreed to in writing, software
12
+ # distributed under the License is distributed on an "AS IS" BASIS,
13
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
+ # See the License for the specific language governing permissions and
15
+ # limitations under the License.
16
+
17
+ from typing import Union, List, Optional
18
+ from transformers import AutoConfig, PretrainedConfig
19
+
20
+
21
+ def candidate_ids(token_list: List[Union[str, int]], vocab: dict) -> List[Optional[int]]:
22
+ token_ids = []
23
+ for item in token_list:
24
+ if isinstance(item, str):
25
+ val = vocab.get(item)
26
+ if val is not None:
27
+ token_ids.append(val)
28
+ elif isinstance(item, int):
29
+ if 0 <= item < len(vocab):
30
+ token_ids.append(item)
31
+ return token_ids
32
+
33
+
34
+ def candidate_id(token_list: List[Union[str, int]], vocab: dict) -> Optional[int]:
35
+ token_ids = candidate_ids(token_list=token_list, vocab=vocab)
36
+ return token_ids[0] if token_ids else None
37
+
38
+
39
+ def config_path(obj) -> Optional[str]:
40
+ path = getattr(obj, "name_or_path", None)
41
+ return path
42
+
43
+
44
+ def auto_config(path, trust_remote) -> Optional[PretrainedConfig]:
45
+ config = AutoConfig.from_pretrained(path, trust_remote_code=trust_remote)
46
+ model_config = None
47
+ if isinstance(config, PretrainedConfig):
48
+ model_config = config
49
+ return model_config
@@ -0,0 +1,97 @@
1
+ Metadata-Version: 2.2
2
+ Name: tokenicer
3
+ Version: 0.0.2
4
+ Summary: A (nicer) tokenizer you want to use for model `inference` and `training`: with all known peventable `gotchas` normalized or auto-fixed.
5
+ Home-page: https://github.com/ModelCloud/Tokenicer
6
+ Author: ModelCloud
7
+ Author-email: qubitium@modelcloud.ai
8
+ Classifier: Programming Language :: Python :: 3
9
+ Classifier: License :: OSI Approved :: Apache Software License
10
+ Classifier: Operating System :: OS Independent
11
+ Requires-Python: >=3
12
+ Description-Content-Type: text/markdown
13
+ License-File: LICENSE
14
+ Requires-Dist: transformers
15
+ Dynamic: author
16
+ Dynamic: author-email
17
+ Dynamic: classifier
18
+ Dynamic: description
19
+ Dynamic: description-content-type
20
+ Dynamic: home-page
21
+ Dynamic: requires-dist
22
+ Dynamic: requires-python
23
+ Dynamic: summary
24
+
25
+ <h1 align="center">Toke(n)icer</h1>
26
+ <p align="center">A (nicer) tokenizer you want to use for model inference and training: with all known peventable gotchas normalized or auto-fixed.</p>
27
+ <p align="center">
28
+ <a href="https://github.com/ModelCloud/Tokenicer/releases" style="text-decoration:none;"><img alt="GitHub release" src="https://img.shields.io/github/release/ModelCloud/Tokenicer.svg"></a>
29
+ <a href="https://pypi.org/project/tokenicer/" style="text-decoration:none;"><img alt="PyPI - Version" src="https://img.shields.io/pypi/v/tokenicer"></a>
30
+ <a href="https://pepy.tech/projects/tokenicer" style="text-decoration:none;"><img src="https://static.pepy.tech/badge/tokenicer" alt="PyPI Downloads"></a>
31
+ <a href="https://github.com/ModelCloud/tokenicer/blob/main/LICENSE"><img src="https://img.shields.io/pypi/l/tokenicer"></a>
32
+ <a href="https://huggingface.co/modelcloud/"><img src="https://img.shields.io/badge/🤗%20Hugging%20Face-ModelCloud-%23ff8811.svg"></a>
33
+ </p>
34
+
35
+ ## News
36
+ * 02/10/2025 [0.0.2](https://github.com/ModelCloud/Tokenicer/releases/tag/v0.0.2): 🤗 Initial release!
37
+
38
+ ## Features:
39
+
40
+ * Compatible with all HF `Transformers` recognized tokenizers
41
+ * Auto-fix `models` not setting `padding_token`
42
+ * Auto-Fix `models` released with wrong `padding_token`: many `models` incorrectly use `eos_token` as `pad_token` which leads to subtle and hidden errors in post-training and inference when `batching` is used which is almost always.
43
+ * Zero external dependency outside of `Transformers`
44
+
45
+ ## Upcoming Features:
46
+
47
+ * Add `automatic` tokenizer validation to `model` `training` and subsequent `inference` so that not only tokenizer config but actual `decode`/`encode` are 100% re-validated on model load. Often the case, `inference` and `training` engines modifies the traditional tokenizers causing subtle and inaccurate output when `inference` performed on a platform that is disjointed from the `trainer`.
48
+
49
+ ## Install
50
+
51
+ ### PIP/UV
52
+
53
+ ```bash
54
+ pip install -v tokenicer
55
+ uv pip install -v tokenicer
56
+ ```
57
+
58
+ ### Install from source
59
+
60
+ ```bash
61
+ # clone repo
62
+ git clone https://github.com/ModelCloud/Tokencier.git && cd Tokenicer
63
+
64
+ # compile
65
+ pip install -v .
66
+ ```
67
+
68
+ ## Usage
69
+
70
+ * Replace all calls to `AutoTokenizer.from_pretrained()` with `Tokenizer.load()`: args are 100% compatible with `AutoTokenizer`
71
+
72
+ ```py
73
+ # Replace `AutoTokenizer.from_pretrained()`
74
+ # from tokenizer import AutoTokenizer
75
+ # tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-0.5B-Instruct')
76
+
77
+ # With `Tokenicer.load()`
78
+ from tokenicer import Tokenicer
79
+ tokenizer = Tokenicer.load('Qwen/Qwen2.5-0.5B-Instruct')
80
+
81
+ # That's it! Toke(n)icer has auto-fixed Qwen2.5-0.5B-Instruct's incorrect `pad_token`.
82
+ # Now this this model can be `trained` and `inferenced` correctly with `batch` and `masks`.
83
+ print(f"pad_token: `{tokenizer.pad_token}`")
84
+ ```
85
+
86
+ ## Citation
87
+
88
+ ```
89
+ @misc{gptqmodel,
90
+ author = {ModelCloud.ai and qubitium@modelcloud.ai},
91
+ title = {Toke(n)icer},
92
+ year = {2025},
93
+ publisher = {GitHub},
94
+ journal = {GitHub repository},
95
+ howpublished = {\url{https://github.com/modelcloud/tokenicer}},
96
+ note = {Contact: qubitium@modelcloud.ai}
97
+ }
@@ -0,0 +1,14 @@
1
+ LICENSE
2
+ MANIFEST.in
3
+ README.md
4
+ requirements.txt
5
+ setup.py
6
+ tokenicer/__init__.py
7
+ tokenicer/const.py
8
+ tokenicer/tokenicer.py
9
+ tokenicer/util.py
10
+ tokenicer.egg-info/PKG-INFO
11
+ tokenicer.egg-info/SOURCES.txt
12
+ tokenicer.egg-info/dependency_links.txt
13
+ tokenicer.egg-info/requires.txt
14
+ tokenicer.egg-info/top_level.txt
@@ -0,0 +1 @@
1
+ transformers
@@ -1,23 +0,0 @@
1
- Metadata-Version: 2.2
2
- Name: tokenicer
3
- Version: 0.0.1.dev0
4
- Summary: A nicer tokenizer
5
- Home-page: https://github.com/ModelCloud/Tokenicer
6
- Author: ModelCloud
7
- Author-email: qubitium@modelcloud.ai
8
- Classifier: Programming Language :: Python :: 3
9
- Classifier: License :: OSI Approved :: Apache Software License
10
- Classifier: Operating System :: OS Independent
11
- Requires-Python: >=3
12
- Description-Content-Type: text/markdown
13
- License-File: LICENSE
14
- Dynamic: author
15
- Dynamic: author-email
16
- Dynamic: classifier
17
- Dynamic: description
18
- Dynamic: description-content-type
19
- Dynamic: home-page
20
- Dynamic: requires-python
21
- Dynamic: summary
22
-
23
- # Tokenicer
@@ -1 +0,0 @@
1
- # Tokenicer
@@ -1,24 +0,0 @@
1
- from setuptools import setup, find_packages
2
- from pathlib import Path
3
-
4
- __version__ = "0.0.1-dev"
5
-
6
- setup(
7
- name="tokenicer",
8
- version=__version__,
9
- author="ModelCloud",
10
- author_email="qubitium@modelcloud.ai",
11
- description="A nicer tokenizer",
12
- long_description=(Path(__file__).parent / "README.md").read_text(encoding="UTF-8"),
13
- long_description_content_type="text/markdown",
14
- url="https://github.com/ModelCloud/Tokenicer",
15
- packages=find_packages(),
16
- install_requires=[],
17
- platform=["linux"],
18
- classifiers=[
19
- "Programming Language :: Python :: 3",
20
- "License :: OSI Approved :: Apache Software License",
21
- "Operating System :: OS Independent",
22
- ],
23
- python_requires=">=3",
24
- )
File without changes
@@ -1,23 +0,0 @@
1
- Metadata-Version: 2.2
2
- Name: tokenicer
3
- Version: 0.0.1.dev0
4
- Summary: A nicer tokenizer
5
- Home-page: https://github.com/ModelCloud/Tokenicer
6
- Author: ModelCloud
7
- Author-email: qubitium@modelcloud.ai
8
- Classifier: Programming Language :: Python :: 3
9
- Classifier: License :: OSI Approved :: Apache Software License
10
- Classifier: Operating System :: OS Independent
11
- Requires-Python: >=3
12
- Description-Content-Type: text/markdown
13
- License-File: LICENSE
14
- Dynamic: author
15
- Dynamic: author-email
16
- Dynamic: classifier
17
- Dynamic: description
18
- Dynamic: description-content-type
19
- Dynamic: home-page
20
- Dynamic: requires-python
21
- Dynamic: summary
22
-
23
- # Tokenicer
@@ -1,8 +0,0 @@
1
- LICENSE
2
- README.md
3
- setup.py
4
- tokenicer/__init__.py
5
- tokenicer.egg-info/PKG-INFO
6
- tokenicer.egg-info/SOURCES.txt
7
- tokenicer.egg-info/dependency_links.txt
8
- tokenicer.egg-info/top_level.txt
File without changes