UniTok 4.3.0__tar.gz → 4.3.1__tar.gz

{UniTok-4.3.0 → UniTok-4.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: UniTok
-Version: 4.3.0
+Version: 4.3.1
 Summary: Unified Tokenizer
 Home-page: https://github.com/Jyonn/UnifiedTokenizer
 Author: Jyonn Liu
@@ -13,17 +13,16 @@ License-File: LICENSE
 
 # UniTok V4
 
-If you want to use the old version, please refer to [v3](README_v3.md) in Chinese.
+The documentation for v3, old version, can be found [here](README_v3.md) in Chinese.
 
 ## Overview
 
 [![PyPI version](https://badge.fury.io/py/unitok.svg)](https://badge.fury.io/py/unitok)
 
-Welcome to the UniTok documentation! 
+Welcome to the UniTok v4! 
 This library provides a unified preprocessing solution for machine learning datasets, handling diverse data types like text, categorical features, and numerical values. 
-It introduces **SQL-like** data table combinations and a modular workflow that transitions datasets through three states: `initialized`, `tokenized`, and `organized`.
 
-UniTok is designed to simplify preprocessing by offering reusable components such as tokenizers and vocabularies, making it flexible for various datasets and scenarios.
+Please refer to [UniTok Handbook](https://unitok.qijiong.work) for more detailed information.
 
 ## Road from V3 to V4
 

{UniTok-4.3.0 → UniTok-4.3.1}/README.md

@@ -1,16 +1,15 @@
 # UniTok V4
 
-If you want to use the old version, please refer to [v3](README_v3.md) in Chinese.
+The documentation for v3, old version, can be found [here](README_v3.md) in Chinese.
 
 ## Overview
 
 [![PyPI version](https://badge.fury.io/py/unitok.svg)](https://badge.fury.io/py/unitok)
 
-Welcome to the UniTok documentation! 
+Welcome to the UniTok v4! 
 This library provides a unified preprocessing solution for machine learning datasets, handling diverse data types like text, categorical features, and numerical values. 
-It introduces **SQL-like** data table combinations and a modular workflow that transitions datasets through three states: `initialized`, `tokenized`, and `organized`.
 
-UniTok is designed to simplify preprocessing by offering reusable components such as tokenizers and vocabularies, making it flexible for various datasets and scenarios.
+Please refer to [UniTok Handbook](https://unitok.qijiong.work) for more detailed information.
 
 ## Road from V3 to V4
 

{UniTok-4.3.0 → UniTok-4.3.1}/UniTok.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: UniTok
-Version: 4.3.0
+Version: 4.3.1
 Summary: Unified Tokenizer
 Home-page: https://github.com/Jyonn/UnifiedTokenizer
 Author: Jyonn Liu
@@ -13,17 +13,16 @@ License-File: LICENSE
 
 # UniTok V4
 
-If you want to use the old version, please refer to [v3](README_v3.md) in Chinese.
+The documentation for v3, old version, can be found [here](README_v3.md) in Chinese.
 
 ## Overview
 
 [![PyPI version](https://badge.fury.io/py/unitok.svg)](https://badge.fury.io/py/unitok)
 
-Welcome to the UniTok documentation! 
+Welcome to the UniTok v4! 
 This library provides a unified preprocessing solution for machine learning datasets, handling diverse data types like text, categorical features, and numerical values. 
-It introduces **SQL-like** data table combinations and a modular workflow that transitions datasets through three states: `initialized`, `tokenized`, and `organized`.
 
-UniTok is designed to simplify preprocessing by offering reusable components such as tokenizers and vocabularies, making it flexible for various datasets and scenarios.
+Please refer to [UniTok Handbook](https://unitok.qijiong.work) for more detailed information.
 
 ## Road from V3 to V4
 

{UniTok-4.3.0 → UniTok-4.3.1}/setup.py

@@ -6,7 +6,7 @@ long_description = (this_directory / "README.md").read_text(encoding='utf8')
 
 setup(
     name='UniTok',
-    version='4.3.0',
+    version='4.3.1',
     keywords=['token', 'tokenizer', 'NLP', 'transformers', 'glove', 'bert', 'llama'],
     description='Unified Tokenizer',
     long_description=long_description,

{UniTok-4.3.0 → UniTok-4.3.1}/unitok/unitok.py

@@ -51,7 +51,7 @@ class UniTok(Status):
         if self._union_type is None:
             self._union_type = union_type
         elif self._union_type != union_type:
-            raise ValueError(f'union type is already set: {self._union_type}')
+            raise ValueError(f'Union type is already set: {self._union_type}')
 
     @Status.require_not_initialized
     def init_indices(self):
@@ -340,6 +340,7 @@ class UniTok(Status):
             sample[job.name] = self.data[job.name][index]
         return sample
 
+    @Status.require_not_initialized
     def pack(self, index):
         if self.is_soft_union:
             return self._pack_soft_union(index)
@@ -393,10 +394,6 @@ class UniTok(Status):
             selector = Selector(self.meta, *selector)
         return selector(sample)
 
-    def get_sample_by_id(self, key_id):
-        index = self.key_job.tokenizer.vocab[key_id]
-        return self[index]
-
     def __len__(self):
         return len(self._legal_indices)