UniTok 4.3.9__tar.gz → 4.4.1__tar.gz

{UniTok-4.3.9 → UniTok-4.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: UniTok
-Version: 4.3.9
+Version: 4.4.1
 Summary: Unified Tokenizer
 Home-page: https://github.com/Jyonn/UnifiedTokenizer
 Author: Jyonn Liu
@@ -10,6 +10,15 @@ Keywords: token,tokenizer,NLP,transformers,glove,bert,llama
 Platform: any
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: termplot==0.0.2
+Requires-Dist: tqdm
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: transformers
+Requires-Dist: oba
+Requires-Dist: prettytable
+Requires-Dist: rich
+Requires-Dist: fastparquet
 
 # UniTok V4
 
@@ -28,16 +37,18 @@ Please refer to [UniTok Handbook](https://unitok.qijiong.work) for more detailed
 
 ### Changes and Comparisons
 
+> After UniTok 4.4.0, `Job` is renamed to `Feature`. 
+
 | Feature                         | UniTok v3                                                   | UniTok v4                                           | Comments                                                                      |
 |---------------------------------|-------------------------------------------------------------|-----------------------------------------------------|-------------------------------------------------------------------------------|
 | `UniTok` class                  | Solely for tokenization                                     | Manages the entire preprocessing lifecycle          |                                                                               |
 | `UniDep` class                  | Data loading and combining                                  | Removed                                             | V4 combines the functionalities of `UniTok` and `UniDep` into a single class. |
-| `Column` class                  | Column name is for both the original and tokenized datasets | N/A                                                 | V4 introduces a `Job` class.                                                  |
-| `Job` class                     | N/A                                                         | Defines how a specific column should be tokenized   |                                                                               |
+| `Column` class                  | Column name is for both the original and tokenized datasets | N/A                                                 | V4 introduces a `Feature` class.                                              |
+| `Feature` class                 | N/A                                                         | Defines how a specific column should be tokenized   |                                                                               |
 | `Tokenizer` class               | Ambiguous return type definition                            | `return_list` parameter must be of type `bool`      |                                                                               |
 | `Tokenizer` class               | Only supports `BertTokenizer` for text processing           | Supports all Tokenizers in the transformers library | New `TransformersTokenizer` class                                             |
 | `analyse` method                | Supported                                                   | Not supported Currently                             |                                                                               |
-| `Meta` class                    | Only for human-friendly displaying                          | Manager for `Job`, `Tokenizer`, and `Vocab`         |                                                                               |
+| `Meta` class                    | Only for human-friendly displaying                          | Manager for `Feature`, `Tokenizer`, and `Vocab`     |                                                                               |
 | `unitok` command                | Visualization in the terminal                               | More colorful and detailed output                   |                                                                               |
 | `Vocab` class (unitok >= 4.1.0) | Save and load vocabulary using text files                   | Save and load vocabulary using pickle files         | Avoids issues with special characters in text files                           |
 
@@ -75,13 +86,13 @@ pip install unitok
 **Components**
 
 - UniTok: Manages the dataset preprocessing lifecycle.
-- Job: Defines how a specific column should be tokenized.
+- Feature: Defines how a specific column should be tokenized.
 - Tokenizer: Encodes data using various methods (e.g., BERT, splitting by delimiters).
 - Vocabulary: Stores and manages unique tokens across datasets.
 
-**Primary Key (key_job)**
+**Primary Key (key_feature)**
 
-The `key_job` acts as the primary key for operations like `getitem` and `union`, ensuring consistency across datasets.
+The `key_feature` acts as the primary key for operations like `getitem` and `union`, ensuring consistency across datasets.
 
 ## Usage Guide
 
@@ -113,9 +124,9 @@ interaction = pd.read_csv(
 )
 ```
 
-### Defining and Adding Jobs
+### Defining and Adding Features
 
-Define tokenization jobs for different columns:
+Define tokenization features for different columns:
 
 ```python
 from unitok import UniTok, Vocab
@@ -128,23 +139,23 @@ with UniTok() as item_ut:
     bert_tokenizer = BertTokenizer(vocab='bert')
     llama_tokenizer = TransformersTokenizer(vocab='llama', key='huggyllama/llama-7b')
 
-    item_ut.add_job(tokenizer=EntityTokenizer(vocab=item_vocab), column='nid', key=True)
-    item_ut.add_job(tokenizer=bert_tokenizer, column='title', name='title@bert', truncate=20)
-    item_ut.add_job(tokenizer=llama_tokenizer, column='title', name='title@llama', truncate=20)
-    item_ut.add_job(tokenizer=bert_tokenizer, column='abstract', name='abstract@bert', truncate=50)
-    item_ut.add_job(tokenizer=llama_tokenizer, column='abstract', name='abstract@llama', truncate=50)
-    item_ut.add_job(tokenizer=EntityTokenizer(vocab='category'), column='category')
-    item_ut.add_job(tokenizer=EntityTokenizer(vocab='subcategory'), column='subcategory')
+    item_ut.add_feature(tokenizer=EntityTokenizer(vocab=item_vocab), column='nid', key=True)
+    item_ut.add_feature(tokenizer=bert_tokenizer, column='title', name='title@bert', truncate=20)
+    item_ut.add_feature(tokenizer=llama_tokenizer, column='title', name='title@llama', truncate=20)
+    item_ut.add_feature(tokenizer=bert_tokenizer, column='abstract', name='abstract@bert', truncate=50)
+    item_ut.add_feature(tokenizer=llama_tokenizer, column='abstract', name='abstract@llama', truncate=50)
+    item_ut.add_feature(tokenizer=EntityTokenizer(vocab='category'), column='category')
+    item_ut.add_feature(tokenizer=EntityTokenizer(vocab='subcategory'), column='subcategory')
 
 with UniTok() as user_ut:
-    user_ut.add_job(tokenizer=EntityTokenizer(vocab=user_vocab), column='uid', key=True)
-    user_ut.add_job(tokenizer=SplitTokenizer(vocab=item_vocab, sep=','), column='history', truncate=30)
+    user_ut.add_feature(tokenizer=EntityTokenizer(vocab=user_vocab), column='uid', key=True)
+    user_ut.add_feature(tokenizer=SplitTokenizer(vocab=item_vocab, sep=','), column='history', truncate=30)
 
 with UniTok() as inter_ut:
-    inter_ut.add_index_job(name='index')
-    inter_ut.add_job(tokenizer=EntityTokenizer(vocab=user_vocab), column='uid')
-    inter_ut.add_job(tokenizer=EntityTokenizer(vocab=item_vocab), column='nid')
-    inter_ut.add_job(tokenizer=DigitTokenizer(vocab='click', vocab_size=2), column='click')
+    inter_ut.add_index_feature(name='index')
+    inter_ut.add_feature(tokenizer=EntityTokenizer(vocab=user_vocab), column='uid')
+    inter_ut.add_feature(tokenizer=EntityTokenizer(vocab=item_vocab), column='nid')
+    inter_ut.add_feature(tokenizer=DigitTokenizer(vocab='click', vocab_size=2), column='click')
 ```
 
 ### Tokenizing Data
@@ -190,7 +201,7 @@ UniTok (4beta)
 Sample Size: 10
 ID Column: nid
 
-                                                                                 Jobs                                                                                  
+                                                                                 Features                                                                                  
 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
 ┃ Tokenizer                            ┃     Tokenizer ID      ┃ Column Mapping                               ┃ Vocab                             ┃    Max Length     ┃
 ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
@@ -224,5 +235,3 @@ Our TODO list includes:
 ## License
 
 This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
-
-

{UniTok-4.3.9 → UniTok-4.4.1}/README.md

@@ -15,16 +15,18 @@ Please refer to [UniTok Handbook](https://unitok.qijiong.work) for more detailed
 
 ### Changes and Comparisons
 
+> After UniTok 4.4.0, `Job` is renamed to `Feature`. 
+
 | Feature                         | UniTok v3                                                   | UniTok v4                                           | Comments                                                                      |
 |---------------------------------|-------------------------------------------------------------|-----------------------------------------------------|-------------------------------------------------------------------------------|
 | `UniTok` class                  | Solely for tokenization                                     | Manages the entire preprocessing lifecycle          |                                                                               |
 | `UniDep` class                  | Data loading and combining                                  | Removed                                             | V4 combines the functionalities of `UniTok` and `UniDep` into a single class. |
-| `Column` class                  | Column name is for both the original and tokenized datasets | N/A                                                 | V4 introduces a `Job` class.                                                  |
-| `Job` class                     | N/A                                                         | Defines how a specific column should be tokenized   |                                                                               |
+| `Column` class                  | Column name is for both the original and tokenized datasets | N/A                                                 | V4 introduces a `Feature` class.                                              |
+| `Feature` class                 | N/A                                                         | Defines how a specific column should be tokenized   |                                                                               |
 | `Tokenizer` class               | Ambiguous return type definition                            | `return_list` parameter must be of type `bool`      |                                                                               |
 | `Tokenizer` class               | Only supports `BertTokenizer` for text processing           | Supports all Tokenizers in the transformers library | New `TransformersTokenizer` class                                             |
 | `analyse` method                | Supported                                                   | Not supported Currently                             |                                                                               |
-| `Meta` class                    | Only for human-friendly displaying                          | Manager for `Job`, `Tokenizer`, and `Vocab`         |                                                                               |
+| `Meta` class                    | Only for human-friendly displaying                          | Manager for `Feature`, `Tokenizer`, and `Vocab`     |                                                                               |
 | `unitok` command                | Visualization in the terminal                               | More colorful and detailed output                   |                                                                               |
 | `Vocab` class (unitok >= 4.1.0) | Save and load vocabulary using text files                   | Save and load vocabulary using pickle files         | Avoids issues with special characters in text files                           |
 
@@ -62,13 +64,13 @@ pip install unitok
 **Components**
 
 - UniTok: Manages the dataset preprocessing lifecycle.
-- Job: Defines how a specific column should be tokenized.
+- Feature: Defines how a specific column should be tokenized.
 - Tokenizer: Encodes data using various methods (e.g., BERT, splitting by delimiters).
 - Vocabulary: Stores and manages unique tokens across datasets.
 
-**Primary Key (key_job)**
+**Primary Key (key_feature)**
 
-The `key_job` acts as the primary key for operations like `getitem` and `union`, ensuring consistency across datasets.
+The `key_feature` acts as the primary key for operations like `getitem` and `union`, ensuring consistency across datasets.
 
 ## Usage Guide
 
@@ -100,9 +102,9 @@ interaction = pd.read_csv(
 )
 ```
 
-### Defining and Adding Jobs
+### Defining and Adding Features
 
-Define tokenization jobs for different columns:
+Define tokenization features for different columns:
 
 ```python
 from unitok import UniTok, Vocab
@@ -115,23 +117,23 @@ with UniTok() as item_ut:
     bert_tokenizer = BertTokenizer(vocab='bert')
     llama_tokenizer = TransformersTokenizer(vocab='llama', key='huggyllama/llama-7b')
 
-    item_ut.add_job(tokenizer=EntityTokenizer(vocab=item_vocab), column='nid', key=True)
-    item_ut.add_job(tokenizer=bert_tokenizer, column='title', name='title@bert', truncate=20)
-    item_ut.add_job(tokenizer=llama_tokenizer, column='title', name='title@llama', truncate=20)
-    item_ut.add_job(tokenizer=bert_tokenizer, column='abstract', name='abstract@bert', truncate=50)
-    item_ut.add_job(tokenizer=llama_tokenizer, column='abstract', name='abstract@llama', truncate=50)
-    item_ut.add_job(tokenizer=EntityTokenizer(vocab='category'), column='category')
-    item_ut.add_job(tokenizer=EntityTokenizer(vocab='subcategory'), column='subcategory')
+    item_ut.add_feature(tokenizer=EntityTokenizer(vocab=item_vocab), column='nid', key=True)
+    item_ut.add_feature(tokenizer=bert_tokenizer, column='title', name='title@bert', truncate=20)
+    item_ut.add_feature(tokenizer=llama_tokenizer, column='title', name='title@llama', truncate=20)
+    item_ut.add_feature(tokenizer=bert_tokenizer, column='abstract', name='abstract@bert', truncate=50)
+    item_ut.add_feature(tokenizer=llama_tokenizer, column='abstract', name='abstract@llama', truncate=50)
+    item_ut.add_feature(tokenizer=EntityTokenizer(vocab='category'), column='category')
+    item_ut.add_feature(tokenizer=EntityTokenizer(vocab='subcategory'), column='subcategory')
 
 with UniTok() as user_ut:
-    user_ut.add_job(tokenizer=EntityTokenizer(vocab=user_vocab), column='uid', key=True)
-    user_ut.add_job(tokenizer=SplitTokenizer(vocab=item_vocab, sep=','), column='history', truncate=30)
+    user_ut.add_feature(tokenizer=EntityTokenizer(vocab=user_vocab), column='uid', key=True)
+    user_ut.add_feature(tokenizer=SplitTokenizer(vocab=item_vocab, sep=','), column='history', truncate=30)
 
 with UniTok() as inter_ut:
-    inter_ut.add_index_job(name='index')
-    inter_ut.add_job(tokenizer=EntityTokenizer(vocab=user_vocab), column='uid')
-    inter_ut.add_job(tokenizer=EntityTokenizer(vocab=item_vocab), column='nid')
-    inter_ut.add_job(tokenizer=DigitTokenizer(vocab='click', vocab_size=2), column='click')
+    inter_ut.add_index_feature(name='index')
+    inter_ut.add_feature(tokenizer=EntityTokenizer(vocab=user_vocab), column='uid')
+    inter_ut.add_feature(tokenizer=EntityTokenizer(vocab=item_vocab), column='nid')
+    inter_ut.add_feature(tokenizer=DigitTokenizer(vocab='click', vocab_size=2), column='click')
 ```
 
 ### Tokenizing Data
@@ -177,7 +179,7 @@ UniTok (4beta)
 Sample Size: 10
 ID Column: nid
 
-                                                                                 Jobs                                                                                  
+                                                                                 Features                                                                                  
 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
 ┃ Tokenizer                            ┃     Tokenizer ID      ┃ Column Mapping                               ┃ Vocab                             ┃    Max Length     ┃
 ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩

{UniTok-4.3.9 → UniTok-4.4.1}/UniTok.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: UniTok
-Version: 4.3.9
+Version: 4.4.1
 Summary: Unified Tokenizer
 Home-page: https://github.com/Jyonn/UnifiedTokenizer
 Author: Jyonn Liu
@@ -10,6 +10,15 @@ Keywords: token,tokenizer,NLP,transformers,glove,bert,llama
 Platform: any
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: termplot==0.0.2
+Requires-Dist: tqdm
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: transformers
+Requires-Dist: oba
+Requires-Dist: prettytable
+Requires-Dist: rich
+Requires-Dist: fastparquet
 
 # UniTok V4
 
@@ -28,16 +37,18 @@ Please refer to [UniTok Handbook](https://unitok.qijiong.work) for more detailed
 
 ### Changes and Comparisons
 
+> After UniTok 4.4.0, `Job` is renamed to `Feature`. 
+
 | Feature                         | UniTok v3                                                   | UniTok v4                                           | Comments                                                                      |
 |---------------------------------|-------------------------------------------------------------|-----------------------------------------------------|-------------------------------------------------------------------------------|
 | `UniTok` class                  | Solely for tokenization                                     | Manages the entire preprocessing lifecycle          |                                                                               |
 | `UniDep` class                  | Data loading and combining                                  | Removed                                             | V4 combines the functionalities of `UniTok` and `UniDep` into a single class. |
-| `Column` class                  | Column name is for both the original and tokenized datasets | N/A                                                 | V4 introduces a `Job` class.                                                  |
-| `Job` class                     | N/A                                                         | Defines how a specific column should be tokenized   |                                                                               |
+| `Column` class                  | Column name is for both the original and tokenized datasets | N/A                                                 | V4 introduces a `Feature` class.                                              |
+| `Feature` class                 | N/A                                                         | Defines how a specific column should be tokenized   |                                                                               |
 | `Tokenizer` class               | Ambiguous return type definition                            | `return_list` parameter must be of type `bool`      |                                                                               |
 | `Tokenizer` class               | Only supports `BertTokenizer` for text processing           | Supports all Tokenizers in the transformers library | New `TransformersTokenizer` class                                             |
 | `analyse` method                | Supported                                                   | Not supported Currently                             |                                                                               |
-| `Meta` class                    | Only for human-friendly displaying                          | Manager for `Job`, `Tokenizer`, and `Vocab`         |                                                                               |
+| `Meta` class                    | Only for human-friendly displaying                          | Manager for `Feature`, `Tokenizer`, and `Vocab`     |                                                                               |
 | `unitok` command                | Visualization in the terminal                               | More colorful and detailed output                   |                                                                               |
 | `Vocab` class (unitok >= 4.1.0) | Save and load vocabulary using text files                   | Save and load vocabulary using pickle files         | Avoids issues with special characters in text files                           |
 
@@ -75,13 +86,13 @@ pip install unitok
 **Components**
 
 - UniTok: Manages the dataset preprocessing lifecycle.
-- Job: Defines how a specific column should be tokenized.
+- Feature: Defines how a specific column should be tokenized.
 - Tokenizer: Encodes data using various methods (e.g., BERT, splitting by delimiters).
 - Vocabulary: Stores and manages unique tokens across datasets.
 
-**Primary Key (key_job)**
+**Primary Key (key_feature)**
 
-The `key_job` acts as the primary key for operations like `getitem` and `union`, ensuring consistency across datasets.
+The `key_feature` acts as the primary key for operations like `getitem` and `union`, ensuring consistency across datasets.
 
 ## Usage Guide
 
@@ -113,9 +124,9 @@ interaction = pd.read_csv(
 )
 ```
 
-### Defining and Adding Jobs
+### Defining and Adding Features
 
-Define tokenization jobs for different columns:
+Define tokenization features for different columns:
 
 ```python
 from unitok import UniTok, Vocab
@@ -128,23 +139,23 @@ with UniTok() as item_ut:
     bert_tokenizer = BertTokenizer(vocab='bert')
     llama_tokenizer = TransformersTokenizer(vocab='llama', key='huggyllama/llama-7b')
 
-    item_ut.add_job(tokenizer=EntityTokenizer(vocab=item_vocab), column='nid', key=True)
-    item_ut.add_job(tokenizer=bert_tokenizer, column='title', name='title@bert', truncate=20)
-    item_ut.add_job(tokenizer=llama_tokenizer, column='title', name='title@llama', truncate=20)
-    item_ut.add_job(tokenizer=bert_tokenizer, column='abstract', name='abstract@bert', truncate=50)
-    item_ut.add_job(tokenizer=llama_tokenizer, column='abstract', name='abstract@llama', truncate=50)
-    item_ut.add_job(tokenizer=EntityTokenizer(vocab='category'), column='category')
-    item_ut.add_job(tokenizer=EntityTokenizer(vocab='subcategory'), column='subcategory')
+    item_ut.add_feature(tokenizer=EntityTokenizer(vocab=item_vocab), column='nid', key=True)
+    item_ut.add_feature(tokenizer=bert_tokenizer, column='title', name='title@bert', truncate=20)
+    item_ut.add_feature(tokenizer=llama_tokenizer, column='title', name='title@llama', truncate=20)
+    item_ut.add_feature(tokenizer=bert_tokenizer, column='abstract', name='abstract@bert', truncate=50)
+    item_ut.add_feature(tokenizer=llama_tokenizer, column='abstract', name='abstract@llama', truncate=50)
+    item_ut.add_feature(tokenizer=EntityTokenizer(vocab='category'), column='category')
+    item_ut.add_feature(tokenizer=EntityTokenizer(vocab='subcategory'), column='subcategory')
 
 with UniTok() as user_ut:
-    user_ut.add_job(tokenizer=EntityTokenizer(vocab=user_vocab), column='uid', key=True)
-    user_ut.add_job(tokenizer=SplitTokenizer(vocab=item_vocab, sep=','), column='history', truncate=30)
+    user_ut.add_feature(tokenizer=EntityTokenizer(vocab=user_vocab), column='uid', key=True)
+    user_ut.add_feature(tokenizer=SplitTokenizer(vocab=item_vocab, sep=','), column='history', truncate=30)
 
 with UniTok() as inter_ut:
-    inter_ut.add_index_job(name='index')
-    inter_ut.add_job(tokenizer=EntityTokenizer(vocab=user_vocab), column='uid')
-    inter_ut.add_job(tokenizer=EntityTokenizer(vocab=item_vocab), column='nid')
-    inter_ut.add_job(tokenizer=DigitTokenizer(vocab='click', vocab_size=2), column='click')
+    inter_ut.add_index_feature(name='index')
+    inter_ut.add_feature(tokenizer=EntityTokenizer(vocab=user_vocab), column='uid')
+    inter_ut.add_feature(tokenizer=EntityTokenizer(vocab=item_vocab), column='nid')
+    inter_ut.add_feature(tokenizer=DigitTokenizer(vocab='click', vocab_size=2), column='click')
 ```
 
 ### Tokenizing Data
@@ -190,7 +201,7 @@ UniTok (4beta)
 Sample Size: 10
 ID Column: nid
 
-                                                                                 Jobs                                                                                  
+                                                                                 Features                                                                                  
 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
 ┃ Tokenizer                            ┃     Tokenizer ID      ┃ Column Mapping                               ┃ Vocab                             ┃    Max Length     ┃
 ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
@@ -224,5 +235,3 @@ Our TODO list includes:
 ## License
 
 This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
-
-

{UniTok-4.3.9 → UniTok-4.4.1}/UniTok.egg-info/SOURCES.txt

@@ -31,17 +31,12 @@ UniTokv3/tok/split_tok.py
 UniTokv3/tok/tok.py
 unitok/__init__.py
 unitok/__main__.py
+unitok/feature.py
 unitok/job.py
 unitok/meta.py
 unitok/selector.py
 unitok/status.py
 unitok/unitok.py
-unitok.egg-info/PKG-INFO
-unitok.egg-info/SOURCES.txt
-unitok.egg-info/dependency_links.txt
-unitok.egg-info/entry_points.txt
-unitok.egg-info/requires.txt
-unitok.egg-info/top_level.txt
 unitok/tokenizer/__init__.py
 unitok/tokenizer/base_tokenizer.py
 unitok/tokenizer/digit_tokenizer.py
@@ -67,6 +62,7 @@ unitok/utils/hub/__init__.py
 unitok/utils/hub/hub.py
 unitok/utils/hub/param_hub.py
 unitok/utils/index_set/__init__.py
+unitok/utils/index_set/feature_set.py
 unitok/utils/index_set/index_set.py
 unitok/utils/index_set/job_set.py
 unitok/utils/index_set/tokenizer_set.py

{UniTok-4.3.9 → UniTok-4.4.1}/UniTok.egg-info/entry_points.txt

@@ -2,4 +2,3 @@
 unidep-upgrade-v4 = UniTokv3.__main__:upgrade
 unitok = unitok.__main__:main
 unitokv3 = UniTokv3.__main__:main
-

{UniTok-4.3.9 → UniTok-4.4.1}/UniTok.egg-info/requires.txt

@@ -6,3 +6,4 @@ transformers
 oba
 prettytable
 rich
+fastparquet

{UniTok-4.3.9 → UniTok-4.4.1}/UniTokv3/__main__.py

@@ -9,7 +9,7 @@ from rich.table import Table
 
 from UniTokv3 import UniDep, Meta, Vocab
 from unitok.vocabulary import Vocab as Vocabv4
-from unitok.job import Job as Jobv4
+from unitok.feature import Feature as Featurev4
 from unitok.tokenizer.unknown_tokenizer import UnknownTokenizer
 from unitok.unitok import UniTok as UniTokv4
 from unitok.meta import Meta as Metav4
@@ -127,7 +127,7 @@ def upgrade():
             ut.meta.vocabularies.add(vocab_beta)
 
             for col in voc.cols:
-                print(f'\tUpgrade job {col.name}')
+                print(f'\tUpgrade feature {col.name}')
                 col_data = data[col.name]
                 if not len(col_data):
                     print(f'\t\tWarning: empty column {col.name}, defaulting to an atom column')
@@ -149,7 +149,7 @@ def upgrade():
                     tokenizer_id='upgrade_' + col.name,
                     vocab=vocab_beta,
                 )
-                job = Jobv4(
+                feature = Featurev4(
                     name=col.name,
                     column=col.name,
                     tokenizer=tokenizer,
@@ -159,7 +159,7 @@ def upgrade():
                     max_len=max_len,
                 )
                 ut.meta.tokenizers.add(tokenizer)
-                ut.meta.jobs.add(job)
+                ut.meta.features.add(feature)
 
     ut.meta.save(path)
 

{UniTok-4.3.9 → UniTok-4.4.1}/setup.py

@@ -6,7 +6,7 @@ long_description = (this_directory / "README.md").read_text(encoding='utf8')
 
 setup(
     name='UniTok',
-    version='4.3.9',
+    version='4.4.1',
     keywords=['token', 'tokenizer', 'NLP', 'transformers', 'glove', 'bert', 'llama'],
     description='Unified Tokenizer',
     long_description=long_description,
@@ -25,7 +25,8 @@ setup(
         'transformers',
         'oba',
         'prettytable',
-        'rich'
+        'rich',
+        'fastparquet'
     ],
     entry_points={
         'console_scripts': [

{UniTok-4.3.9 → UniTok-4.4.1}/unitok/__init__.py

@@ -11,8 +11,9 @@ from unitok.tokenizer import TransformersTokenizer, BertTokenizer
 from unitok.tokenizer import SplitTokenizer, DigitTokenizer, DigitsTokenizer
 from unitok.tokenizer import GloVeTokenizer
 from unitok.job import Job, JobHub
+from unitok.feature import Feature, FeatureHub
 
-from unitok.utils.index_set import IndexSet, VocabSet, TokenizerSet, JobSet
+from unitok.utils.index_set import IndexSet, VocabSet, TokenizerSet, JobSet, FeatureSet
 
 from unitok.meta import Meta
 from unitok.status import Status
@@ -32,7 +33,8 @@ __all__ = [
     'SplitTokenizer', 'DigitTokenizer', 'DigitsTokenizer',
     'GloVeTokenizer',
     'Job', 'JobHub',
-    'IndexSet', 'VocabSet', 'TokenizerSet', 'JobSet',
+    'Feature', 'FeatureHub',
+    'IndexSet', 'VocabSet', 'TokenizerSet', 'JobSet', 'FeatureSet',
     'Meta',
     'Status',
     'UniTok',

{UniTok-4.3.9 → UniTok-4.4.1}/unitok/__main__.py

@@ -15,7 +15,7 @@ def integrate():
     parser.add_argument('--file', '-f', type=str, help='csv, tsv, parquet format data')
     parser.add_argument('--lib', type=str, default=None, help='custom tokenizer library')
     parser.add_argument('--column', '-c', type=str, help='column name to tokenize')
-    parser.add_argument('--name', '-n', type=str, help='job name and export column name')
+    parser.add_argument('--name', '-n', type=str, help='export feature name name')
     parser.add_argument('--vocab', '-v', type=str, default=None, help='vocabulary name')
     parser.add_argument('--tokenizer', '-t', type=str, default=None, help='tokenizer classname')
     parser.add_argument('--tokenizer_id', type=str, default=None, help='tokenizer id')
@@ -69,7 +69,7 @@ def integrate():
                 raise ValueError(f'Unknown tokenizer: {args.tokenizer}. Available tokenizers: {tokenizers.keys()}')
             tokenizer = tokenizers[args.tokenizer](vocab=args.vocab, **tokenizer_params)
 
-        ut.add_job(tokenizer=tokenizer, column=args.column, name=args.name, truncate=args.truncate)
+        ut.add_feature(tokenizer=tokenizer, column=args.column, name=args.name, truncate=args.truncate)
         ut.tokenize(df).save(args.path)
 
 
@@ -85,11 +85,11 @@ def summarize():
 def remove():
     parser = argparse.ArgumentParser()
     parser.add_argument('path', type=str, default='.', help='path to a unitok data directory')
-    parser.add_argument('--name', type=str, help='job name to remove')
+    parser.add_argument('--name', type=str, help='feature name to remove')
     args, _ = parser.parse_known_args()
 
     with UniTok.load(args.path) as ut:
-        ut.remove_job(args.name)
+        ut.remove_feature(args.name)
         ut.save(args.path)
 
 

UniTok-4.3.9/unitok/job.py → UniTok-4.4.1/unitok/feature.py

@@ -8,7 +8,7 @@ from unitok.utils import Symbols, Instance
 from unitok.utils.hub import Hub
 
 
-class Job:
+class Feature:
     def __init__(
             self,
             tokenizer: Union[BaseTokenizer, str],
@@ -35,7 +35,7 @@ class Job:
         self.max_len = max_len
         self.from_union = isinstance(self.tokenizer, UnionTokenizer)
 
-        JobHub.add(self)
+        FeatureHub.add(self)
         VocabHub.add(self.tokenizer.vocab)
 
     @property
@@ -48,12 +48,12 @@ class Job:
         for attr in attributes:
             params[attr] = kwargs[attr] if attr in kwargs else getattr(self, attr)
 
-        return Job(**params)
+        return Feature(**params)
 
     def __str__(self):
         if self.key:
-            return f'Job({self.column} => {self.name}) [PK]'
-        return f'Job({self.column} => {self.name})'
+            return f'Feature({self.column} => {self.name}) [PK]'
+        return f'Feature({self.column} => {self.name})'
 
     def __repr__(self):
         return str(self)
@@ -85,10 +85,10 @@ class Job:
         return slice(None)
 
 
-class JobHub(Hub[Job]):
+class FeatureHub(Hub[Feature]):
     _instance = Instance(compulsory_space=True)
 
     @classmethod
-    def add(cls, key, obj: Job = None):
+    def add(cls, key, obj: Feature = None):
         key, obj = key.name, key
         return super().add(key, obj)

UniTok-4.4.1/unitok/job.py

@@ -0,0 +1,11 @@
+import warnings
+from unitok.feature import Feature, FeatureHub
+
+
+class Job(Feature):
+    def __init__(self, **kwargs):
+        warnings.warn(f'`Job` class is deprecated, use `Feature`.', DeprecationWarning, stacklevel=2)
+        super().__init__(**kwargs)
+
+
+JobHub = FeatureHub