e2k 0.2.2__tar.gz → 0.3.0__tar.gz

{e2k-0.2.2/src/e2k.egg-info → e2k-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: e2k
-Version: 0.2.2
+Version: 0.3.0
 Summary: A tool for automatic English to Katakana conversion
 Author-email: Patchethium <asguftuikh@gmail.com>
 Requires-Python: >=3.10
@@ -37,20 +37,31 @@ p2k = P2K()
 
 g2p = G2p()
 
-katakana = p2k(g2p("word"))
+word = "word"
+
+word = word.lower()
+
+# you can check the accepted symbols using
+in_table = c2k.in_table # `c2k` accepts lowercase characters, space and apostrophe
+in_table = p2k.in_table # `p2k` accepts the phonemes from the CMUdict and space
+# for output symbols
+out_table = c2k.out_table
+out_table = p2k.out_table
+
+katakana = p2k(g2p(word))
 
 print(katakana) # "ワード"
 
-# character directly to katakana
+# characters directly to katakana
 c2k = C2K()
 
-katakana = c2k("word")
+katakana = c2k(word)
 
 print(katakana) # "ワード"
 
 # decode strategy
 # greedy by default, top_k and top_p are available
-# TODO: beam searh
+# I tried implementing beam search but it barely differs from greedy
 c2k.set_decode_strategy("top_k", top_k=2) # low quality results, not recommended
 c2k.set_decode_strategy("top_p", top_p=0.6, t=0.8)
 # for further information, see
@@ -58,14 +69,17 @@ c2k.set_decode_strategy("top_p", top_p=0.6, t=0.8)
 print(katakana) # "ワード"
 ```
 
+> [!WARNING]
+> For any symbols not in the `in_table`, the model will ignore them and may produce unexpected results.
+
 ### Performance
 
 The BLEU score is calculated on a random subset with size of 10% of the dataset.
 
 | Model | BLEU Score ↑ |
 | ----- | ------------ |
-| P2K   | 0.87         |
-| C2K   | 0.91         |
+| P2K   | 0.89         |
+| C2K   | 0.92         |
 
 ## Dictionary
 
@@ -164,10 +178,10 @@ The model should be exported to `numpy` format for production use.
 
 ```bash
 # --p2k for phoneme to katakana, if not provided, it will be character to katakana
-# --fp16 for half precision, there's no reason not to use it
+# --fp32 for double precision, by default we use fp16 to save space
 # --output to specify the output file, in this project it's `model-{p2k/c2k}.npz`
 # --safetenors to use safe tensors, it's for easier binding in some languages
-python export.py --model /path/to/your/model.pth --p2k --fp16 --output /path/to/your/model.npz
+python export.py --model /path/to/your/model.pth --p2k --output /path/to/your/model.npz
 ```
 
 > [!Note]
@@ -175,13 +189,10 @@ python export.py --model /path/to/your/model.pth --p2k --fp16 --output /path/to/
 
 ## License
 
-The code is released under WTFPL, you can do WTF you want with it.
-
-The dictionary should follow the [Wikimedia's license](https://dumps.wikimedia.org/legal.html) and the [JMdict / EDICT's Copyright](https://www.edrdg.org/) license.
-
-In short, as long as you provide the attribution, you can use it for both commercial and non-commercial purposes.
-
-The model weights are released under the same license as the dictionary.
+ - The code is released under WTFPL, you can do WTF you want with it.
+ - The dictionary follows the [Wikimedia's license](https://dumps.wikimedia.org/legal.html) and the [JMdict / EDICT's Copyright](https://www.edrdg.org/) license.
+   - In short, they both fall into CC-BY-SA.
+   - The model weights are trained using the dictionary. I am not a lawyer, whether the machine learning weights is considered as a derivative work is up to you.
 
 ## Credits
 

{e2k-0.2.2 → e2k-0.3.0}/README.md

@@ -28,20 +28,31 @@ p2k = P2K()
 
 g2p = G2p()
 
-katakana = p2k(g2p("word"))
+word = "word"
+
+word = word.lower()
+
+# you can check the accepted symbols using
+in_table = c2k.in_table # `c2k` accepts lowercase characters, space and apostrophe
+in_table = p2k.in_table # `p2k` accepts the phonemes from the CMUdict and space
+# for output symbols
+out_table = c2k.out_table
+out_table = p2k.out_table
+
+katakana = p2k(g2p(word))
 
 print(katakana) # "ワード"
 
-# character directly to katakana
+# characters directly to katakana
 c2k = C2K()
 
-katakana = c2k("word")
+katakana = c2k(word)
 
 print(katakana) # "ワード"
 
 # decode strategy
 # greedy by default, top_k and top_p are available
-# TODO: beam searh
+# I tried implementing beam search but it barely differs from greedy
 c2k.set_decode_strategy("top_k", top_k=2) # low quality results, not recommended
 c2k.set_decode_strategy("top_p", top_p=0.6, t=0.8)
 # for further information, see
@@ -49,14 +60,17 @@ c2k.set_decode_strategy("top_p", top_p=0.6, t=0.8)
 print(katakana) # "ワード"
 ```
 
+> [!WARNING]
+> For any symbols not in the `in_table`, the model will ignore them and may produce unexpected results.
+
 ### Performance
 
 The BLEU score is calculated on a random subset with size of 10% of the dataset.
 
 | Model | BLEU Score ↑ |
 | ----- | ------------ |
-| P2K   | 0.87         |
-| C2K   | 0.91         |
+| P2K   | 0.89         |
+| C2K   | 0.92         |
 
 ## Dictionary
 
@@ -155,10 +169,10 @@ The model should be exported to `numpy` format for production use.
 
 ```bash
 # --p2k for phoneme to katakana, if not provided, it will be character to katakana
-# --fp16 for half precision, there's no reason not to use it
+# --fp32 for double precision, by default we use fp16 to save space
 # --output to specify the output file, in this project it's `model-{p2k/c2k}.npz`
 # --safetenors to use safe tensors, it's for easier binding in some languages
-python export.py --model /path/to/your/model.pth --p2k --fp16 --output /path/to/your/model.npz
+python export.py --model /path/to/your/model.pth --p2k --output /path/to/your/model.npz
 ```
 
 > [!Note]
@@ -166,13 +180,10 @@ python export.py --model /path/to/your/model.pth --p2k --fp16 --output /path/to/
 
 ## License
 
-The code is released under WTFPL, you can do WTF you want with it.
-
-The dictionary should follow the [Wikimedia's license](https://dumps.wikimedia.org/legal.html) and the [JMdict / EDICT's Copyright](https://www.edrdg.org/) license.
-
-In short, as long as you provide the attribution, you can use it for both commercial and non-commercial purposes.
-
-The model weights are released under the same license as the dictionary.
+ - The code is released under WTFPL, you can do WTF you want with it.
+ - The dictionary follows the [Wikimedia's license](https://dumps.wikimedia.org/legal.html) and the [JMdict / EDICT's Copyright](https://www.edrdg.org/) license.
+   - In short, they both fall into CC-BY-SA.
+   - The model weights are trained using the dictionary. I am not a lawyer, whether the machine learning weights is considered as a derivative work is up to you.
 
 ## Credits
 

{e2k-0.2.2 → e2k-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "e2k"
-version = "0.2.2"
+version = "0.3.0"
 description = "A tool for automatic English to Katakana conversion"
 readme = "README.md"
 authors = [{ name = "Patchethium", email = "asguftuikh@gmail.com" }]

{e2k-0.2.2 → e2k-0.3.0}/src/e2k/constants.py

@@ -1,4 +1,4 @@
-from string import ascii_letters
+from string import ascii_lowercase
 
 
 PAD_IDX = 0
@@ -165,7 +165,7 @@ en_phones = [
     "ZH",
 ]
 
-ascii_entries = ["<pad>", "<sos>", "<eos>"] + list(ascii_letters) + [" "]
+ascii_entries = ["<pad>", "<sos>", "<eos>"] + list(ascii_lowercase) + [" ", "'"]
 
 kanas = ["<pad>", "<sos>", "<eos>"] + kanas
-en_phones = ["<pad>", "<sos>", "<eos>"] + en_phones + [" "]
+en_phones = ["<pad>", "<sos>", "<eos>"] + en_phones + [" ", "'"]

{e2k-0.2.2 → e2k-0.3.0}/src/e2k/inference.py

@@ -314,7 +314,7 @@ if __name__ == "__main__":
     args = parser.parse_args()
     p2k = P2K()
     c2k = C2K()
-    word = "gutenberg"
+    word = "vordhosbn"
     phonemes = g2p(word)
     print(word)
     print("P2K: ", p2k(phonemes))

{e2k-0.2.2 → e2k-0.3.0/src/e2k.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: e2k
-Version: 0.2.2
+Version: 0.3.0
 Summary: A tool for automatic English to Katakana conversion
 Author-email: Patchethium <asguftuikh@gmail.com>
 Requires-Python: >=3.10
@@ -37,20 +37,31 @@ p2k = P2K()
 
 g2p = G2p()
 
-katakana = p2k(g2p("word"))
+word = "word"
+
+word = word.lower()
+
+# you can check the accepted symbols using
+in_table = c2k.in_table # `c2k` accepts lowercase characters, space and apostrophe
+in_table = p2k.in_table # `p2k` accepts the phonemes from the CMUdict and space
+# for output symbols
+out_table = c2k.out_table
+out_table = p2k.out_table
+
+katakana = p2k(g2p(word))
 
 print(katakana) # "ワード"
 
-# character directly to katakana
+# characters directly to katakana
 c2k = C2K()
 
-katakana = c2k("word")
+katakana = c2k(word)
 
 print(katakana) # "ワード"
 
 # decode strategy
 # greedy by default, top_k and top_p are available
-# TODO: beam searh
+# I tried implementing beam search but it barely differs from greedy
 c2k.set_decode_strategy("top_k", top_k=2) # low quality results, not recommended
 c2k.set_decode_strategy("top_p", top_p=0.6, t=0.8)
 # for further information, see
@@ -58,14 +69,17 @@ c2k.set_decode_strategy("top_p", top_p=0.6, t=0.8)
 print(katakana) # "ワード"
 ```
 
+> [!WARNING]
+> For any symbols not in the `in_table`, the model will ignore them and may produce unexpected results.
+
 ### Performance
 
 The BLEU score is calculated on a random subset with size of 10% of the dataset.
 
 | Model | BLEU Score ↑ |
 | ----- | ------------ |
-| P2K   | 0.87         |
-| C2K   | 0.91         |
+| P2K   | 0.89         |
+| C2K   | 0.92         |
 
 ## Dictionary
 
@@ -164,10 +178,10 @@ The model should be exported to `numpy` format for production use.
 
 ```bash
 # --p2k for phoneme to katakana, if not provided, it will be character to katakana
-# --fp16 for half precision, there's no reason not to use it
+# --fp32 for double precision, by default we use fp16 to save space
 # --output to specify the output file, in this project it's `model-{p2k/c2k}.npz`
 # --safetenors to use safe tensors, it's for easier binding in some languages
-python export.py --model /path/to/your/model.pth --p2k --fp16 --output /path/to/your/model.npz
+python export.py --model /path/to/your/model.pth --p2k --output /path/to/your/model.npz
 ```
 
 > [!Note]
@@ -175,13 +189,10 @@ python export.py --model /path/to/your/model.pth --p2k --fp16 --output /path/to/
 
 ## License
 
-The code is released under WTFPL, you can do WTF you want with it.
-
-The dictionary should follow the [Wikimedia's license](https://dumps.wikimedia.org/legal.html) and the [JMdict / EDICT's Copyright](https://www.edrdg.org/) license.
-
-In short, as long as you provide the attribution, you can use it for both commercial and non-commercial purposes.
-
-The model weights are released under the same license as the dictionary.
+ - The code is released under WTFPL, you can do WTF you want with it.
+ - The dictionary follows the [Wikimedia's license](https://dumps.wikimedia.org/legal.html) and the [JMdict / EDICT's Copyright](https://www.edrdg.org/) license.
+   - In short, they both fall into CC-BY-SA.
+   - The model weights are trained using the dictionary. I am not a lawyer, whether the machine learning weights is considered as a derivative work is up to you.
 
 ## Credits