e2k 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (15) hide show
  1. e2k-0.1.0/LICENSE +13 -0
  2. e2k-0.1.0/PKG-INFO +147 -0
  3. e2k-0.1.0/README.md +138 -0
  4. e2k-0.1.0/pyproject.toml +40 -0
  5. e2k-0.1.0/setup.cfg +4 -0
  6. e2k-0.1.0/src/e2k/__init__.py +8 -0
  7. e2k-0.1.0/src/e2k/constants.py +171 -0
  8. e2k-0.1.0/src/e2k/inference.py +272 -0
  9. e2k-0.1.0/src/e2k/models/model-c2k.npz +0 -0
  10. e2k-0.1.0/src/e2k/models/model-p2k.npz +0 -0
  11. e2k-0.1.0/src/e2k.egg-info/PKG-INFO +147 -0
  12. e2k-0.1.0/src/e2k.egg-info/SOURCES.txt +13 -0
  13. e2k-0.1.0/src/e2k.egg-info/dependency_links.txt +1 -0
  14. e2k-0.1.0/src/e2k.egg-info/requires.txt +1 -0
  15. e2k-0.1.0/src/e2k.egg-info/top_level.txt +1 -0
e2k-0.1.0/LICENSE ADDED
@@ -0,0 +1,13 @@
1
+ DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
2
+ Version 2, December 2004
3
+
4
+ Copyright (C) 2004 Sam Hocevar <sam@hocevar.net>
5
+
6
+ Everyone is permitted to copy and distribute verbatim or modified
7
+ copies of this license document, and changing it is allowed as long
8
+ as the name is changed.
9
+
10
+ DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
11
+ TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
12
+
13
+ 0. You just DO WHAT THE FUCK YOU WANT TO.
e2k-0.1.0/PKG-INFO ADDED
@@ -0,0 +1,147 @@
1
+ Metadata-Version: 2.2
2
+ Name: e2k
3
+ Version: 0.1.0
4
+ Summary: A tool for automatic English to Katakana conversion
5
+ Author-email: Patchethium <asguftuikh@gmail.com>
6
+ Requires-Python: >=3.10
7
+ Description-Content-Type: text/markdown
8
+ Requires-Dist: numpy>=2.2.2
9
+
10
+ # English to Katakana Dictionary
11
+
12
+ It's an extraction of the Wikitionary's Japanese dictionary and `JMdict / EDICT` to create a dictionary of English words and their Katakana representation.
13
+
14
+ We also provide a GRU model to convert English to Katakana without a dictionary.
15
+
16
+ ```json
17
+ {"word": "word", "katakana": "ワード"}
18
+ {"word": "another", "katakana": "アナザー"}
19
+ {"word": "example", "katakana": "エグザンプル"}
20
+ ```
21
+
22
+ It's for the purpose of converting English words to Katakana and further use in Japanese Text-to-Speech applications or Japanese learning.
23
+
24
+ ## Re-creating the dictionary
25
+
26
+ The script has zero dependencies, as long as you have a Python 3 interpreter it should work.
27
+
28
+ ### Download the dictionary
29
+
30
+ Download the raw dump of the Japanese Wikitionary from https://kaikki.org/dictionary/rawdata.html, they kindly provide the parsed data in a JSONL format.
31
+
32
+ Look for the `Japanese ja-extract.jsonl.gz (compressed 37.5MB)` entry and download it. If you prefer command line, use
33
+
34
+ ```bash
35
+ curl -O https://kaikki.org/dictionary/downloads/ja/ja-extract.jsonl.gz
36
+ ```
37
+
38
+ ### Extract it
39
+
40
+ Extract it into `/vendor` folder.
41
+
42
+ On Linux, you can use
43
+
44
+ ```bash
45
+ gzip -d ja-extract.jsonl.gz
46
+ ```
47
+
48
+ ### Run the script
49
+
50
+ ```bash
51
+ python extract.py
52
+ # if you have another name for the file
53
+ python extract.py --path /path/to/your_file.jsonl
54
+ ```
55
+
56
+ And a `katakana_dict.jsonl` file will be created in the root folder.
57
+
58
+ ## E2K model
59
+
60
+ We also provide a seq2seq GRU model to convert English automatically to Katakana. It's trained on the aforementioned dictionary.
61
+
62
+ We provide 2 types of models, one converts phoneme to Katakana and one that converts character to Katakana. Choose the one that fits your use case.
63
+
64
+ ```bash
65
+ pip install e2k
66
+ ```
67
+
68
+ The weights are included in PyPI, you can simply use the model like this:
69
+
70
+ ```python
71
+ from e2k import P2K, C2K
72
+ from g2p_en import G2p # or `phonemizer`, as long as it outputs CMUdict phoneme
73
+
74
+ # cmudict phoneme to katakana
75
+ p2k = P2K()
76
+
77
+ g2p = G2p()
78
+
79
+ katakana = p2k(g2p("word"))
80
+
81
+ print(katakana) # "ワード"
82
+
83
+ # character directly to katakana
84
+ c2k = C2K()
85
+
86
+ # since it's an autoregressive model, you can set the decoding strategy
87
+ # to greedy, top-k or nucleus sampling (top-p)
88
+ # see https://huggingface.co/docs/transformers/en/generation_strategies
89
+ c2k.set_decoding_strategy("top-p", p=0.3)
90
+
91
+ katakana = c2k("word")
92
+
93
+ print(katakana) # "ワード"
94
+ ```
95
+
96
+ We rewrite the inference of GRU model in `numpy`, minimizing the dependencies to `numpy` only.
97
+
98
+ ### Benchmark
99
+
100
+ ```bash
101
+ # --p2k for phoneme to katakana, if not provided, it will be character to katakana
102
+ python eval.py --data ./vendor/katakana_dict.jsonl --model /path/to/your/model.pth --p2k
103
+ ```
104
+
105
+ | Model | BLEU Score |
106
+ | --------------------- | ---------- |
107
+ | Phoneme to Katakana | 0.85 |
108
+ | Character to Katakana | 0.90 |
109
+
110
+ ### Training the model
111
+
112
+ You'll need `torch` and `g2p_en`. After that, you can run the `train.py` script to train the model.
113
+
114
+ ```bash
115
+ python train.py --data ./vendor/kanji_dict.jsonl
116
+ ```
117
+
118
+ It takes around 10 minutes to train the model on a desktop CPU. The model will be saved as `model.pth` in the root folder.
119
+
120
+ Also, you'll need to either download the `kanji_dict.jsonl` from the releases or create it yourself using the `extract.py` script.
121
+
122
+ Be noted that the training script is not included in the PyPI package, you'll need to clone the repository to train the model.
123
+
124
+ ## Development
125
+
126
+ ### Install the dependencies
127
+
128
+ I use [`uv`](https://docs.astral.sh/uv/) to manage the dependencies and publish the package.
129
+
130
+ ```bash
131
+ uv sync
132
+ ```
133
+
134
+ ## License
135
+
136
+ The code is released under WTFPL.
137
+
138
+ The dictionary should follow the [Wikimedia's license](https://dumps.wikimedia.org/legal.html) and the [JMdict / EDICT's Copyright](https://www.edrdg.org/) license.
139
+
140
+ In short, as long as you provide the attribution, you can use it for both commercial and non-commercial purposes.
141
+
142
+ The model weights are released under the same license as the dictionary.
143
+
144
+ ## Credits
145
+
146
+ - [Wikitionary](https://www.wiktionary.org/)
147
+ - [JMdict / EDICT](http://www.edrdg.org/jmdict/edict.html)
e2k-0.1.0/README.md ADDED
@@ -0,0 +1,138 @@
1
+ # English to Katakana Dictionary
2
+
3
+ It's an extraction of the Wikitionary's Japanese dictionary and `JMdict / EDICT` to create a dictionary of English words and their Katakana representation.
4
+
5
+ We also provide a GRU model to convert English to Katakana without a dictionary.
6
+
7
+ ```json
8
+ {"word": "word", "katakana": "ワード"}
9
+ {"word": "another", "katakana": "アナザー"}
10
+ {"word": "example", "katakana": "エグザンプル"}
11
+ ```
12
+
13
+ It's for the purpose of converting English words to Katakana and further use in Japanese Text-to-Speech applications or Japanese learning.
14
+
15
+ ## Re-creating the dictionary
16
+
17
+ The script has zero dependencies, as long as you have a Python 3 interpreter it should work.
18
+
19
+ ### Download the dictionary
20
+
21
+ Download the raw dump of the Japanese Wikitionary from https://kaikki.org/dictionary/rawdata.html, they kindly provide the parsed data in a JSONL format.
22
+
23
+ Look for the `Japanese ja-extract.jsonl.gz (compressed 37.5MB)` entry and download it. If you prefer command line, use
24
+
25
+ ```bash
26
+ curl -O https://kaikki.org/dictionary/downloads/ja/ja-extract.jsonl.gz
27
+ ```
28
+
29
+ ### Extract it
30
+
31
+ Extract it into `/vendor` folder.
32
+
33
+ On Linux, you can use
34
+
35
+ ```bash
36
+ gzip -d ja-extract.jsonl.gz
37
+ ```
38
+
39
+ ### Run the script
40
+
41
+ ```bash
42
+ python extract.py
43
+ # if you have another name for the file
44
+ python extract.py --path /path/to/your_file.jsonl
45
+ ```
46
+
47
+ And a `katakana_dict.jsonl` file will be created in the root folder.
48
+
49
+ ## E2K model
50
+
51
+ We also provide a seq2seq GRU model to convert English automatically to Katakana. It's trained on the aforementioned dictionary.
52
+
53
+ We provide 2 types of models, one converts phoneme to Katakana and one that converts character to Katakana. Choose the one that fits your use case.
54
+
55
+ ```bash
56
+ pip install e2k
57
+ ```
58
+
59
+ The weights are included in PyPI, you can simply use the model like this:
60
+
61
+ ```python
62
+ from e2k import P2K, C2K
63
+ from g2p_en import G2p # or `phonemizer`, as long as it outputs CMUdict phoneme
64
+
65
+ # cmudict phoneme to katakana
66
+ p2k = P2K()
67
+
68
+ g2p = G2p()
69
+
70
+ katakana = p2k(g2p("word"))
71
+
72
+ print(katakana) # "ワード"
73
+
74
+ # character directly to katakana
75
+ c2k = C2K()
76
+
77
+ # since it's an autoregressive model, you can set the decoding strategy
78
+ # to greedy, top-k or nucleus sampling (top-p)
79
+ # see https://huggingface.co/docs/transformers/en/generation_strategies
80
+ c2k.set_decoding_strategy("top-p", p=0.3)
81
+
82
+ katakana = c2k("word")
83
+
84
+ print(katakana) # "ワード"
85
+ ```
86
+
87
+ We rewrite the inference of GRU model in `numpy`, minimizing the dependencies to `numpy` only.
88
+
89
+ ### Benchmark
90
+
91
+ ```bash
92
+ # --p2k for phoneme to katakana, if not provided, it will be character to katakana
93
+ python eval.py --data ./vendor/katakana_dict.jsonl --model /path/to/your/model.pth --p2k
94
+ ```
95
+
96
+ | Model | BLEU Score |
97
+ | --------------------- | ---------- |
98
+ | Phoneme to Katakana | 0.85 |
99
+ | Character to Katakana | 0.90 |
100
+
101
+ ### Training the model
102
+
103
+ You'll need `torch` and `g2p_en`. After that, you can run the `train.py` script to train the model.
104
+
105
+ ```bash
106
+ python train.py --data ./vendor/kanji_dict.jsonl
107
+ ```
108
+
109
+ It takes around 10 minutes to train the model on a desktop CPU. The model will be saved as `model.pth` in the root folder.
110
+
111
+ Also, you'll need to either download the `kanji_dict.jsonl` from the releases or create it yourself using the `extract.py` script.
112
+
113
+ Be noted that the training script is not included in the PyPI package, you'll need to clone the repository to train the model.
114
+
115
+ ## Development
116
+
117
+ ### Install the dependencies
118
+
119
+ I use [`uv`](https://docs.astral.sh/uv/) to manage the dependencies and publish the package.
120
+
121
+ ```bash
122
+ uv sync
123
+ ```
124
+
125
+ ## License
126
+
127
+ The code is released under WTFPL.
128
+
129
+ The dictionary should follow the [Wikimedia's license](https://dumps.wikimedia.org/legal.html) and the [JMdict / EDICT's Copyright](https://www.edrdg.org/) license.
130
+
131
+ In short, as long as you provide the attribution, you can use it for both commercial and non-commercial purposes.
132
+
133
+ The model weights are released under the same license as the dictionary.
134
+
135
+ ## Credits
136
+
137
+ - [Wikitionary](https://www.wiktionary.org/)
138
+ - [JMdict / EDICT](http://www.edrdg.org/jmdict/edict.html)
e2k-0.1.0/pyproject.toml ADDED
@@ -0,0 +1,40 @@
1
+ [project]
2
+ name = "e2k"
3
+ version = "0.1.0"
4
+ description = "A tool for automatic English to Katakana conversion"
5
+ readme = "README.md"
6
+ authors = [{ name = "Patchethium", email = "asguftuikh@gmail.com" }]
7
+ requires-python = ">=3.10"
8
+ dependencies = ["numpy>=2.2.2"]
9
+
10
+ [build-system]
11
+ requires = ["setuptools", "wheel"]
12
+ build-backend = "setuptools.build_meta"
13
+
14
+ [tool.setuptools]
15
+ license-files = []
16
+
17
+ [dependency-groups]
18
+ dev = [
19
+ "black>=25.1.0",
20
+ "build>=1.2.2.post1",
21
+ "g2p-en>=2.1.0",
22
+ "tensorboard>=2.18.0",
23
+ "torch>=2.6.0",
24
+ "torcheval>=0.0.7",
25
+ "tqdm>=4.67.1",
26
+ ]
27
+
28
+ [tool.uv.sources]
29
+ torch = [{ index = "pytorch-cpu" }]
30
+
31
+ [[tool.uv.index]]
32
+ name = "pytorch-cpu"
33
+ url = "https://download.pytorch.org/whl/cpu"
34
+ explicit = true
35
+
36
+ [tool.setuptools.packages.find]
37
+ where = ["src"]
38
+
39
+ [tool.setuptools.package-data]
40
+ e2k = ["models/*.npz"]
e2k-0.1.0/setup.cfg ADDED
@@ -0,0 +1,4 @@
1
+ [egg_info]
2
+ tag_build =
3
+ tag_date = 0
4
+
e2k-0.1.0/src/e2k/__init__.py ADDED
@@ -0,0 +1,8 @@
1
+ from .inference import P2K, C2K
2
+ from .constants import en_phones, kanas, ascii_entries
3
+ import os.path
4
+
5
+ # make internal modules invisible
6
+ __path__ = [os.path.dirname(__file__)]
7
+ # re-export the public API
8
+ __all__ = ["P2K", "C2K", "en_phones", "kanas", "ascii_entries"]
e2k-0.1.0/src/e2k/constants.py ADDED
@@ -0,0 +1,171 @@
1
+ from string import ascii_letters
2
+
3
+
4
+ PAD_IDX = 0
5
+ SOS_IDX = 1
6
+ EOS_IDX = 2
7
+
8
+ kanas = [
9
+ "ァ",
10
+ "ア",
11
+ "ィ",
12
+ "イ",
13
+ "ゥ",
14
+ "ウ",
15
+ "ェ",
16
+ "エ",
17
+ "ォ",
18
+ "オ",
19
+ "カ",
20
+ "ガ",
21
+ "キ",
22
+ "ギ",
23
+ "ク",
24
+ "グ",
25
+ "ケ",
26
+ "ゲ",
27
+ "コ",
28
+ "ゴ",
29
+ "サ",
30
+ "ザ",
31
+ "シ",
32
+ "ジ",
33
+ "ス",
34
+ "ズ",
35
+ "セ",
36
+ "ゼ",
37
+ "ソ",
38
+ "ゾ",
39
+ "タ",
40
+ "ダ",
41
+ "チ",
42
+ "ヂ",
43
+ "ッ",
44
+ "ツ",
45
+ "ヅ",
46
+ "テ",
47
+ "デ",
48
+ "ト",
49
+ "ド",
50
+ "ナ",
51
+ "ニ",
52
+ "ヌ",
53
+ "ネ",
54
+ "ノ",
55
+ "ハ",
56
+ "バ",
57
+ "パ",
58
+ "ヒ",
59
+ "ビ",
60
+ "ピ",
61
+ "フ",
62
+ "ブ",
63
+ "プ",
64
+ "ヘ",
65
+ "ベ",
66
+ "ペ",
67
+ "ホ",
68
+ "ボ",
69
+ "ポ",
70
+ "マ",
71
+ "ミ",
72
+ "ム",
73
+ "メ",
74
+ "モ",
75
+ "ャ",
76
+ "ヤ",
77
+ "ュ",
78
+ "ユ",
79
+ "ョ",
80
+ "ヨ",
81
+ "ラ",
82
+ "リ",
83
+ "ル",
84
+ "レ",
85
+ "ロ",
86
+ "ヮ",
87
+ "ワ",
88
+ "ヰ",
89
+ "ヱ",
90
+ "ヲ",
91
+ "ン",
92
+ "ヴ",
93
+ "ー",
94
+ ]
95
+
96
+ en_phones = [
97
+ "AA0",
98
+ "AA1",
99
+ "AA2",
100
+ "AE0",
101
+ "AE1",
102
+ "AE2",
103
+ "AH0",
104
+ "AH1",
105
+ "AH2",
106
+ "AO0",
107
+ "AO1",
108
+ "AO2",
109
+ "AW0",
110
+ "AW1",
111
+ "AW2",
112
+ "AY0",
113
+ "AY1",
114
+ "AY2",
115
+ "B",
116
+ "CH",
117
+ "D",
118
+ "DH",
119
+ "EH0",
120
+ "EH1",
121
+ "EH2",
122
+ "ER0",
123
+ "ER1",
124
+ "ER2",
125
+ "EY0",
126
+ "EY1",
127
+ "EY2",
128
+ "F",
129
+ "G",
130
+ "HH",
131
+ "IH0",
132
+ "IH1",
133
+ "IH2",
134
+ "IY0",
135
+ "IY1",
136
+ "IY2",
137
+ "JH",
138
+ "K",
139
+ "L",
140
+ "M",
141
+ "N",
142
+ "NG",
143
+ "OW0",
144
+ "OW1",
145
+ "OW2",
146
+ "OY0",
147
+ "OY1",
148
+ "OY2",
149
+ "P",
150
+ "R",
151
+ "S",
152
+ "SH",
153
+ "T",
154
+ "TH",
155
+ "UH0",
156
+ "UH1",
157
+ "UH2",
158
+ "UW0",
159
+ "UW1",
160
+ "UW2",
161
+ "V",
162
+ "W",
163
+ "Y",
164
+ "Z",
165
+ "ZH",
166
+ ]
167
+
168
+ ascii_entries = ["<pad>", "<sos>", "<eos>"] + list(ascii_letters) + [" "]
169
+
170
+ kanas = ["<pad>", "<sos>", "<eos>"] + kanas
171
+ en_phones = ["<pad>", "<sos>", "<eos>"] + en_phones + [" "]
e2k-0.1.0/src/e2k/inference.py ADDED
@@ -0,0 +1,272 @@
1
+ # Description: Inference functions for the E2K model in numpy
2
+ import numpy as np
3
+ from typing import Optional, Dict
4
+ import importlib.resources
5
+ from e2k.constants import SOS_IDX, EOS_IDX, en_phones, kanas, ascii_entries
6
+
7
+
8
+ class Linear:
9
+ def __init__(self, weight, bias):
10
+ self.weight = weight
11
+ self.bias = bias
12
+
13
+ def forward(self, x):
14
+ return np.matmul(x, self.weight.T) + self.bias
15
+
16
+
17
+ class Embedding:
18
+ def __init__(self, weight):
19
+ self.weight = weight
20
+
21
+ def forward(self, x):
22
+ return self.weight[x]
23
+
24
+
25
+ def sigmoid(x):
26
+ return 1 / (1 + np.exp(-x))
27
+
28
+
29
+ def tanh(x):
30
+ return np.tanh(x)
31
+
32
+
33
+ class GRUCell:
34
+ def __init__(self, weight_ih, weight_hh, bias_ih, bias_hh):
35
+ self.ih = Linear(weight_ih, bias_ih)
36
+ self.hh = Linear(weight_hh, bias_hh)
37
+
38
+ def forward(self, x: np.ndarray, h: Optional[np.ndarray] = None):
39
+ """
40
+ x: [D]
41
+ h: [D]
42
+ """
43
+ if h is None:
44
+ h = np.zeros(
45
+ [
46
+ self.hh.weight.shape[-1],
47
+ ]
48
+ )
49
+ rzn_ih = self.ih.forward(x)
50
+ rzn_hh = self.hh.forward(h)
51
+
52
+ rz_ih, n_ih = (
53
+ rzn_ih[: rzn_ih.shape[-1] * 2 // 3],
54
+ rzn_ih[rzn_ih.shape[-1] * 2 // 3 :],
55
+ )
56
+ rz_hh, n_hh = (
57
+ rzn_hh[: rzn_hh.shape[-1] * 2 // 3],
58
+ rzn_hh[rzn_hh.shape[-1] * 2 // 3 :],
59
+ )
60
+
61
+ rz = sigmoid(rz_ih + rz_hh)
62
+ r, z = np.split(rz, 2, axis=-1)
63
+
64
+ n = np.tanh(n_ih + r * n_hh)
65
+ h = (1 - z) * n + z * h
66
+
67
+ return h
68
+
69
+
70
+ class GRU:
71
+ def __init__(self, cell: GRUCell, reverse: bool = False):
72
+ self.cell = cell
73
+ self.reverse = reverse
74
+
75
+ def forward(self, x, h: Optional[np.ndarray] = None):
76
+ """
77
+ x: [T, D], unbatched
78
+ """
79
+ if self.reverse:
80
+ x = np.flip(x, axis=0)
81
+ outputs = []
82
+ for i in range(x.shape[0]):
83
+ h = self.cell.forward(x[i], h)
84
+ outputs.append(h)
85
+ outputs = np.stack(outputs)
86
+ if self.reverse:
87
+ outputs = np.flip(outputs, axis=0)
88
+ return outputs, h
89
+
90
+
91
+ class MHA:
92
+ """
93
+ Multi-head attention
94
+ """
95
+
96
+ def __init__(
97
+ self,
98
+ in_proj_weight,
99
+ in_proj_bias,
100
+ out_proj_weight,
101
+ out_proj_bias,
102
+ num_heads: int,
103
+ ):
104
+ [q_w, k_w, v_w] = np.split(in_proj_weight, 3, axis=0)
105
+ [q_b, k_b, v_b] = np.split(in_proj_bias, 3, axis=0)
106
+ self.dim = q_w.shape[-1]
107
+ self.q_proj = Linear(q_w, q_b)
108
+ self.k_proj = Linear(k_w, k_b)
109
+ self.v_proj = Linear(v_w, v_b)
110
+ self.o_proj = Linear(out_proj_weight, out_proj_bias)
111
+ self.num_heads = num_heads
112
+ self.d_heads = self.dim // num_heads
113
+ self.scale = np.sqrt(self.dim)
114
+
115
+ def forward(self, q: np.ndarray, k: np.ndarray, v: np.ndarray):
116
+ """
117
+ q: [T, D]
118
+ k: [T, D]
119
+ v: [T, D]
120
+ """
121
+ q = self.q_proj.forward(q)
122
+ k = self.k_proj.forward(k)
123
+ v = self.v_proj.forward(v)
124
+ q = np.split(q, self.num_heads, axis=-1)
125
+ q = np.stack(q, axis=0)
126
+ k = np.split(k, self.num_heads, axis=-1)
127
+ k = np.stack(k, axis=0)
128
+ v = np.split(v, self.num_heads, axis=-1)
129
+ v = np.stack(v, axis=0)
130
+ attn = np.matmul(q, np.transpose(k, (0, 2, 1)))
131
+ attn = attn / self.scale
132
+ attn = np.exp(attn)
133
+ attn = attn / attn.sum(axis=-1, keepdims=True)
134
+ o = np.matmul(attn, v)
135
+ o = np.transpose(o, (1, 0, 2))
136
+ o = o.reshape([o.shape[0], -1])
137
+ return self.o_proj.forward(o)
138
+
139
+
140
+ class S2S:
141
+ def __init__(self, weights: Dict[str, np.ndarray], max_len: int = 16):
142
+ self.e_emb = Embedding(weights["e_emb.weight"])
143
+ self.k_emb = Embedding(weights["k_emb.weight"])
144
+ self.encoder = GRU(
145
+ GRUCell(
146
+ weights["encoder.weight_ih_l0"],
147
+ weights["encoder.weight_hh_l0"],
148
+ weights["encoder.bias_ih_l0"],
149
+ weights["encoder.bias_hh_l0"],
150
+ )
151
+ )
152
+ self.encoder_reverse = GRU(
153
+ GRUCell(
154
+ weights["encoder.weight_ih_l0_reverse"],
155
+ weights["encoder.weight_hh_l0_reverse"],
156
+ weights["encoder.bias_ih_l0_reverse"],
157
+ weights["encoder.bias_hh_l0_reverse"],
158
+ ),
159
+ reverse=True,
160
+ )
161
+ self.encoder_fc = Linear(
162
+ weights["encoder_fc.0.weight"], weights["encoder_fc.0.bias"]
163
+ )
164
+ self.pre_decoder = GRU(
165
+ GRUCell(
166
+ weights["pre_decoder.weight_ih_l0"],
167
+ weights["pre_decoder.weight_hh_l0"],
168
+ weights["pre_decoder.bias_ih_l0"],
169
+ weights["pre_decoder.bias_hh_l0"],
170
+ )
171
+ )
172
+ self.post_decoder = GRU(
173
+ GRUCell(
174
+ weights["post_decoder.weight_ih_l0"],
175
+ weights["post_decoder.weight_hh_l0"],
176
+ weights["post_decoder.bias_ih_l0"],
177
+ weights["post_decoder.bias_hh_l0"],
178
+ )
179
+ )
180
+ self.attn = MHA(
181
+ weights["attn.in_proj_weight"],
182
+ weights["attn.in_proj_bias"],
183
+ weights["attn.out_proj.weight"],
184
+ weights["attn.out_proj.bias"],
185
+ 4,
186
+ )
187
+ self.fc = Linear(weights["fc.weight"], weights["fc.bias"])
188
+ self.max_len = max_len
189
+
190
+ def greedy(self, step_dec: np.ndarray):
191
+ """
192
+ step_dec: [N]
193
+ """
194
+ return np.argmax(step_dec, axis=-1)
195
+
196
+ def forward(self, src):
197
+ """
198
+ In numpy, only inference is supported.
199
+ """
200
+ e_emb = self.e_emb.forward(src)
201
+ enc_out, _ = self.encoder.forward(e_emb)
202
+ enc_out_rev, _ = self.encoder_reverse.forward(e_emb)
203
+ enc_out = np.concatenate([enc_out, enc_out_rev], axis=-1)
204
+ enc_out = self.encoder_fc.forward(enc_out)
205
+ enc_out = tanh(enc_out)
206
+ res = [SOS_IDX]
207
+ h1 = None
208
+ h2 = None
209
+ for _ in range(self.max_len):
210
+ dec_emb = self.k_emb.forward(np.array([res[-1]]))
211
+ dec_out, h1 = self.pre_decoder.forward(dec_emb, h1)
212
+ attn_out = self.attn.forward(dec_out, enc_out, enc_out)
213
+ x = np.concatenate([dec_out, attn_out], axis=-1)
214
+ x, h2 = self.post_decoder.forward(x, h2)
215
+ x = self.fc.forward(x)
216
+ x = x[0]
217
+ res.append(self.greedy(x))
218
+ if res[-1] == EOS_IDX:
219
+ break
220
+ return res
221
+
222
+
223
+ class BaseE2K:
224
+ def __init__(self):
225
+ self.s2s = None
226
+ self.in_table = None
227
+ self.out_table = None
228
+ raise NotImplementedError
229
+
230
+ def __call__(self, src: str) -> str:
231
+ src = list(filter(lambda x: x in self.in_table, src))
232
+ src = [self.in_table[c] for c in src]
233
+ src = [SOS_IDX] + src + [EOS_IDX]
234
+ src = np.array(src)
235
+ tgt = self.s2s.forward(src)
236
+ tgt = [int(v) for v in tgt[1:-1]]
237
+ tgt = [self.out_table[c] for c in tgt]
238
+ return "".join(tgt)
239
+
240
+ def get_weight_path(filename) -> str:
241
+ return importlib.resources.files("e2k.models").joinpath(filename)
242
+
243
+ class P2K(BaseE2K):
244
+ def __init__(self, max_len: int = 16):
245
+ weights = np.load(get_weight_path("model-p2k.npz"), allow_pickle=True)
246
+ self.s2s = S2S(weights, max_len)
247
+ self.in_table = {c: i for i, c in enumerate(en_phones)}
248
+ self.out_table = {i: c for i, c in enumerate(kanas)}
249
+
250
+
251
+ class C2K(BaseE2K):
252
+ def __init__(self, max_len: int = 16):
253
+ weights = np.load(get_weight_path("model-c2k.npz"), allow_pickle=True)
254
+ self.s2s = S2S(weights, max_len)
255
+ self.in_table = {c: i for i, c in enumerate(ascii_entries)}
256
+ self.out_table = {i: c for i, c in enumerate(kanas)}
257
+
258
+
259
+ if __name__ == "__main__":
260
+ import argparse
261
+ from g2p_en import G2p
262
+
263
+ g2p = G2p()
264
+ parser = argparse.ArgumentParser()
265
+ args = parser.parse_args()
266
+ p2k = P2K()
267
+ c2k = C2K()
268
+ word = "voicevox"
269
+ phonemes = g2p(word)
270
+ print(word)
271
+ print("P2K: ", p2k(phonemes)) # ザワールド
272
+ print("C2K: ", c2k(word)) # ザワールド
e2k-0.1.0/src/e2k/models/model-c2k.npz ADDED
Binary file
e2k-0.1.0/src/e2k/models/model-p2k.npz ADDED
Binary file
e2k-0.1.0/src/e2k.egg-info/PKG-INFO ADDED
@@ -0,0 +1,147 @@
1
+ Metadata-Version: 2.2
2
+ Name: e2k
3
+ Version: 0.1.0
4
+ Summary: A tool for automatic English to Katakana conversion
5
+ Author-email: Patchethium <asguftuikh@gmail.com>
6
+ Requires-Python: >=3.10
7
+ Description-Content-Type: text/markdown
8
+ Requires-Dist: numpy>=2.2.2
9
+
10
+ # English to Katakana Dictionary
11
+
12
+ It's an extraction of the Wikitionary's Japanese dictionary and `JMdict / EDICT` to create a dictionary of English words and their Katakana representation.
13
+
14
+ We also provide a GRU model to convert English to Katakana without a dictionary.
15
+
16
+ ```json
17
+ {"word": "word", "katakana": "ワード"}
18
+ {"word": "another", "katakana": "アナザー"}
19
+ {"word": "example", "katakana": "エグザンプル"}
20
+ ```
21
+
22
+ It's for the purpose of converting English words to Katakana and further use in Japanese Text-to-Speech applications or Japanese learning.
23
+
24
+ ## Re-creating the dictionary
25
+
26
+ The script has zero dependencies, as long as you have a Python 3 interpreter it should work.
27
+
28
+ ### Download the dictionary
29
+
30
+ Download the raw dump of the Japanese Wikitionary from https://kaikki.org/dictionary/rawdata.html, they kindly provide the parsed data in a JSONL format.
31
+
32
+ Look for the `Japanese ja-extract.jsonl.gz (compressed 37.5MB)` entry and download it. If you prefer command line, use
33
+
34
+ ```bash
35
+ curl -O https://kaikki.org/dictionary/downloads/ja/ja-extract.jsonl.gz
36
+ ```
37
+
38
+ ### Extract it
39
+
40
+ Extract it into `/vendor` folder.
41
+
42
+ On Linux, you can use
43
+
44
+ ```bash
45
+ gzip -d ja-extract.jsonl.gz
46
+ ```
47
+
48
+ ### Run the script
49
+
50
+ ```bash
51
+ python extract.py
52
+ # if you have another name for the file
53
+ python extract.py --path /path/to/your_file.jsonl
54
+ ```
55
+
56
+ And a `katakana_dict.jsonl` file will be created in the root folder.
57
+
58
+ ## E2K model
59
+
60
+ We also provide a seq2seq GRU model to convert English automatically to Katakana. It's trained on the aforementioned dictionary.
61
+
62
+ We provide 2 types of models, one converts phoneme to Katakana and one that converts character to Katakana. Choose the one that fits your use case.
63
+
64
+ ```bash
65
+ pip install e2k
66
+ ```
67
+
68
+ The weights are included in PyPI, you can simply use the model like this:
69
+
70
+ ```python
71
+ from e2k import P2K, C2K
72
+ from g2p_en import G2p # or `phonemizer`, as long as it outputs CMUdict phoneme
73
+
74
+ # cmudict phoneme to katakana
75
+ p2k = P2K()
76
+
77
+ g2p = G2p()
78
+
79
+ katakana = p2k(g2p("word"))
80
+
81
+ print(katakana) # "ワード"
82
+
83
+ # character directly to katakana
84
+ c2k = C2K()
85
+
86
+ # since it's an autoregressive model, you can set the decoding strategy
87
+ # to greedy, top-k or nucleus sampling (top-p)
88
+ # see https://huggingface.co/docs/transformers/en/generation_strategies
89
+ c2k.set_decoding_strategy("top-p", p=0.3)
90
+
91
+ katakana = c2k("word")
92
+
93
+ print(katakana) # "ワード"
94
+ ```
95
+
96
+ We rewrite the inference of GRU model in `numpy`, minimizing the dependencies to `numpy` only.
97
+
98
+ ### Benchmark
99
+
100
+ ```bash
101
+ # --p2k for phoneme to katakana, if not provided, it will be character to katakana
102
+ python eval.py --data ./vendor/katakana_dict.jsonl --model /path/to/your/model.pth --p2k
103
+ ```
104
+
105
+ | Model | BLEU Score |
106
+ | --------------------- | ---------- |
107
+ | Phoneme to Katakana | 0.85 |
108
+ | Character to Katakana | 0.90 |
109
+
110
+ ### Training the model
111
+
112
+ You'll need `torch` and `g2p_en`. After that, you can run the `train.py` script to train the model.
113
+
114
+ ```bash
115
+ python train.py --data ./vendor/kanji_dict.jsonl
116
+ ```
117
+
118
+ It takes around 10 minutes to train the model on a desktop CPU. The model will be saved as `model.pth` in the root folder.
119
+
120
+ Also, you'll need to either download the `kanji_dict.jsonl` from the releases or create it yourself using the `extract.py` script.
121
+
122
+ Be noted that the training script is not included in the PyPI package, you'll need to clone the repository to train the model.
123
+
124
+ ## Development
125
+
126
+ ### Install the dependencies
127
+
128
+ I use [`uv`](https://docs.astral.sh/uv/) to manage the dependencies and publish the package.
129
+
130
+ ```bash
131
+ uv sync
132
+ ```
133
+
134
+ ## License
135
+
136
+ The code is released under WTFPL.
137
+
138
+ The dictionary should follow the [Wikimedia's license](https://dumps.wikimedia.org/legal.html) and the [JMdict / EDICT's Copyright](https://www.edrdg.org/) license.
139
+
140
+ In short, as long as you provide the attribution, you can use it for both commercial and non-commercial purposes.
141
+
142
+ The model weights are released under the same license as the dictionary.
143
+
144
+ ## Credits
145
+
146
+ - [Wikitionary](https://www.wiktionary.org/)
147
+ - [JMdict / EDICT](http://www.edrdg.org/jmdict/edict.html)
e2k-0.1.0/src/e2k.egg-info/SOURCES.txt ADDED
@@ -0,0 +1,13 @@
1
+ LICENSE
2
+ README.md
3
+ pyproject.toml
4
+ src/e2k/__init__.py
5
+ src/e2k/constants.py
6
+ src/e2k/inference.py
7
+ src/e2k.egg-info/PKG-INFO
8
+ src/e2k.egg-info/SOURCES.txt
9
+ src/e2k.egg-info/dependency_links.txt
10
+ src/e2k.egg-info/requires.txt
11
+ src/e2k.egg-info/top_level.txt
12
+ src/e2k/models/model-c2k.npz
13
+ src/e2k/models/model-p2k.npz
e2k-0.1.0/src/e2k.egg-info/requires.txt ADDED
@@ -0,0 +1 @@
1
+ numpy>=2.2.2
e2k-0.1.0/src/e2k.egg-info/top_level.txt ADDED
@@ -0,0 +1 @@
1
+ e2k