voxnorm 0.1.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- voxnorm/__init__.py +145 -0
- voxnorm/cn.py +40 -0
- voxnorm/en.py +118 -0
- voxnorm/ja.py +89 -0
- voxnorm/ko.py +103 -0
- voxnorm/py.typed +0 -0
- voxnorm/tokens.py +91 -0
- voxnorm/zh.py +157 -0
- voxnorm-0.1.0.dist-info/METADATA +105 -0
- voxnorm-0.1.0.dist-info/RECORD +12 -0
- voxnorm-0.1.0.dist-info/WHEEL +4 -0
- voxnorm-0.1.0.dist-info/licenses/LICENSE +201 -0
voxnorm/__init__.py
ADDED
|
@@ -0,0 +1,145 @@
|
|
|
1
|
+
"""voxnorm: written forms into spoken forms, so a TTS voice reads them the way a person would.
|
|
2
|
+
|
|
3
|
+
Language models and databases write for the eye: `12:00`, `NT$1,200`,
|
|
4
|
+
`0912-345-678`. A small TTS frontend handed that text verbatim has to guess,
|
|
5
|
+
and guesses badly -- digits read in the wrong language, a phone number read as
|
|
6
|
+
one enormous quantity. voxnorm rewrites the written forms into what a speaker
|
|
7
|
+
would actually say (`十二點`, `新台幣一千兩百元`, `zero nine one two...`) and
|
|
8
|
+
leaves everything else character-for-character alone.
|
|
9
|
+
|
|
10
|
+
The shape is two layers, split where languages stop sharing:
|
|
11
|
+
|
|
12
|
+
- **Detection is language-neutral** (`tokens.py`). `14:30` and `50%` are the
|
|
13
|
+
same written form in any supported language, so one scanner finds the spans
|
|
14
|
+
and names their kind.
|
|
15
|
+
- **Verbalisation is per-language** (`zh.py`, `cn.py`, `en.py`, `ja.py`,
|
|
16
|
+
`ko.py`, one module each, registered in `_VERBALIZERS`). A new language is a new module
|
|
17
|
+
and one dict entry; the scanner does not change. English, Japanese and
|
|
18
|
+
Korean ride on num2words for their number words, each adding the local
|
|
19
|
+
readings the library cannot know (a Japanese year without the era default,
|
|
20
|
+
Korean native-numeral hours); Chinese is spelled out here, because the
|
|
21
|
+
兩/二 reading choice (一千兩百, never 一千二百 in zh-TW speech) is exactly
|
|
22
|
+
what no general-purpose converter gets right, and the Simplified variant
|
|
23
|
+
(`cn.py`) re-spells that one output vocabulary rather than re-deriving it.
|
|
24
|
+
|
|
25
|
+
Why not an FST toolkit: the established ones (nemo_text_processing,
|
|
26
|
+
WeTextProcessing) were measured before this package was written. Both depend
|
|
27
|
+
on pynini, which publishes no aarch64 wheel at all -- on an ARM box they need
|
|
28
|
+
OpenFst compiled from source and a permanently vendored `libfst.so` -- and
|
|
29
|
+
both mis-read the very classes this package exists for (phone numbers as
|
|
30
|
+
quantities, room numbers as dates, and in one case force-simplifying every
|
|
31
|
+
Traditional character in the sentence). The classes a spoken product actually
|
|
32
|
+
needs are a bounded set; implementing them directly costs ~3 MB of pure
|
|
33
|
+
Python and stays debuggable.
|
|
34
|
+
|
|
35
|
+
The charter is **structural only, never guess**. A form whose reading is
|
|
36
|
+
ambiguous does not convert: `1/2` the fraction and `8/17` the date are the
|
|
37
|
+
same slash string, so no slash form converts at all. What is not converted is
|
|
38
|
+
simply spoken as written, which is where it started.
|
|
39
|
+
|
|
40
|
+
Language is decided per call: an explicit `lang` wins, otherwise script
|
|
41
|
+
evidence decides -- kana anywhere means Japanese, hangul Korean, then one Han
|
|
42
|
+
character means Chinese, else English. Kana is checked before Han because
|
|
43
|
+
Japanese text carries kanji too (which also means one katakana loanword
|
|
44
|
+
inside an otherwise-Chinese sentence flips that sentence to the Japanese
|
|
45
|
+
reading -- pass `lang` where such text is expected); the residual is a Japanese sentence written
|
|
46
|
+
entirely in kanji, indistinguishable from Chinese by script, which lands on
|
|
47
|
+
the Chinese reading. That is as far as auto-detection can honestly go --
|
|
48
|
+
Latin-script languages are indistinguishable without a real detector -- so
|
|
49
|
+
callers that know their conversation language should pass `lang`, and the
|
|
50
|
+
detector stays the no-hint fallback.
|
|
51
|
+
"""
|
|
52
|
+
|
|
53
|
+
import re
|
|
54
|
+
|
|
55
|
+
from voxnorm import cn, en, ja, ko, zh
|
|
56
|
+
from voxnorm.tokens import TOKEN_KINDS, TOKEN_PATTERN
|
|
57
|
+
|
|
58
|
+
__version__ = "0.1.0"
|
|
59
|
+
|
|
60
|
+
|
|
61
|
+
def normalize(text: str, lang: str | None = None) -> str:
|
|
62
|
+
"""Return `text` with every recognised written form replaced by its spoken form.
|
|
63
|
+
|
|
64
|
+
Args:
|
|
65
|
+
text: Text bound for a synthesizer. Anything the scanner does not
|
|
66
|
+
recognise is returned character-for-character.
|
|
67
|
+
lang: A language tag (`zh`, `zh-TW`, `zh-CN`, `zh-Hans`, `en`, `ja`,
|
|
68
|
+
`ko`, `en_US`, case-insensitive). A bare `zh` is Traditional
|
|
69
|
+
(the package's home locale); a `CN`/`Hans`/`SG` subtag selects
|
|
70
|
+
the Simplified re-spelling. `None` auto-detects on script: kana
|
|
71
|
+
means Japanese, hangul Korean, a Han character Chinese (with a
|
|
72
|
+
distinctive-glyph check choosing Simplified when the text shows
|
|
73
|
+
it), otherwise English.
|
|
74
|
+
|
|
75
|
+
Raises:
|
|
76
|
+
ValueError: If `lang` names a language no verbaliser is registered for.
|
|
77
|
+
"""
|
|
78
|
+
verbalize = _VERBALIZERS[_resolve_lang(text, lang)]
|
|
79
|
+
return TOKEN_PATTERN.sub(lambda match: verbalize(_kind_of(match), match), text)
|
|
80
|
+
|
|
81
|
+
|
|
82
|
+
_VERBALIZERS = {
|
|
83
|
+
"zh": zh.verbalize,
|
|
84
|
+
"zh_cn": cn.verbalize,
|
|
85
|
+
"en": en.verbalize,
|
|
86
|
+
"ja": ja.verbalize,
|
|
87
|
+
"ko": ko.verbalize,
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
_SIMPLIFIED_SUBTAGS = frozenset({"cn", "hans", "sg"})
|
|
91
|
+
|
|
92
|
+
_HAN = re.compile(r"[㐀-䶿一-鿿豈-]")
|
|
93
|
+
"""CJK Unified Ideographs (extension A, the base block, and the compatibility
|
|
94
|
+
block) -- one match anywhere is the evidence the text is Chinese, once kana
|
|
95
|
+
and hangul have had their look."""
|
|
96
|
+
|
|
97
|
+
_KANA = re.compile(r"[ぁ-ゖァ-ヺー]")
|
|
98
|
+
"""Hiragana and katakana (plus the prolonged-sound mark) -- checked before
|
|
99
|
+
`_HAN`, because Japanese text carries kanji too."""
|
|
100
|
+
|
|
101
|
+
_HANGUL = re.compile(r"[가-힣]")
|
|
102
|
+
"""Precomposed hangul syllables -- Korean shares no script with the others."""
|
|
103
|
+
|
|
104
|
+
|
|
105
|
+
_SIMPLIFIED_ONLY = frozenset("万亿点号两说对时会开关议历后发东车长门问间头买卖体见")
|
|
106
|
+
_TRADITIONAL_ONLY = frozenset("萬億點號兩說對時會開關議歷後發東車長門問間頭買賣體見")
|
|
107
|
+
"""Twenty-six high-frequency characters whose glyph differs between the two
|
|
108
|
+
scripts -- a heuristic, not a converter. Text showing Simplified evidence and
|
|
109
|
+
no Traditional evidence reads as zh-CN; anything else Chinese defaults to
|
|
110
|
+
Traditional, the package's home locale."""
|
|
111
|
+
|
|
112
|
+
|
|
113
|
+
def _resolve_lang(text: str, lang: str | None) -> str:
|
|
114
|
+
if lang is None:
|
|
115
|
+
if _KANA.search(text):
|
|
116
|
+
return "ja"
|
|
117
|
+
if _HANGUL.search(text):
|
|
118
|
+
return "ko"
|
|
119
|
+
if not _HAN.search(text):
|
|
120
|
+
return "en"
|
|
121
|
+
chars = set(text)
|
|
122
|
+
if chars & _SIMPLIFIED_ONLY and not chars & _TRADITIONAL_ONLY:
|
|
123
|
+
return "zh_cn"
|
|
124
|
+
return "zh"
|
|
125
|
+
parts = re.split(r"[-_]", lang.lower())
|
|
126
|
+
primary = parts[0]
|
|
127
|
+
if primary == "zh" and any(subtag in _SIMPLIFIED_SUBTAGS for subtag in parts[1:]):
|
|
128
|
+
return "zh_cn"
|
|
129
|
+
if primary not in _VERBALIZERS:
|
|
130
|
+
known = ", ".join(sorted(_VERBALIZERS))
|
|
131
|
+
raise ValueError(f"unknown voxnorm language {lang!r}; expected one of {known}")
|
|
132
|
+
return primary
|
|
133
|
+
|
|
134
|
+
|
|
135
|
+
def _kind_of(match: re.Match) -> str:
|
|
136
|
+
"""Name the top-level token class `match` matched.
|
|
137
|
+
|
|
138
|
+
Not `match.lastgroup`: with named part-groups nested inside each class
|
|
139
|
+
(`time_h` inside `time`), `lastgroup` names whichever part matched last,
|
|
140
|
+
not the class.
|
|
141
|
+
"""
|
|
142
|
+
for kind in TOKEN_KINDS:
|
|
143
|
+
if match.group(kind) is not None:
|
|
144
|
+
return kind
|
|
145
|
+
raise AssertionError("TOKEN_PATTERN matched outside its own kinds")
|
voxnorm/cn.py
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
"""The zh-CN verbaliser: the zh-TW readings, re-spelled for the mainland.
|
|
2
|
+
|
|
3
|
+
The spoken readings are the same language -- 两/二 placement, interior zeros,
|
|
4
|
+
digit-by-digit codes all match -- so this module does not duplicate the
|
|
5
|
+
algorithm: it takes `zh`'s output and re-spells it through one glyph table.
|
|
6
|
+
Two entries are more than glyphs: 兆 becomes 万亿 (the mainland reading of
|
|
7
|
+
10^12), and `¥` means renminbi here, not yen, so the currency rule overrides
|
|
8
|
+
that one symbol before the shared table applies.
|
|
9
|
+
"""
|
|
10
|
+
|
|
11
|
+
import re
|
|
12
|
+
|
|
13
|
+
from voxnorm import zh
|
|
14
|
+
|
|
15
|
+
_TO_SIMPLIFIED = str.maketrans(
|
|
16
|
+
{
|
|
17
|
+
"兩": "两",
|
|
18
|
+
"萬": "万",
|
|
19
|
+
"億": "亿",
|
|
20
|
+
"兆": "万亿",
|
|
21
|
+
"點": "点",
|
|
22
|
+
"號": "号",
|
|
23
|
+
"幣": "币",
|
|
24
|
+
"歐": "欧",
|
|
25
|
+
"鎊": "镑",
|
|
26
|
+
"韓": "韩",
|
|
27
|
+
"圓": "元",
|
|
28
|
+
}
|
|
29
|
+
)
|
|
30
|
+
"""Every character the zh verbaliser can emit that differs in Simplified
|
|
31
|
+
script. Not a general converter -- the domain is exactly `zh.py`'s own output
|
|
32
|
+
vocabulary, which is what keeps a one-table re-spelling honest."""
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
def verbalize(kind: str, match: re.Match) -> str:
|
|
36
|
+
"""Spell the token `match` of class `kind` the way zh-CN says it."""
|
|
37
|
+
if kind == "currency" and match["cur_sym"] == "¥":
|
|
38
|
+
# In the mainland locale ¥ is renminbi (元), not the yen zh-TW reads.
|
|
39
|
+
return f"{zh.amount_to_zh(match['cur_amt']).translate(_TO_SIMPLIFIED)}元"
|
|
40
|
+
return zh.verbalize(kind, match).translate(_TO_SIMPLIFIED)
|
voxnorm/en.py
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
"""The English verbaliser: num2words for the number words, local rules for everything else.
|
|
2
|
+
|
|
3
|
+
num2words carries the cardinal/ordinal/year word forms (for English and, when
|
|
4
|
+
voxnorm grows, some fifty further languages); what it cannot know is which
|
|
5
|
+
reading a written form takes -- that a clock's `:05` is "oh five", that a
|
|
6
|
+
phone number is digits with pauses, that a room number is never a quantity.
|
|
7
|
+
Those readings live here.
|
|
8
|
+
"""
|
|
9
|
+
|
|
10
|
+
import re
|
|
11
|
+
|
|
12
|
+
from num2words import num2words
|
|
13
|
+
|
|
14
|
+
_DIGIT_WORDS = ("zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine")
|
|
15
|
+
|
|
16
|
+
_MONTHS = (
|
|
17
|
+
"January",
|
|
18
|
+
"February",
|
|
19
|
+
"March",
|
|
20
|
+
"April",
|
|
21
|
+
"May",
|
|
22
|
+
"June",
|
|
23
|
+
"July",
|
|
24
|
+
"August",
|
|
25
|
+
"September",
|
|
26
|
+
"October",
|
|
27
|
+
"November",
|
|
28
|
+
"December",
|
|
29
|
+
)
|
|
30
|
+
|
|
31
|
+
_CURRENCIES = {
|
|
32
|
+
"NT$": ("NT dollar", "NT dollars"),
|
|
33
|
+
"NTD": ("NT dollar", "NT dollars"),
|
|
34
|
+
"US$": ("US dollar", "US dollars"),
|
|
35
|
+
"USD": ("US dollar", "US dollars"),
|
|
36
|
+
"$": ("dollar", "dollars"),
|
|
37
|
+
"€": ("euro", "euros"),
|
|
38
|
+
"¥": ("yen", "yen"),
|
|
39
|
+
"₩": ("won", "won"),
|
|
40
|
+
"£": ("pound", "pounds"),
|
|
41
|
+
}
|
|
42
|
+
"""(singular, plural) spoken after the amount. A bare `$` reads as plain
|
|
43
|
+
dollars; the text saying which kind is the text's job, not a guess."""
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
def verbalize(kind: str, match: re.Match) -> str:
|
|
47
|
+
"""Spell the token `match` of class `kind` the way English says it."""
|
|
48
|
+
return _RULES[kind](match)
|
|
49
|
+
|
|
50
|
+
|
|
51
|
+
def digits_to_en(digits: str) -> str:
|
|
52
|
+
"""`digits` read one at a time, zeros kept: `0912` -> "zero nine one two"."""
|
|
53
|
+
return " ".join(_DIGIT_WORDS[int(char)] for char in digits if char.isdigit())
|
|
54
|
+
|
|
55
|
+
|
|
56
|
+
def _cardinal(value: int) -> str:
|
|
57
|
+
return num2words(value)
|
|
58
|
+
|
|
59
|
+
|
|
60
|
+
def _time(match: re.Match) -> str:
|
|
61
|
+
hour, minute = int(match["time_h"]), int(match["time_m"])
|
|
62
|
+
if minute == 0:
|
|
63
|
+
return f"{_cardinal(hour)} o'clock"
|
|
64
|
+
if minute < 10:
|
|
65
|
+
return f"{_cardinal(hour)} oh {_DIGIT_WORDS[minute]}"
|
|
66
|
+
return f"{_cardinal(hour)} {_cardinal(minute)}"
|
|
67
|
+
|
|
68
|
+
|
|
69
|
+
def _iso_date(match: re.Match) -> str:
|
|
70
|
+
month = _MONTHS[int(match["iso_mo"]) - 1]
|
|
71
|
+
day = num2words(int(match["iso_d"]), to="ordinal")
|
|
72
|
+
year = num2words(int(match["iso_y"]), to="year")
|
|
73
|
+
return f"{month} {day}, {year}"
|
|
74
|
+
|
|
75
|
+
|
|
76
|
+
def _phone(match: re.Match) -> str:
|
|
77
|
+
groups = re.split(r"[-\ ]", match[0])
|
|
78
|
+
return ", ".join(digits_to_en(group) for group in groups)
|
|
79
|
+
|
|
80
|
+
|
|
81
|
+
def _amount_to_en(amount: str) -> str:
|
|
82
|
+
integer, _, fraction = amount.replace(",", "").partition(".")
|
|
83
|
+
spoken = _cardinal(int(integer))
|
|
84
|
+
return f"{spoken} point {digits_to_en(fraction)}" if fraction else spoken
|
|
85
|
+
|
|
86
|
+
|
|
87
|
+
def _currency(match: re.Match) -> str:
|
|
88
|
+
amount = match["cur_amt"]
|
|
89
|
+
singular, plural = _CURRENCIES[match["cur_sym"]]
|
|
90
|
+
unit = singular if amount.replace(",", "") == "1" else plural
|
|
91
|
+
return f"{_amount_to_en(amount)} {unit}"
|
|
92
|
+
|
|
93
|
+
|
|
94
|
+
def _bare_int(match: re.Match) -> str:
|
|
95
|
+
run = match[0]
|
|
96
|
+
# A leading zero is never a quantity -- it is a code, read digit by digit.
|
|
97
|
+
if len(run) > 1 and int(run[0]) == 0:
|
|
98
|
+
return digits_to_en(run)
|
|
99
|
+
return _cardinal(int(run))
|
|
100
|
+
|
|
101
|
+
|
|
102
|
+
_RULES = {
|
|
103
|
+
"time": _time,
|
|
104
|
+
"iso_date": _iso_date,
|
|
105
|
+
"phone": _phone,
|
|
106
|
+
"currency": _currency,
|
|
107
|
+
"percent": lambda match: f"{_amount_to_en(match['pct_num'])} percent",
|
|
108
|
+
"room_en": lambda match: f"{match['room_word']} {digits_to_en(match['room_num'])}",
|
|
109
|
+
# The zh-anchored kinds (a digit run in front of 號/室/年) stay written
|
|
110
|
+
# under a forced `lang="en"`: converting the digits while the CJK anchor
|
|
111
|
+
# character stays behind glues English number words onto raw 年/室
|
|
112
|
+
# ("twenty twenty-six年") -- worse than the written form it replaced.
|
|
113
|
+
"room_cjk": lambda match: match[0],
|
|
114
|
+
"year": lambda match: match[0],
|
|
115
|
+
"decimal": lambda match: _amount_to_en(match[0]),
|
|
116
|
+
"comma_int": lambda match: _cardinal(int(match[0].replace(",", ""))),
|
|
117
|
+
"bare_int": _bare_int,
|
|
118
|
+
}
|
voxnorm/ja.py
ADDED
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
"""The Japanese verbaliser: kanji number words via num2words, local readings for the rest.
|
|
2
|
+
|
|
3
|
+
num2words' `ja` cardinals are solid (cross-checked against kanjize during the
|
|
4
|
+
language survey: byte-identical on every probe), so numbers ride on it. What
|
|
5
|
+
is decided here is which reading each written form takes: a year is a plain
|
|
6
|
+
cardinal with `era=False` (二千二十六年, not the era reading 令和八年 the
|
|
7
|
+
library defaults to), a phone number is digits one at a time with 〇 for
|
|
8
|
+
zero, a room number before 号/室 likewise, and a clock's :30 is 半. The
|
|
9
|
+
kana-vs-kanji pronunciation of what this module emits (四時 read よじ, not
|
|
10
|
+
よんじ) is the TTS frontend's business -- the written spoken-form here is
|
|
11
|
+
standard Japanese that any ja frontend already reads correctly.
|
|
12
|
+
"""
|
|
13
|
+
|
|
14
|
+
import re
|
|
15
|
+
|
|
16
|
+
from num2words import num2words
|
|
17
|
+
|
|
18
|
+
_DIGITS = "〇一二三四五六七八九"
|
|
19
|
+
|
|
20
|
+
_CURRENCIES = {
|
|
21
|
+
"NT$": "台湾ドル",
|
|
22
|
+
"NTD": "台湾ドル",
|
|
23
|
+
"US$": "米ドル",
|
|
24
|
+
"USD": "米ドル",
|
|
25
|
+
"$": "ドル",
|
|
26
|
+
"€": "ユーロ",
|
|
27
|
+
"¥": "円",
|
|
28
|
+
"₩": "ウォン",
|
|
29
|
+
"£": "ポンド",
|
|
30
|
+
}
|
|
31
|
+
"""Spoken after the amount -- Japanese puts every currency word last."""
|
|
32
|
+
|
|
33
|
+
|
|
34
|
+
def verbalize(kind: str, match: re.Match) -> str:
|
|
35
|
+
"""Spell the token `match` of class `kind` the way Japanese says it."""
|
|
36
|
+
return _RULES[kind](match)
|
|
37
|
+
|
|
38
|
+
|
|
39
|
+
def digits_to_ja(digits: str) -> str:
|
|
40
|
+
"""`digits` read one at a time, zeros kept: `0912` -> 〇九一二."""
|
|
41
|
+
return "".join(_DIGITS[int(char)] for char in digits if char.isdigit())
|
|
42
|
+
|
|
43
|
+
|
|
44
|
+
def _cardinal(value: int) -> str:
|
|
45
|
+
return num2words(value, lang="ja")
|
|
46
|
+
|
|
47
|
+
|
|
48
|
+
def _time(match: re.Match) -> str:
|
|
49
|
+
hour, minute = int(match["time_h"]), int(match["time_m"])
|
|
50
|
+
if minute == 0:
|
|
51
|
+
return f"{_cardinal(hour)}時"
|
|
52
|
+
if minute == 30:
|
|
53
|
+
return f"{_cardinal(hour)}時半"
|
|
54
|
+
return f"{_cardinal(hour)}時{_cardinal(minute)}分"
|
|
55
|
+
|
|
56
|
+
|
|
57
|
+
def _iso_date(match: re.Match) -> str:
|
|
58
|
+
# Plain cardinal, not to="year": num2words' year form appends its own 年
|
|
59
|
+
# (and defaults to the era reading besides).
|
|
60
|
+
year = _cardinal(int(match["iso_y"]))
|
|
61
|
+
return f"{year}年{_cardinal(int(match['iso_mo']))}月{_cardinal(int(match['iso_d']))}日"
|
|
62
|
+
|
|
63
|
+
|
|
64
|
+
def _spoken_int(digits: str) -> str:
|
|
65
|
+
# A leading zero is never a quantity -- it is a code, read digit by digit.
|
|
66
|
+
if len(digits) > 1 and int(digits[0]) == 0:
|
|
67
|
+
return digits_to_ja(digits)
|
|
68
|
+
return _cardinal(int(digits))
|
|
69
|
+
|
|
70
|
+
|
|
71
|
+
def _amount_to_ja(amount: str) -> str:
|
|
72
|
+
integer, _, fraction = amount.replace(",", "").partition(".")
|
|
73
|
+
spoken = _spoken_int(integer)
|
|
74
|
+
return f"{spoken}点{digits_to_ja(fraction)}" if fraction else spoken
|
|
75
|
+
|
|
76
|
+
|
|
77
|
+
_RULES = {
|
|
78
|
+
"time": _time,
|
|
79
|
+
"iso_date": _iso_date,
|
|
80
|
+
"phone": lambda match: digits_to_ja(match[0]),
|
|
81
|
+
"currency": lambda match: f"{_amount_to_ja(match['cur_amt'])}{_CURRENCIES[match['cur_sym']]}",
|
|
82
|
+
"percent": lambda match: f"{_amount_to_ja(match['pct_num'])}パーセント",
|
|
83
|
+
"room_en": lambda match: f"{match['room_word']} {digits_to_ja(match['room_num'])}",
|
|
84
|
+
"room_cjk": lambda match: digits_to_ja(match[0]),
|
|
85
|
+
"year": lambda match: _cardinal(int(match[0])),
|
|
86
|
+
"decimal": lambda match: _amount_to_ja(match[0]),
|
|
87
|
+
"comma_int": lambda match: _spoken_int(match[0].replace(",", "")),
|
|
88
|
+
"bare_int": lambda match: _spoken_int(match[0]),
|
|
89
|
+
}
|
voxnorm/ko.py
ADDED
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
"""The Korean verbaliser: sino-Korean via num2words, the native numerals written here.
|
|
2
|
+
|
|
3
|
+
Korean reads numbers in two systems and the choice is grammatical, not
|
|
4
|
+
stylistic: hours take the *native* numerals (한 시, 두 시 -- never 일 시),
|
|
5
|
+
while minutes, dates, money and digit strings take the *sino* numerals
|
|
6
|
+
(이천이십육년, 삼십 분). num2words' `ko` is sino-only, and the language
|
|
7
|
+
survey found no working pure-Python native-numeral library at all -- but the
|
|
8
|
+
attributive native forms an hour clock needs are a closed set of twelve, so
|
|
9
|
+
they live in `_NATIVE_HOURS`. Zero in a digit string is 공 (the phone-number
|
|
10
|
+
reading), not 영.
|
|
11
|
+
"""
|
|
12
|
+
|
|
13
|
+
import re
|
|
14
|
+
|
|
15
|
+
from num2words import num2words
|
|
16
|
+
|
|
17
|
+
_DIGITS = ("공", "일", "이", "삼", "사", "오", "육", "칠", "팔", "구")
|
|
18
|
+
|
|
19
|
+
_NATIVE_HOURS = ("영", "한", "두", "세", "네", "다섯", "여섯", "일곱", "여덟", "아홉", "열", "열한", "열두")
|
|
20
|
+
"""Attributive native numerals 1-12, the only forms 시 accepts, with 영 at
|
|
21
|
+
index 0 so a loosened hour guard can never emit an empty hour. A 24-hour
|
|
22
|
+
clock past twelve (14시) is read sino (십사 시), matching broadcast usage."""
|
|
23
|
+
|
|
24
|
+
_MONTHS = ("일월", "이월", "삼월", "사월", "오월", "유월", "칠월", "팔월", "구월", "시월", "십일월", "십이월")
|
|
25
|
+
"""June and October are irregular (유월/시월, not 육월/십월), so the month is a
|
|
26
|
+
lookup, never a composed sino number."""
|
|
27
|
+
|
|
28
|
+
_CURRENCIES = {
|
|
29
|
+
"NT$": "대만 달러",
|
|
30
|
+
"NTD": "대만 달러",
|
|
31
|
+
"US$": "미국 달러",
|
|
32
|
+
"USD": "미국 달러",
|
|
33
|
+
"$": "달러",
|
|
34
|
+
"€": "유로",
|
|
35
|
+
"¥": "엔",
|
|
36
|
+
"₩": "원",
|
|
37
|
+
"£": "파운드",
|
|
38
|
+
}
|
|
39
|
+
"""Spoken after the amount -- Korean puts every currency word last."""
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
def verbalize(kind: str, match: re.Match) -> str:
|
|
43
|
+
"""Spell the token `match` of class `kind` the way Korean says it."""
|
|
44
|
+
return _RULES[kind](match)
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
def digits_to_ko(digits: str) -> str:
|
|
48
|
+
"""`digits` read one at a time, zeros kept as 공: `0912` -> 공구일이."""
|
|
49
|
+
return "".join(_DIGITS[int(char)] for char in digits if char.isdigit())
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
def _sino(value: int) -> str:
|
|
53
|
+
return num2words(value, lang="ko")
|
|
54
|
+
|
|
55
|
+
|
|
56
|
+
def _time(match: re.Match) -> str:
|
|
57
|
+
hour, minute = int(match["time_h"]), int(match["time_m"])
|
|
58
|
+
spoken_hour = _NATIVE_HOURS[hour] if 1 <= hour <= 12 else _sino(hour)
|
|
59
|
+
if minute == 0:
|
|
60
|
+
return f"{spoken_hour} 시"
|
|
61
|
+
if minute == 30:
|
|
62
|
+
return f"{spoken_hour} 시 반"
|
|
63
|
+
return f"{spoken_hour} 시 {_sino(minute)} 분"
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
def _iso_date(match: re.Match) -> str:
|
|
67
|
+
year, month, day = int(match["iso_y"]), int(match["iso_mo"]), int(match["iso_d"])
|
|
68
|
+
return f"{_sino(year)}년 {_MONTHS[month - 1]} {_sino(day)}일"
|
|
69
|
+
|
|
70
|
+
|
|
71
|
+
def _spoken_int(digits: str) -> str:
|
|
72
|
+
# A leading zero is never a quantity -- it is a code, read digit by digit.
|
|
73
|
+
if len(digits) > 1 and int(digits[0]) == 0:
|
|
74
|
+
return digits_to_ko(digits)
|
|
75
|
+
return _sino(int(digits))
|
|
76
|
+
|
|
77
|
+
|
|
78
|
+
def _amount_to_ko(amount: str) -> str:
|
|
79
|
+
integer, _, fraction = amount.replace(",", "").partition(".")
|
|
80
|
+
spoken = _spoken_int(integer)
|
|
81
|
+
# Unspaced around 점: the decimal reading is written as one clump
|
|
82
|
+
# (삼점일사), unlike the counter nouns 시/분/원 that take a space.
|
|
83
|
+
return f"{spoken}점{digits_to_ko(fraction)}" if fraction else spoken
|
|
84
|
+
|
|
85
|
+
|
|
86
|
+
def _phone(match: re.Match) -> str:
|
|
87
|
+
groups = re.split(r"[-\ ]", match[0])
|
|
88
|
+
return " ".join(digits_to_ko(group) for group in groups)
|
|
89
|
+
|
|
90
|
+
|
|
91
|
+
_RULES = {
|
|
92
|
+
"time": _time,
|
|
93
|
+
"iso_date": _iso_date,
|
|
94
|
+
"phone": _phone,
|
|
95
|
+
"currency": lambda match: f"{_amount_to_ko(match['cur_amt'])} {_CURRENCIES[match['cur_sym']]}",
|
|
96
|
+
"percent": lambda match: f"{_amount_to_ko(match['pct_num'])} 퍼센트",
|
|
97
|
+
"room_en": lambda match: f"{match['room_word']} {digits_to_ko(match['room_num'])}",
|
|
98
|
+
"room_cjk": lambda match: digits_to_ko(match[0]),
|
|
99
|
+
"year": lambda match: _sino(int(match[0])),
|
|
100
|
+
"decimal": lambda match: _amount_to_ko(match[0]),
|
|
101
|
+
"comma_int": lambda match: _spoken_int(match[0].replace(",", "")),
|
|
102
|
+
"bare_int": lambda match: _spoken_int(match[0]),
|
|
103
|
+
}
|
voxnorm/py.typed
ADDED
|
File without changes
|
voxnorm/tokens.py
ADDED
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
"""The language-neutral scanner: one pattern that finds every written form voxnorm speaks.
|
|
2
|
+
|
|
3
|
+
Each alternative is a named group whose name is the token's kind; the part
|
|
4
|
+
groups inside it (`time_h`, `cur_amt`, ...) are what the verbalisers read.
|
|
5
|
+
Order in the alternation is priority: a clock time must win over the bare
|
|
6
|
+
integers inside it, a phone number over the currency-less digit runs it is
|
|
7
|
+
made of.
|
|
8
|
+
|
|
9
|
+
The pattern compiles with `re.ASCII`, so `\\d` means `[0-9]` and nothing
|
|
10
|
+
else. Full-width (zenkaku) digits are deliberately NOT scanned: without the
|
|
11
|
+
flag they matched the loose fallback alternatives but never the strict ones
|
|
12
|
+
(the phone lead, the ISO-date years, the literal `%` and `,`), so
|
|
13
|
+
`0912-345-678` half-converted into a chimera of digit-readings and
|
|
14
|
+
stranded hyphens. Under the charter an unrecognised form passes through
|
|
15
|
+
whole -- and TTS frontends in the locales that write zenkaku digits read
|
|
16
|
+
them natively anyway.
|
|
17
|
+
|
|
18
|
+
Two hard-won rules shape every alternative:
|
|
19
|
+
|
|
20
|
+
- **No `\\b` against CJK.** Han characters are word characters to `re`, so
|
|
21
|
+
`\\b` never fires between 是 and 12, and every "boundary" here is written
|
|
22
|
+
as an explicit digit lookaround instead.
|
|
23
|
+
- **Ambiguous forms are absent, not clever.** No slash form matches at all
|
|
24
|
+
(`1/2` the fraction and `8/17` the date are the same string); `號` takes a
|
|
25
|
+
digit-string reading only at three digits and up, because `302號` is a room
|
|
26
|
+
and `17號` is a day of the month.
|
|
27
|
+
"""
|
|
28
|
+
|
|
29
|
+
import re
|
|
30
|
+
|
|
31
|
+
TOKEN_KINDS = (
|
|
32
|
+
"time",
|
|
33
|
+
"iso_date",
|
|
34
|
+
"phone",
|
|
35
|
+
"currency",
|
|
36
|
+
"percent",
|
|
37
|
+
"room_en",
|
|
38
|
+
"room_cjk",
|
|
39
|
+
"year",
|
|
40
|
+
"decimal",
|
|
41
|
+
"comma_int",
|
|
42
|
+
"bare_int",
|
|
43
|
+
)
|
|
44
|
+
"""Every top-level group in `TOKEN_PATTERN`, in the pattern's own priority order."""
|
|
45
|
+
|
|
46
|
+
TOKEN_PATTERN = re.compile(
|
|
47
|
+
r"""
|
|
48
|
+
(?P<time>(?<![\d:])(?P<time_h>[01]?\d|2[0-3]):(?P<time_m>[0-5]\d)(?![\d:]))
|
|
49
|
+
| (?P<iso_date>(?<!\d)(?P<iso_y>(?:19|20)\d\d)-(?P<iso_mo>0?[1-9]|1[0-2])-(?P<iso_d>0?[1-9]|[12]\d|3[01])(?!\d))
|
|
50
|
+
| (?P<phone>(?<!\d)0\d{1,3}(?:[-\ ]\d{3,4}){2}(?![\d-]))
|
|
51
|
+
| (?P<currency>(?P<cur_sym>NT\$|US\$|USD|NTD|\$|€|¥|£|₩)\ ?(?P<cur_amt>\d[\d,]*(?:\.\d+)?))
|
|
52
|
+
| (?P<percent>(?<![\d.])(?P<pct_num>\d[\d,]*(?:\.\d+)?)\ ?%)
|
|
53
|
+
| (?P<room_en>(?P<room_word>\b[Rr]oom|\b[Ee]xt\.?|房號|分機)\ ?\#?(?P<room_num>\d{2,5})(?!\d))
|
|
54
|
+
| (?P<room_cjk>(?<!\d)\d{3,10}(?=號|号|室|호))
|
|
55
|
+
| (?P<year>(?<!\d)(?:19|20)\d\d(?=年|년))
|
|
56
|
+
| (?P<decimal>(?<![\d./::])\d+\.\d+(?![\d./::]))
|
|
57
|
+
| (?P<comma_int>(?<![\d,/::])\d{1,3}(?:,\d{3})+(?![\d,/::]))
|
|
58
|
+
| (?P<bare_int>(?<![\d./::])\d+(?![\d./::]))
|
|
59
|
+
""",
|
|
60
|
+
re.VERBOSE | re.ASCII,
|
|
61
|
+
)
|
|
62
|
+
"""Notes per alternative, in order:
|
|
63
|
+
|
|
64
|
+
- `time` — `HH:MM`, 24-hour, minutes always two digits; the trailing guard
|
|
65
|
+
keeps `12:00:05` (a duration with seconds, unhandled) from half-matching.
|
|
66
|
+
- `iso_date` — `YYYY-MM-DD` with a plausibility window on every field, so an
|
|
67
|
+
id-like `1234-56-78` stays text.
|
|
68
|
+
- `phone` — a leading-zero group and exactly two 3-4 digit separator groups:
|
|
69
|
+
the shapes Taiwanese numbers are actually written in (`0912-345-678`,
|
|
70
|
+
`02-2345-6789`, `0800-092-000`). A separator-less `0912345678` is caught by
|
|
71
|
+
`bare_int`'s leading-zero rule instead.
|
|
72
|
+
- `currency` — symbol before amount, the only order this package reads.
|
|
73
|
+
- `room_en` / `room_cjk` — a digit string read digit by digit, licensed by an
|
|
74
|
+
explicit context word. The CJK suffix (Traditional 號, Japanese 号/室,
|
|
75
|
+
Korean 호) stays outside the match (lookahead) so `302號房` keeps its 號房;
|
|
76
|
+
the en prefix is inside and re-emitted, because a variable-width lookbehind
|
|
77
|
+
is not a thing `re` has.
|
|
78
|
+
- `year` — four digits directly before 年/년 read as a year (二零二六年
|
|
79
|
+
digit by digit in Chinese, 二千二十六年 / 이천이십육년 as cardinals in
|
|
80
|
+
Japanese and Korean — the split lives in the verbalisers); without the
|
|
81
|
+
suffix a four-digit integer is a quantity.
|
|
82
|
+
- `decimal` / `comma_int` / `bare_int` — the residue: point decimals,
|
|
83
|
+
comma-grouped integers, then any digit run. The lookarounds keep each from
|
|
84
|
+
matching inside a form a higher alternative half-consumed, keep version-ish
|
|
85
|
+
strings (`1.2.3`) whole and untouched, and refuse any digit touching a
|
|
86
|
+
slash or a colon. The `/` guards are the charter's "no slash form converts"
|
|
87
|
+
applied to the digits inside the form (`1/2` must not become 一/二); the
|
|
88
|
+
`:`/`:` guards do the same for a colon form the `time` alternative
|
|
89
|
+
rejected -- an invalid `24:00` must stay written, not half-convert to
|
|
90
|
+
二十四:零零 around a stranded colon.
|
|
91
|
+
"""
|
voxnorm/zh.py
ADDED
|
@@ -0,0 +1,157 @@
|
|
|
1
|
+
"""The zh-TW verbaliser: numbers written the way a Taiwanese speaker reads them out.
|
|
2
|
+
|
|
3
|
+
Chinese is the one language voxnorm spells out itself rather than through a
|
|
4
|
+
number library: num2words has no Chinese at all, and the converters that do
|
|
5
|
+
exist write `一千二百` where a zh-TW speaker says `一千兩百`, in Simplified
|
|
6
|
+
glyphs besides. The 兩/二 choice is positional -- 兩 before a multiplier
|
|
7
|
+
(兩百, 兩千, 兩萬) and for two o'clock (兩點), 二 everywhere else (二十二,
|
|
8
|
+
十二) -- and `_LIANG` is the one rule that encodes it. Output is Traditional
|
|
9
|
+
throughout, 台 not 臺.
|
|
10
|
+
"""
|
|
11
|
+
|
|
12
|
+
import re
|
|
13
|
+
|
|
14
|
+
_DIGITS = "零一二三四五六七八九"
|
|
15
|
+
_SECTION_UNITS = ("", "十", "百", "千")
|
|
16
|
+
_GROUP_UNITS = ("", "萬", "億", "兆")
|
|
17
|
+
|
|
18
|
+
_LIANG_IN_SECTION = re.compile("二(?=[百千])")
|
|
19
|
+
"""兩 replaces 二 where a multiplier follows *within a section* -- 兩百二十二,
|
|
20
|
+
never 兩十. The group units (萬/億/兆) are handled separately in
|
|
21
|
+
`number_to_zh`, and only when the whole section reads 二: adversarial review
|
|
22
|
+
proved a flat rule over the joined string turns 120000 into 十兩萬, because
|
|
23
|
+
the 二 of 十二 happens to sit before the 萬."""
|
|
24
|
+
|
|
25
|
+
_CURRENCIES = {
|
|
26
|
+
"NT$": ("新台幣", "元"),
|
|
27
|
+
"NTD": ("新台幣", "元"),
|
|
28
|
+
"US$": ("", "美元"),
|
|
29
|
+
"USD": ("", "美元"),
|
|
30
|
+
"$": ("", "元"),
|
|
31
|
+
"€": ("", "歐元"),
|
|
32
|
+
"¥": ("", "日圓"),
|
|
33
|
+
"₩": ("", "韓元"),
|
|
34
|
+
"£": ("", "英鎊"),
|
|
35
|
+
}
|
|
36
|
+
"""(prefix, suffix) around the spoken amount. A bare `$` reads as 元: in this
|
|
37
|
+
package's home locale a bill is in NTD unless the text says otherwise."""
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
def verbalize(kind: str, match: re.Match) -> str:
|
|
41
|
+
"""Spell the token `match` of class `kind` the way zh-TW says it."""
|
|
42
|
+
return _RULES[kind](match)
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
def number_to_zh(value: int) -> str:
|
|
46
|
+
"""`value` in grouped spoken form: `1200` -> 一千兩百, `100050` -> 十萬零五十.
|
|
47
|
+
|
|
48
|
+
Standard positional reading in 4-digit sections (萬/億/兆), interior zeros
|
|
49
|
+
compressed to one 零, 10-19 and 十萬-class leads dropping the leading 一
|
|
50
|
+
(十二 and 十萬, but the interior 一十 of 一百一十二 stays).
|
|
51
|
+
|
|
52
|
+
Raises:
|
|
53
|
+
ValueError: At 10^16 and beyond -- 兆 is the largest unit spoken here,
|
|
54
|
+
and truncating silently was a measured bug, not a feature. Callers
|
|
55
|
+
with digit runs that long want `digits_to_zh` (`_spoken_int`).
|
|
56
|
+
"""
|
|
57
|
+
if value >= 10_000_000_000_000_000:
|
|
58
|
+
raise ValueError(f"{value} is beyond 兆 range; read it digit by digit instead")
|
|
59
|
+
if value == 0:
|
|
60
|
+
return _DIGITS[0]
|
|
61
|
+
sections: list[tuple[int, str]] = []
|
|
62
|
+
for unit in _GROUP_UNITS:
|
|
63
|
+
value, section = divmod(value, 10000)
|
|
64
|
+
sections.append((section, unit))
|
|
65
|
+
if value == 0:
|
|
66
|
+
break
|
|
67
|
+
parts: list[str] = []
|
|
68
|
+
previous_index: int | None = None
|
|
69
|
+
for index in range(len(sections) - 1, -1, -1):
|
|
70
|
+
section, unit = sections[index]
|
|
71
|
+
if section == 0:
|
|
72
|
+
continue
|
|
73
|
+
# A positional gap must be voiced as one 零, or the magnitudes merge:
|
|
74
|
+
# 100050 is 十萬零五十, not 十萬五十 (which reads as 105,000). A gap is
|
|
75
|
+
# a section under 1000, or a whole section of zeros skipped between
|
|
76
|
+
# this one and the last one spoken (一億零兩千).
|
|
77
|
+
if parts and (section < 1000 or previous_index != index + 1):
|
|
78
|
+
parts.append(_DIGITS[0])
|
|
79
|
+
spoken = _LIANG_IN_SECTION.sub("兩", _section_to_zh(section))
|
|
80
|
+
# A section that is exactly 2 reads 兩 before its group unit (兩萬),
|
|
81
|
+
# but a section merely *ending* in 2 keeps its 二 (十二萬, 二十二億).
|
|
82
|
+
if unit and spoken == "二":
|
|
83
|
+
spoken = "兩"
|
|
84
|
+
parts.append(spoken + unit)
|
|
85
|
+
previous_index = index
|
|
86
|
+
rendered = "".join(parts)
|
|
87
|
+
if rendered.startswith("一十"):
|
|
88
|
+
rendered = rendered[1:]
|
|
89
|
+
return rendered
|
|
90
|
+
|
|
91
|
+
|
|
92
|
+
def digits_to_zh(digits: str) -> str:
|
|
93
|
+
"""`digits` read one at a time, zeros kept: `0912` -> 零九一二."""
|
|
94
|
+
return "".join(_DIGITS[int(char)] for char in digits if char.isdigit())
|
|
95
|
+
|
|
96
|
+
|
|
97
|
+
def _section_to_zh(section: int) -> str:
|
|
98
|
+
rendered = ""
|
|
99
|
+
for index in (3, 2, 1, 0):
|
|
100
|
+
digit = section // 10**index % 10
|
|
101
|
+
rendered += _DIGITS[0] if digit == 0 else _DIGITS[digit] + _SECTION_UNITS[index]
|
|
102
|
+
rendered = re.sub("零+", "零", rendered).strip("零")
|
|
103
|
+
return rendered or _DIGITS[0]
|
|
104
|
+
|
|
105
|
+
|
|
106
|
+
def _hour_to_zh(hour: int) -> str:
|
|
107
|
+
return "兩" if hour == 2 else number_to_zh(hour)
|
|
108
|
+
|
|
109
|
+
|
|
110
|
+
def _time(match: re.Match) -> str:
|
|
111
|
+
hour, minute = int(match["time_h"]), int(match["time_m"])
|
|
112
|
+
if minute == 0:
|
|
113
|
+
return f"{_hour_to_zh(hour)}點"
|
|
114
|
+
if minute == 30:
|
|
115
|
+
return f"{_hour_to_zh(hour)}點半"
|
|
116
|
+
spoken_minute = f"零{_DIGITS[minute]}" if minute < 10 else number_to_zh(minute)
|
|
117
|
+
return f"{_hour_to_zh(hour)}點{spoken_minute}分"
|
|
118
|
+
|
|
119
|
+
|
|
120
|
+
def _iso_date(match: re.Match) -> str:
|
|
121
|
+
year, month, day = match["iso_y"], int(match["iso_mo"]), int(match["iso_d"])
|
|
122
|
+
return f"{digits_to_zh(year)}年{number_to_zh(month)}月{number_to_zh(day)}日"
|
|
123
|
+
|
|
124
|
+
|
|
125
|
+
def _spoken_int(digits: str) -> str:
|
|
126
|
+
"""Grouped reading, unless `digits` is a code: a leading zero or a run past
|
|
127
|
+
兆 range (16 digits) reads digit by digit. The scanner is ASCII-only, so
|
|
128
|
+
`digits` is always plain 0-9 here."""
|
|
129
|
+
if (len(digits) > 1 and int(digits[0]) == 0) or len(digits) > 16:
|
|
130
|
+
return digits_to_zh(digits)
|
|
131
|
+
return number_to_zh(int(digits))
|
|
132
|
+
|
|
133
|
+
|
|
134
|
+
def amount_to_zh(amount: str) -> str:
|
|
135
|
+
integer, _, fraction = amount.replace(",", "").partition(".")
|
|
136
|
+
spoken = _spoken_int(integer)
|
|
137
|
+
return f"{spoken}點{digits_to_zh(fraction)}" if fraction else spoken
|
|
138
|
+
|
|
139
|
+
|
|
140
|
+
def _currency(match: re.Match) -> str:
|
|
141
|
+
prefix, suffix = _CURRENCIES[match["cur_sym"]]
|
|
142
|
+
return f"{prefix}{amount_to_zh(match['cur_amt'])}{suffix}"
|
|
143
|
+
|
|
144
|
+
|
|
145
|
+
_RULES = {
|
|
146
|
+
"time": _time,
|
|
147
|
+
"iso_date": _iso_date,
|
|
148
|
+
"phone": lambda match: digits_to_zh(match[0]),
|
|
149
|
+
"currency": _currency,
|
|
150
|
+
"percent": lambda match: f"百分之{amount_to_zh(match['pct_num'])}",
|
|
151
|
+
"room_en": lambda match: f"{match['room_word']} {digits_to_zh(match['room_num'])}",
|
|
152
|
+
"room_cjk": lambda match: digits_to_zh(match[0]),
|
|
153
|
+
"year": lambda match: digits_to_zh(match[0]),
|
|
154
|
+
"decimal": lambda match: amount_to_zh(match[0]),
|
|
155
|
+
"comma_int": lambda match: _spoken_int(match[0].replace(",", "")),
|
|
156
|
+
"bare_int": lambda match: _spoken_int(match[0]),
|
|
157
|
+
}
|
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: voxnorm
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Text normalization for TTS: written forms into spoken forms, multilingual, pure Python
|
|
5
|
+
Project-URL: Repository, https://github.com/allen2c/voxnorm
|
|
6
|
+
Author: Allen Chou
|
|
7
|
+
License-Expression: Apache-2.0
|
|
8
|
+
License-File: LICENSE
|
|
9
|
+
Keywords: speech,spoken-form,text-normalization,tts,verbalization,zh-tw
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
|
|
14
|
+
Classifier: Topic :: Text Processing :: Linguistic
|
|
15
|
+
Requires-Python: >=3.10
|
|
16
|
+
Requires-Dist: num2words>=0.5.14
|
|
17
|
+
Description-Content-Type: text/markdown
|
|
18
|
+
|
|
19
|
+
# voxnorm
|
|
20
|
+
|
|
21
|
+
Text normalization for TTS: written forms into spoken forms, multilingual,
|
|
22
|
+
pure Python. Currently speaks **Traditional Chinese (zh-TW),
|
|
23
|
+
Simplified Chinese (zh-CN), English, Japanese, and Korean**.
|
|
24
|
+
|
|
25
|
+
Language models and databases write for the eye — `12:00`, `NT$1,200`,
|
|
26
|
+
`0912-345-678` — and a small TTS frontend handed that text verbatim has to
|
|
27
|
+
guess, and guesses badly. voxnorm rewrites the written forms into what a
|
|
28
|
+
speaker would actually say, and leaves everything else character-for-character
|
|
29
|
+
alone:
|
|
30
|
+
|
|
31
|
+
```python
|
|
32
|
+
>>> import voxnorm
|
|
33
|
+
>>> voxnorm.normalize("會議在12:00開始")
|
|
34
|
+
'會議在十二點開始'
|
|
35
|
+
>>> voxnorm.normalize("The meeting starts at 12:00")
|
|
36
|
+
"The meeting starts at twelve o'clock"
|
|
37
|
+
>>> voxnorm.normalize("会議は12:00からです") # kana → Japanese
|
|
38
|
+
'会議は十二時からです'
|
|
39
|
+
>>> voxnorm.normalize("회의는 12:00에 시작합니다") # hangul → Korean (native-numeral hours)
|
|
40
|
+
'회의는 열두 시에 시작합니다'
|
|
41
|
+
>>> voxnorm.normalize("電話是0912-345-678。")
|
|
42
|
+
'電話是零九一二三四五六七八。'
|
|
43
|
+
>>> voxnorm.normalize("NT$1,200", lang="zh")
|
|
44
|
+
'新台幣一千兩百元'
|
|
45
|
+
>>> voxnorm.normalize("会议在12:00开始") # simplified glyphs → zh-CN
|
|
46
|
+
'会议在十二点开始'
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
## Design
|
|
50
|
+
|
|
51
|
+
Two layers, split where languages stop sharing:
|
|
52
|
+
|
|
53
|
+
- **Detection is language-neutral** (`voxnorm/tokens.py`): one scanner finds
|
|
54
|
+
the spans — times, ISO dates, phone numbers, currency, percentages, room
|
|
55
|
+
numbers, decimals, integers — and names their kind.
|
|
56
|
+
- **Verbalisation is per-language** (`voxnorm/zh.py`, `cn.py`, `en.py`,
|
|
57
|
+
`ja.py`, `ko.py`): a new language is a new module and one registry entry. English,
|
|
58
|
+
Japanese and Korean ride on
|
|
59
|
+
[num2words](https://pypi.org/project/num2words/) for their number words,
|
|
60
|
+
each adding the readings the library cannot know — a Japanese year without
|
|
61
|
+
the era default, Korean native-numeral hours (한 시, never 일 시), the
|
|
62
|
+
irregular Korean months 유월/시월. Chinese is spelled out in-repo,
|
|
63
|
+
Traditional first, because the 兩/二 reading choice (一千兩百, never
|
|
64
|
+
一千二百 in zh-TW speech) is exactly what general-purpose converters get
|
|
65
|
+
wrong; the Simplified variant re-spells that one output vocabulary
|
|
66
|
+
(`兆` → `万亿`, and `¥` reads as renminbi rather than yen).
|
|
67
|
+
|
|
68
|
+
Language is decided per call: an explicit hint wins
|
|
69
|
+
(`normalize(text, lang="zh-TW")`), otherwise script evidence decides — kana
|
|
70
|
+
means Japanese, hangul Korean, a Han character Chinese (Simplified when
|
|
71
|
+
distinctive simplified glyphs appear, Traditional otherwise), else English. (A
|
|
72
|
+
Japanese sentence written entirely in kanji is script-identical to Chinese
|
|
73
|
+
and lands on the Chinese reading — pass `lang="ja"` when you know better.)
|
|
74
|
+
|
|
75
|
+
The charter is **structural only, never guess**: a form whose reading is
|
|
76
|
+
ambiguous does not convert (`1/2` the fraction and `8/17` the date are the
|
|
77
|
+
same slash string, so no slash form converts at all). What is not converted is
|
|
78
|
+
simply spoken as written — which is where it started.
|
|
79
|
+
|
|
80
|
+
## Why not an FST toolkit
|
|
81
|
+
|
|
82
|
+
nemo_text_processing and WeTextProcessing were measured before this package
|
|
83
|
+
was written. Both depend on pynini, which publishes no aarch64 wheel — on an
|
|
84
|
+
ARM box they need OpenFst compiled from source and a permanently vendored
|
|
85
|
+
`libfst.so` — and both mis-read the very classes this package exists for:
|
|
86
|
+
phone numbers as quantities, `302號房` as a date, and (WeTextProcessing)
|
|
87
|
+
force-simplifying every Traditional character in the sentence it touches. The
|
|
88
|
+
classes a spoken product needs are a bounded set; implementing them directly
|
|
89
|
+
costs ~3 MB of pure Python and stays debuggable.
|
|
90
|
+
|
|
91
|
+
## Install
|
|
92
|
+
|
|
93
|
+
```sh
|
|
94
|
+
uv add voxnorm # or: pip install voxnorm
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Development
|
|
98
|
+
|
|
99
|
+
```sh
|
|
100
|
+
uv sync
|
|
101
|
+
uv run pytest
|
|
102
|
+
uv run ruff check .
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
Apache-2.0.
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
voxnorm/__init__.py,sha256=u-G31qaUUVpCn-_zALxa0ZtQENsxdFIAOG1gLVvpDI8,6760
|
|
2
|
+
voxnorm/cn.py,sha256=m7917eMQw3bAH6XlA7pVUr-vBbMlY3Pqz-oofJp8_9A,1464
|
|
3
|
+
voxnorm/en.py,sha256=8NRmHa4fZXUKWIaynQAtwcnRjAIS6Id_Ws83n-TWpQw,3826
|
|
4
|
+
voxnorm/ja.py,sha256=8eK0A9iknaSw58rIL4qPmcq4ALJYHD4LQS8ktdVFGjg,3293
|
|
5
|
+
voxnorm/ko.py,sha256=vZ68X1z7gKyDAZkwS4KiFHA7NeoA4PAryiermj-U9Go,4028
|
|
6
|
+
voxnorm/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
|
7
|
+
voxnorm/tokens.py,sha256=mEsyblP8MofSJKLKFcXVmwhcVTXUu7hizBYfzYsPtLA,4551
|
|
8
|
+
voxnorm/zh.py,sha256=xjrNnFwASmx-odBHPQshSIL9AzqC8btGOqOLt4M9UsI,6328
|
|
9
|
+
voxnorm-0.1.0.dist-info/METADATA,sha256=xaV0NbQbReN8PwSrJIgySCX20YtptLSsprwOY28z27A,4479
|
|
10
|
+
voxnorm-0.1.0.dist-info/WHEEL,sha256=zOwg4jB6zX2kU910N-cMawjivD6tO8NEWvE12je1bVk,87
|
|
11
|
+
voxnorm-0.1.0.dist-info/licenses/LICENSE,sha256=xx0jnfkXJvxRnG63LTGOxlggYnIysveWIZ6H3PNdCrQ,11357
|
|
12
|
+
voxnorm-0.1.0.dist-info/RECORD,,
|
|
@@ -0,0 +1,201 @@
|
|
|
1
|
+
Apache License
|
|
2
|
+
Version 2.0, January 2004
|
|
3
|
+
http://www.apache.org/licenses/
|
|
4
|
+
|
|
5
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
6
|
+
|
|
7
|
+
1. Definitions.
|
|
8
|
+
|
|
9
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
|
10
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
|
11
|
+
|
|
12
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
|
13
|
+
the copyright owner that is granting the License.
|
|
14
|
+
|
|
15
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
|
16
|
+
other entities that control, are controlled by, or are under common
|
|
17
|
+
control with that entity. For the purposes of this definition,
|
|
18
|
+
"control" means (i) the power, direct or indirect, to cause the
|
|
19
|
+
direction or management of such entity, whether by contract or
|
|
20
|
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
21
|
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
22
|
+
|
|
23
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
|
24
|
+
exercising permissions granted by this License.
|
|
25
|
+
|
|
26
|
+
"Source" form shall mean the preferred form for making modifications,
|
|
27
|
+
including but not limited to software source code, documentation
|
|
28
|
+
source, and configuration files.
|
|
29
|
+
|
|
30
|
+
"Object" form shall mean any form resulting from mechanical
|
|
31
|
+
transformation or translation of a Source form, including but
|
|
32
|
+
not limited to compiled object code, generated documentation,
|
|
33
|
+
and conversions to other media types.
|
|
34
|
+
|
|
35
|
+
"Work" shall mean the work of authorship, whether in Source or
|
|
36
|
+
Object form, made available under the License, as indicated by a
|
|
37
|
+
copyright notice that is included in or attached to the work
|
|
38
|
+
(an example is provided in the Appendix below).
|
|
39
|
+
|
|
40
|
+
"Derivative Works" shall mean any work, whether in Source or Object
|
|
41
|
+
form, that is based on (or derived from) the Work and for which the
|
|
42
|
+
editorial revisions, annotations, elaborations, or other modifications
|
|
43
|
+
represent, as a whole, an original work of authorship. For the purposes
|
|
44
|
+
of this License, Derivative Works shall not include works that remain
|
|
45
|
+
separable from, or merely link (or bind by name) to the interfaces of,
|
|
46
|
+
the Work and Derivative Works thereof.
|
|
47
|
+
|
|
48
|
+
"Contribution" shall mean any work of authorship, including
|
|
49
|
+
the original version of the Work and any modifications or additions
|
|
50
|
+
to that Work or Derivative Works thereof, that is intentionally
|
|
51
|
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
52
|
+
or by an individual or Legal Entity authorized to submit on behalf of
|
|
53
|
+
the copyright owner. For the purposes of this definition, "submitted"
|
|
54
|
+
means any form of electronic, verbal, or written communication sent
|
|
55
|
+
to the Licensor or its representatives, including but not limited to
|
|
56
|
+
communication on electronic mailing lists, source code control systems,
|
|
57
|
+
and issue tracking systems that are managed by, or on behalf of, the
|
|
58
|
+
Licensor for the purpose of discussing and improving the Work, but
|
|
59
|
+
excluding communication that is conspicuously marked or otherwise
|
|
60
|
+
designated in writing by the copyright owner as "Not a Contribution."
|
|
61
|
+
|
|
62
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
63
|
+
on behalf of whom a Contribution has been received by Licensor and
|
|
64
|
+
subsequently incorporated within the Work.
|
|
65
|
+
|
|
66
|
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
67
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
68
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
69
|
+
copyright license to reproduce, prepare Derivative Works of,
|
|
70
|
+
publicly display, publicly perform, sublicense, and distribute the
|
|
71
|
+
Work and such Derivative Works in Source or Object form.
|
|
72
|
+
|
|
73
|
+
3. Grant of Patent License. Subject to the terms and conditions of
|
|
74
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
75
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
76
|
+
(except as stated in this section) patent license to make, have made,
|
|
77
|
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
78
|
+
where such license applies only to those patent claims licensable
|
|
79
|
+
by such Contributor that are necessarily infringed by their
|
|
80
|
+
Contribution(s) alone or by combination of their Contribution(s)
|
|
81
|
+
with the Work to which such Contribution(s) was submitted. If You
|
|
82
|
+
institute patent litigation against any entity (including a
|
|
83
|
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
84
|
+
or a Contribution incorporated within the Work constitutes direct
|
|
85
|
+
or contributory patent infringement, then any patent licenses
|
|
86
|
+
granted to You under this License for that Work shall terminate
|
|
87
|
+
as of the date such litigation is filed.
|
|
88
|
+
|
|
89
|
+
4. Redistribution. You may reproduce and distribute copies of the
|
|
90
|
+
Work or Derivative Works thereof in any medium, with or without
|
|
91
|
+
modifications, and in Source or Object form, provided that You
|
|
92
|
+
meet the following conditions:
|
|
93
|
+
|
|
94
|
+
(a) You must give any other recipients of the Work or
|
|
95
|
+
Derivative Works a copy of this License; and
|
|
96
|
+
|
|
97
|
+
(b) You must cause any modified files to carry prominent notices
|
|
98
|
+
stating that You changed the files; and
|
|
99
|
+
|
|
100
|
+
(c) You must retain, in the Source form of any Derivative Works
|
|
101
|
+
that You distribute, all copyright, patent, trademark, and
|
|
102
|
+
attribution notices from the Source form of the Work,
|
|
103
|
+
excluding those notices that do not pertain to any part of
|
|
104
|
+
the Derivative Works; and
|
|
105
|
+
|
|
106
|
+
(d) If the Work includes a "NOTICE" text file as part of its
|
|
107
|
+
distribution, then any Derivative Works that You distribute must
|
|
108
|
+
include a readable copy of the attribution notices contained
|
|
109
|
+
within such NOTICE file, excluding those notices that do not
|
|
110
|
+
pertain to any part of the Derivative Works, in at least one
|
|
111
|
+
of the following places: within a NOTICE text file distributed
|
|
112
|
+
as part of the Derivative Works; within the Source form or
|
|
113
|
+
documentation, if provided along with the Derivative Works; or,
|
|
114
|
+
within a display generated by the Derivative Works, if and
|
|
115
|
+
wherever such third-party notices normally appear. The contents
|
|
116
|
+
of the NOTICE file are for informational purposes only and
|
|
117
|
+
do not modify the License. You may add Your own attribution
|
|
118
|
+
notices within Derivative Works that You distribute, alongside
|
|
119
|
+
or as an addendum to the NOTICE text from the Work, provided
|
|
120
|
+
that such additional attribution notices cannot be construed
|
|
121
|
+
as modifying the License.
|
|
122
|
+
|
|
123
|
+
You may add Your own copyright statement to Your modifications and
|
|
124
|
+
may provide additional or different license terms and conditions
|
|
125
|
+
for use, reproduction, or distribution of Your modifications, or
|
|
126
|
+
for any such Derivative Works as a whole, provided Your use,
|
|
127
|
+
reproduction, and distribution of the Work otherwise complies with
|
|
128
|
+
the conditions stated in this License.
|
|
129
|
+
|
|
130
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
131
|
+
any Contribution intentionally submitted for inclusion in the Work
|
|
132
|
+
by You to the Licensor shall be under the terms and conditions of
|
|
133
|
+
this License, without any additional terms or conditions.
|
|
134
|
+
Notwithstanding the above, nothing herein shall supersede or modify
|
|
135
|
+
the terms of any separate license agreement you may have executed
|
|
136
|
+
with Licensor regarding such Contributions.
|
|
137
|
+
|
|
138
|
+
6. Trademarks. This License does not grant permission to use the trade
|
|
139
|
+
names, trademarks, service marks, or product names of the Licensor,
|
|
140
|
+
except as required for reasonable and customary use in describing the
|
|
141
|
+
origin of the Work and reproducing the content of the NOTICE file.
|
|
142
|
+
|
|
143
|
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
144
|
+
agreed to in writing, Licensor provides the Work (and each
|
|
145
|
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
146
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
147
|
+
implied, including, without limitation, any warranties or conditions
|
|
148
|
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
149
|
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
150
|
+
appropriateness of using or redistributing the Work and assume any
|
|
151
|
+
risks associated with Your exercise of permissions under this License.
|
|
152
|
+
|
|
153
|
+
8. Limitation of Liability. In no event and under no legal theory,
|
|
154
|
+
whether in tort (including negligence), contract, or otherwise,
|
|
155
|
+
unless required by applicable law (such as deliberate and grossly
|
|
156
|
+
negligent acts) or agreed to in writing, shall any Contributor be
|
|
157
|
+
liable to You for damages, including any direct, indirect, special,
|
|
158
|
+
incidental, or consequential damages of any character arising as a
|
|
159
|
+
result of this License or out of the use or inability to use the
|
|
160
|
+
Work (including but not limited to damages for loss of goodwill,
|
|
161
|
+
work stoppage, computer failure or malfunction, or any and all
|
|
162
|
+
other commercial damages or losses), even if such Contributor
|
|
163
|
+
has been advised of the possibility of such damages.
|
|
164
|
+
|
|
165
|
+
9. Accepting Warranty or Additional Liability. While redistributing
|
|
166
|
+
the Work or Derivative Works thereof, You may choose to offer,
|
|
167
|
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
168
|
+
or other liability obligations and/or rights consistent with this
|
|
169
|
+
License. However, in accepting such obligations, You may act only
|
|
170
|
+
on Your own behalf and on Your sole responsibility, not on behalf
|
|
171
|
+
of any other Contributor, and only if You agree to indemnify,
|
|
172
|
+
defend, and hold each Contributor harmless for any liability
|
|
173
|
+
incurred by, or claims asserted against, such Contributor by reason
|
|
174
|
+
of your accepting any such warranty or additional liability.
|
|
175
|
+
|
|
176
|
+
END OF TERMS AND CONDITIONS
|
|
177
|
+
|
|
178
|
+
APPENDIX: How to apply the Apache License to your work.
|
|
179
|
+
|
|
180
|
+
To apply the Apache License to your work, attach the following
|
|
181
|
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
182
|
+
replaced with your own identifying information. (Don't include
|
|
183
|
+
the brackets!) The text should be enclosed in the appropriate
|
|
184
|
+
comment syntax for the file format. We also recommend that a
|
|
185
|
+
file or class name and description of purpose be included on the
|
|
186
|
+
same "printed page" as the copyright notice for easier
|
|
187
|
+
identification within third-party archives.
|
|
188
|
+
|
|
189
|
+
Copyright [yyyy] [name of copyright owner]
|
|
190
|
+
|
|
191
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
192
|
+
you may not use this file except in compliance with the License.
|
|
193
|
+
You may obtain a copy of the License at
|
|
194
|
+
|
|
195
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
196
|
+
|
|
197
|
+
Unless required by applicable law or agreed to in writing, software
|
|
198
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
199
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
200
|
+
See the License for the specific language governing permissions and
|
|
201
|
+
limitations under the License.
|