xjyutping 1.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- xjyutping-1.1.0/CHANGELOG.md +280 -0
- xjyutping-1.1.0/LICENSE +48 -0
- xjyutping-1.1.0/MANIFEST.in +2 -0
- xjyutping-1.1.0/PKG-INFO +245 -0
- xjyutping-1.1.0/README.md +219 -0
- xjyutping-1.1.0/pyproject.toml +40 -0
- xjyutping-1.1.0/setup.cfg +4 -0
- xjyutping-1.1.0/src/xjyutping/__init__.py +250 -0
- xjyutping-1.1.0/src/xjyutping/data/chars.tsv +30089 -0
- xjyutping-1.1.0/src/xjyutping/data/finals.tsv +1 -0
- xjyutping-1.1.0/src/xjyutping/data/variants.tsv +76 -0
- xjyutping-1.1.0/src/xjyutping/data/words.tsv +103712 -0
- xjyutping-1.1.0/src/xjyutping.egg-info/PKG-INFO +245 -0
- xjyutping-1.1.0/src/xjyutping.egg-info/SOURCES.txt +19 -0
- xjyutping-1.1.0/src/xjyutping.egg-info/dependency_links.txt +1 -0
- xjyutping-1.1.0/src/xjyutping.egg-info/requires.txt +3 -0
- xjyutping-1.1.0/src/xjyutping.egg-info/top_level.txt +1 -0
- xjyutping-1.1.0/tests/parity.tex +14 -0
- xjyutping-1.1.0/tests/parity_corpus.txt +64 -0
- xjyutping-1.1.0/tests/parity_expected.txt +109 -0
- xjyutping-1.1.0/tests/test_xjyutping.py +304 -0
|
@@ -0,0 +1,280 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to xjyutping, the Python package, are recorded here. The
|
|
4
|
+
format follows [Keep a Changelog 1.1.0](https://keepachangelog.com/en/1.1.0/),
|
|
5
|
+
and versions follow [Semantic Versioning 2.0.0](https://semver.org/spec/v2.0.0.html).
|
|
6
|
+
|
|
7
|
+
This package is the Python port of the LaTeX package
|
|
8
|
+
[xjyutping](https://github.com/Beyond3345/xjyutping-tex), which has its own
|
|
9
|
+
version numbers and its own changelog. That changelog also holds the full
|
|
10
|
+
history of how both packages were built, including the review rounds whose
|
|
11
|
+
results this package inherits through the shared data. This file has the
|
|
12
|
+
Python releases and the handover notes for the Python side.
|
|
13
|
+
|
|
14
|
+
The version is written in three places: `version` in `pyproject.toml`,
|
|
15
|
+
`__version__` in `src/xjyutping/__init__.py` (`test_version` checks that the
|
|
16
|
+
two agree), and `README.md`. A release bumps all three and gets an entry in
|
|
17
|
+
Part I.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
# Part I: Releases
|
|
22
|
+
|
|
23
|
+
## [Unreleased]
|
|
24
|
+
|
|
25
|
+
Nothing yet. Add entries here under `### Added`, `### Changed`, `### Fixed`
|
|
26
|
+
and so on.
|
|
27
|
+
|
|
28
|
+
## [1.1.0] - 2026-09-28
|
|
29
|
+
|
|
30
|
+
### Added
|
|
31
|
+
|
|
32
|
+
- **More vocabulary:** the data of xjyutping-tex 1.2.0.
|
|
33
|
+
- 2,291 words from CC-Canto and the Cantonese readings of CC-CEDICT, as
|
|
34
|
+
distributed with Jyut Dictionary. They are added only where they change a
|
|
35
|
+
reading and pass consistency checks against rime-cantonese; 480 more
|
|
36
|
+
were excluded by hand.
|
|
37
|
+
- 640 characters that the other sources lack, with readings from the book
|
|
38
|
+
data of 粵音資料集叢 (jyutnet/cantonese-books-data, by 石見田).
|
|
39
|
+
|
|
40
|
+
On the corpora audited earlier nothing got worse; on new test sentences,
|
|
41
|
+
24 readings improved and 3 changed neutrally (see the LaTeX package's
|
|
42
|
+
changelog, Part II, section 7.4).
|
|
43
|
+
- `test_added_vocabulary`, which checks new words, the words that must not
|
|
44
|
+
capture their neighbours, the two hand fixes and a book character.
|
|
45
|
+
- README: an **Acknowledgements and attributions** section.
|
|
46
|
+
- The inspiration: [xpinyin](https://github.com/lxneng/xpinyin) by Eric Lo
|
|
47
|
+
([lxneng](https://lxneng.com)), whose API this package follows.
|
|
48
|
+
- Visual Jyutping and the Visual Cantonese Fonts, which inspired the
|
|
49
|
+
`fancy` and `discord` styles.
|
|
50
|
+
- Every data source: LSHK, rime-cantonese, 粵音資料集叢, Jyut Dictionary
|
|
51
|
+
with CC-Canto, the CC-CEDICT readings and CC-CEDICT, and OpenCC.
|
|
52
|
+
- This `CHANGELOG.md` and a `.gitignore`. Up to 1.0.0 both packages shared
|
|
53
|
+
one changelog in the folder that held them; the history is kept in
|
|
54
|
+
xjyutping-tex's.
|
|
55
|
+
|
|
56
|
+
### Changed
|
|
57
|
+
|
|
58
|
+
- **Readings**, as in xjyutping-tex 1.2.0, for example 機長 zoeng2, 營業額
|
|
59
|
+
ngaak2, 自由行 hang4, 研討會 wui2, 消防隊 deoi2, 冠狀病毒 gun1, 智能卡
|
|
60
|
+
kaat1. 化合物 is now faa3 hap6 mat6 and 會否 wui5 fau2.
|
|
61
|
+
- **Licence of the data files** in `src/xjyutping/data/`: CC BY-SA 4.0, since
|
|
62
|
+
the added word lists are CC BY-SA 3.0 (share-alike). Before this, the data
|
|
63
|
+
carried CC BY 4.0. `LICENSE` lists every source. The code stays MIT.
|
|
64
|
+
- `tests/parity.tex` compiles with XeLaTeX or LuaLaTeX.
|
|
65
|
+
- `tests/parity_expected.txt` was regenerated from xjyutping-tex 1.2.0 and
|
|
66
|
+
changed in one line: 隨着 is now one word, with the same readings.
|
|
67
|
+
- README: shorter, with installing from PyPI or GitHub, and links to the LaTeX
|
|
68
|
+
package's repository. The PyPI summary is now "Convert Traditional Chinese
|
|
69
|
+
text to Cantonese Jyutping (粵拼), with pronunciation chosen from context".
|
|
70
|
+
- **Packaging for PyPI:** `pyproject.toml` declares the licence as the SPDX
|
|
71
|
+
expression `MIT AND CC-BY-SA-4.0` with `license-files` (PEP 639, hence
|
|
72
|
+
setuptools 77 or later to build), and has keywords, classifiers and the
|
|
73
|
+
project URLs (repository, issues, changelog, the LaTeX package).
|
|
74
|
+
`MANIFEST.in` adds `CHANGELOG.md` and the parity files to the sdist. The
|
|
75
|
+
sdist and wheel pass `twine check`.
|
|
76
|
+
|
|
77
|
+
## [1.0.0] - 2026-09-28
|
|
78
|
+
|
|
79
|
+
### Added
|
|
80
|
+
|
|
81
|
+
- **First release** of the Python package `xjyutping`: the Python port of
|
|
82
|
+
the LaTeX package, with the API of
|
|
83
|
+
[xpinyin](https://github.com/lxneng/xpinyin). Python 3.8 or later, no
|
|
84
|
+
dependencies. Class `Jyutping(data_dir=None)`:
|
|
85
|
+
- `get_jyutping(chars, splitter='-', tone_marks='numbers', convert='lower')`
|
|
86
|
+
- `get_jyutpings(chars, splitter, tone_marks, convert, n=10)`: the reading
|
|
87
|
+
in context first, then the other readings.
|
|
88
|
+
- `annotate(text, tone_marks)`: one `(char, reading or None)` pair per
|
|
89
|
+
input character.
|
|
90
|
+
- `segment(text)`: `Segment(text, readings, type)` with the types w/u/m/s
|
|
91
|
+
of the TeX debug log.
|
|
92
|
+
- `set_jyutping(text, reading)`, the equivalent of `\setjyutping`.
|
|
93
|
+
- `get_initial(char, full=False)` and `get_initials(chars, splitter, full)`.
|
|
94
|
+
- The static methods `decode_jyutping` and `convert_jyutping`.
|
|
95
|
+
- **Tone styles:** `'numbers'` (default), `None`, `'fancy'` and `'discord'`.
|
|
96
|
+
The last two use the exact code points of Visual Jyutping's web and
|
|
97
|
+
Discord tone maps: ˉ¹ ˊ² ˗₃ ˎ₄ ˏ₅ ˍ₆ and ˉ¹ ⸍² -₃ ⸜₄ ⸝₅ ˍ₆.
|
|
98
|
+
- **Same readings as xjyutping-tex 1.1.0.** It uses the same data
|
|
99
|
+
(`src/xjyutping/data/*.tsv`, generated by that package's
|
|
100
|
+
`tools/build-data.py`) and the same run splitting, segmentation, variant
|
|
101
|
+
folding, run-final readings and user overrides.
|
|
102
|
+
- **Tests:** 38 pytest tests, including a parity test against a frozen TeX
|
|
103
|
+
debug log (`tests/parity.tex`, `parity_corpus.txt`,
|
|
104
|
+
`parity_expected.txt`). There are 109 runs in the log and no mismatches.
|
|
105
|
+
- MIT licence for the code; the data carried CC BY 4.0 (LSHK,
|
|
106
|
+
rime-cantonese) and Apache-2.0 (OpenCC).
|
|
107
|
+
|
|
108
|
+
---
|
|
109
|
+
|
|
110
|
+
# Part II: Handover notes
|
|
111
|
+
|
|
112
|
+
## 0. Start here
|
|
113
|
+
|
|
114
|
+
**Where things are:**
|
|
115
|
+
|
|
116
|
+
| Path | What it is |
|
|
117
|
+
| --- | --- |
|
|
118
|
+
| `src/xjyutping/__init__.py` | The whole package: `Jyutping`, `Segment`, `TONE_MARKS`. |
|
|
119
|
+
| `src/xjyutping/data/*.tsv` | Generated data (see below); never edit by hand. |
|
|
120
|
+
| `tests/test_xjyutping.py` | pytest suite (39 tests). |
|
|
121
|
+
| `tests/parity_corpus.txt`, `parity_expected.txt`, `parity.tex` | The parity test with the LaTeX package: the corpus, the frozen TeX debug log, and the document that regenerates the log. |
|
|
122
|
+
| `pyproject.toml`, `MANIFEST.in` | Packaging (setuptools; the TSV files are package data; the sdist includes the parity files). |
|
|
123
|
+
| `README.md`, `LICENSE`, `CHANGELOG.md` | Documentation, licences (code MIT, data CC BY-SA 4.0), this file. |
|
|
124
|
+
|
|
125
|
+
**The data** is generated, together with the LaTeX package's `.def` files,
|
|
126
|
+
by `tools/build-data.py` in
|
|
127
|
+
[xjyutping-tex](https://github.com/Beyond3345/xjyutping-tex). It writes here
|
|
128
|
+
when this repository sits next to xjyutping-tex (or with
|
|
129
|
+
`--py-data DIR`). The files are:
|
|
130
|
+
|
|
131
|
+
- `chars.tsv`: character, default reading, other readings, polyphone flag;
|
|
132
|
+
- `finals.tsv`: reading at the end of a run;
|
|
133
|
+
- `variants.tsv`: variant, canonical character;
|
|
134
|
+
- `words.tsv`: word, readings.
|
|
135
|
+
|
|
136
|
+
The sources and every rule for choosing readings are described in that
|
|
137
|
+
repository's README and changelog.
|
|
138
|
+
|
|
139
|
+
**Invariants to keep:**
|
|
140
|
+
|
|
141
|
+
1. **The same readings as the LaTeX package.**
|
|
142
|
+
`test_parity_with_tex_package` compares every segment, reading and type of
|
|
143
|
+
`tests/parity_corpus.txt` with the LaTeX package's debug log.
|
|
144
|
+
- When the data or the segmentation changes on purpose, regenerate
|
|
145
|
+
`parity_expected.txt`: compile `tests/parity.tex` with
|
|
146
|
+
`TEXINPUTS=<xjyutping-tex>: LUAINPUTS=<xjyutping-tex>:` and
|
|
147
|
+
`max_print_line=10000`, then copy the `xjyutping>` lines. Check the diff
|
|
148
|
+
by hand.
|
|
149
|
+
2. **Reading fixes belong in xjyutping-tex's `tools/build-data.py`**, never
|
|
150
|
+
in the TSV files or the code.
|
|
151
|
+
3. **The algorithm mirrors the TeX package** (section 2). A change on one
|
|
152
|
+
side needs the same change on the other.
|
|
153
|
+
4. **Python 3.8 compatibility:** no `match`, no `X | Y` unions evaluated at
|
|
154
|
+
runtime, no 3.9+ string methods.
|
|
155
|
+
5. **Versioning** follows SemVer:
|
|
156
|
+
- MAJOR: an incompatible change of the API (a method, argument or return
|
|
157
|
+
value);
|
|
158
|
+
- MINOR: a backward-compatible feature, or new vocabulary (1.1.0);
|
|
159
|
+
- PATCH: bug fixes and reading corrections.
|
|
160
|
+
|
|
161
|
+
**How to test:**
|
|
162
|
+
|
|
163
|
+
```bash
|
|
164
|
+
python3 -m pip install --upgrade pip && pip install -e '.[test]' && pytest tests
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
Without installing, run `PYTHONPATH=src python3 -m pytest tests`. All tests
|
|
168
|
+
must pass, including `test_parity_with_tex_package`. To check a release,
|
|
169
|
+
build the sdist and wheel and run the tests from the extracted sdist.
|
|
170
|
+
|
|
171
|
+
## 1. How the port was built (with xjyutping-tex 1.1.0)
|
|
172
|
+
|
|
173
|
+
The port was written by an agent against a frozen copy of the LaTeX package
|
|
174
|
+
1.0.0. It took xpinyin's API and reproduced the LaTeX package's algorithm:
|
|
175
|
+
runs, segmentation by dynamic programming, word and character lookup, and
|
|
176
|
+
user settings. The parity with TeX was then checked on a 668-character
|
|
177
|
+
corpus: 0 mismatches.
|
|
178
|
+
|
|
179
|
+
Three independent reviewers then checked it, each from one angle:
|
|
180
|
+
|
|
181
|
+
- reading parity with TeX, on a new 1 021-character corpus: 119 runs
|
|
182
|
+
compared line for line;
|
|
183
|
+
- API, packaging and docs: the README examples run as doctests; wheel and
|
|
184
|
+
sdist contents;
|
|
185
|
+
- edge cases and tone styles.
|
|
186
|
+
|
|
187
|
+
Every finding was reproduced before it was fixed. The author's own 35 tests
|
|
188
|
+
had passed.
|
|
189
|
+
|
|
190
|
+
- **P-1. CR line ends** (reported by all three reviewers). A lone CR was not
|
|
191
|
+
counted as a line end, so `'\r\r'`, a blank line to TeX, did not end a
|
|
192
|
+
run. That changed segmentation and the run-final 呢.
|
|
193
|
+
- Fix: `_runs` counts `\n`, and `\r` not followed by `\n`.
|
|
194
|
+
- Test: `test_runs_cr_line_ends`, with expected values from a TeX log.
|
|
195
|
+
- **P-2. The sdist lacked the parity files,** so its own test suite failed
|
|
196
|
+
(2 tests, FileNotFoundError). Fix: `MANIFEST.in` includes `tests/*.txt`
|
|
197
|
+
and `*.tex`.
|
|
198
|
+
- **P-3. The README's editable install failed** with pip 21.2.4, since
|
|
199
|
+
editable installs need pip 21.3 or later. The README now upgrades pip
|
|
200
|
+
first.
|
|
201
|
+
- **P-4. `license = {text = "MIT"}` in `pyproject.toml`** triggered a
|
|
202
|
+
setuptools deprecation that becomes a build error after 2027-02-18.
|
|
203
|
+
Removed; `License-File` and the classifier remain.
|
|
204
|
+
- **P-5. The README's run-final example showed a word-list entry** instead
|
|
205
|
+
of the run-final rule. It was replaced with 呢張床好平,好麻煩呢?
|
|
206
|
+
- **P-6. `set_jyutping` on a character missing from the data did nothing.**
|
|
207
|
+
TeX reads such a character without context and with type `u`. It is now
|
|
208
|
+
its own run of type `u`; test `test_set_jyutping_char_not_in_data`.
|
|
209
|
+
- **P-7. `set_jyutping` rejected whitespace** that `\setjyutping` ignores.
|
|
210
|
+
It now strips spaces, tabs, CR and LF; test
|
|
211
|
+
`test_set_jyutping_ignores_spaces`.
|
|
212
|
+
|
|
213
|
+
Timing on the development Mac (Python 3.9):
|
|
214
|
+
|
|
215
|
+
- `Jyutping()` takes about 0.08 s;
|
|
216
|
+
- `annotate` on 10 000 characters takes about 0.02 s.
|
|
217
|
+
|
|
218
|
+
## 2. How the port maps to the LaTeX package
|
|
219
|
+
|
|
220
|
+
| TeX (`xjyutping.sty`) | Python (`src/xjyutping/__init__.py`) |
|
|
221
|
+
| --- | --- |
|
|
222
|
+
| a character is Chinese iff `\xjp@c@<char>` exists | `c in self._chars` |
|
|
223
|
+
| runs: spaces and single line breaks between characters continue a run; `\par`/blank line ends it | `_runs` (ASCII space, tab, CR, LF; two line ends end the run; U+3000 ends it) |
|
|
224
|
+
| `\__xjyutping_flush:` (DP, 100000 per segment + 1 per single, `<=` so longer final word wins; bound `\xjp@e@` raw or canonical) | `_segment_run` (same costs and comparison; bound `_longest`, computed from `words.tsv` at load) |
|
|
225
|
+
| `\__xjyutping_emit:nn` word: raw key, else canonical; type `u` if `\xjp@uw@` | same, `self._user_words` |
|
|
226
|
+
| `\__xjyutping_lookup:nn` single: user, then `\xjp@f@` at run end, then default; type m/s from `\xjp@m@` | same (`_user_chars`, `_finals`, polyphone flag in `chars.tsv`) |
|
|
227
|
+
| `\__xjyutping_set:nnn` / `\__xjyutping_set_word:nn` | `set_jyutping` (raw and canonical keys, longest bound) |
|
|
228
|
+
| `debug` log `銀ngan4:w…` | `segment()` |
|
|
229
|
+
|
|
230
|
+
Differences by design:
|
|
231
|
+
|
|
232
|
+
- Python has no markup, so it has nothing like the command table, footnote
|
|
233
|
+
runs, `\xjyutping` manual readings or plain mode. The parity corpus
|
|
234
|
+
therefore has TeX's commands resolved into the runs TeX makes of them.
|
|
235
|
+
- `chars.tsv` lists the other readings of every character (for
|
|
236
|
+
`get_jyutpings`), not only of flagged polyphones.
|
|
237
|
+
- `Segment` has no field for the other readings. The parity test reads them
|
|
238
|
+
from `_chars`; users get them through `get_jyutpings`.
|
|
239
|
+
- `get_jyutping` drops whitespace between Chinese characters, U+3000
|
|
240
|
+
included, although U+3000 ends a run for segmentation.
|
|
241
|
+
|
|
242
|
+
## 3. Version 1.1.0
|
|
243
|
+
|
|
244
|
+
Data only, plus documentation.
|
|
245
|
+
|
|
246
|
+
- The data of xjyutping-tex 1.2.0 was copied in by its build: 640 more
|
|
247
|
+
characters, 2 291 more words and the two hand fixes.
|
|
248
|
+
- The parity file was regenerated with both TeX engines, which agree.
|
|
249
|
+
- `test_added_vocabulary` was added. The README examples still run
|
|
250
|
+
unchanged (24 doctests). `python -m doctest README.md` counts the closing
|
|
251
|
+
code fence as expected output, so strip the fences first:
|
|
252
|
+
|
|
253
|
+
```bash
|
|
254
|
+
PYTHONPATH=src python3 -c "import doctest,re; t=re.sub(r'(?m)^\x60{3}.*$','',open('README.md',encoding='utf8').read()); r=doctest.DocTestRunner(optionflags=doctest.NORMALIZE_WHITESPACE); r.run(doctest.DocTestParser().get_doctest(t,{},'README','README.md',0)); print(r.summarize())"
|
|
255
|
+
```
|
|
256
|
+
- The code did not change.
|
|
257
|
+
- **Release to PyPI** (the name `xjyutping` was free on 2026-09-28):
|
|
258
|
+
|
|
259
|
+
```bash
|
|
260
|
+
python3 -m pip install --upgrade build twine
|
|
261
|
+
python3 -m build
|
|
262
|
+
python3 -m twine check dist/*
|
|
263
|
+
python3 -m twine upload dist/*
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
The 1.1.0 wheel and sdist were built this way and checked: `twine check`
|
|
267
|
+
passes, the wheel installs into a clean virtual environment, and the 39
|
|
268
|
+
tests pass from the extracted sdist. `twine upload` asks for a PyPI API
|
|
269
|
+
token (user name `__token__`); upload to TestPyPI first with
|
|
270
|
+
`--repository testpypi` if in doubt.
|
|
271
|
+
|
|
272
|
+
## 4. Open issues
|
|
273
|
+
|
|
274
|
+
- **Readings that need more context than a word list gives** stay wrong
|
|
275
|
+
until a user sets them. The list is in the LaTeX package's changelog, Part
|
|
276
|
+
II, section 4, item 14, still open after its 1.2.0 (section 7.7), e.g. 為
|
|
277
|
+
wai4/wai6, 同行, 種花, 長得, 重未.
|
|
278
|
+
- **The book characters' licence.** The 640 characters from 粵音資料集叢
|
|
279
|
+
come from data published without a licence statement; see `LICENSE` and
|
|
280
|
+
the LaTeX package's changelog, section 7.4.
|
xjyutping-1.1.0/LICENSE
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 the xjyutping authors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
22
|
+
|
|
23
|
+
-------------------------------------------------------------------------------
|
|
24
|
+
Third-party material
|
|
25
|
+
|
|
26
|
+
The files in src/xjyutping/data/ are generated by tools/build-data.py of
|
|
27
|
+
the LaTeX package (https://github.com/Beyond3345/xjyutping-tex) and are
|
|
28
|
+
distributed under Creative Commons Attribution-ShareAlike 4.0
|
|
29
|
+
(CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/), because
|
|
30
|
+
they adapt share-alike material. They are built from:
|
|
31
|
+
|
|
32
|
+
* the LSHK Cantonese Pronunciation List of Characters for Computers
|
|
33
|
+
(jyutping-table), Creative Commons Attribution 4.0 (CC BY 4.0);
|
|
34
|
+
* rime-cantonese jyut6ping3.chars and jyut6ping3.words, Creative Commons
|
|
35
|
+
Attribution 4.0 (CC BY 4.0);
|
|
36
|
+
* CC-Canto, Copyright (c) 2015-17 Pleco Inc., and the CC-CEDICT Cantonese
|
|
37
|
+
readings, Copyright (c) 2015 Pleco Software Inc., both Creative Commons
|
|
38
|
+
Attribution-ShareAlike 3.0 (CC BY-SA 3.0), as distributed with Jyut
|
|
39
|
+
Dictionary (https://github.com/aaronhktan/jyut-dict);
|
|
40
|
+
* OpenCC HKVariants.txt and TWVariants.txt, Apache License 2.0;
|
|
41
|
+
* readings of 640 characters from the book data of 粵音資料集叢 by 石見田
|
|
42
|
+
(https://github.com/jyutnet/cantonese-books-data), published without a
|
|
43
|
+
licence statement and used with attribution.
|
|
44
|
+
|
|
45
|
+
The 'fancy' and 'discord' tone symbols (TONE_MARKS in
|
|
46
|
+
src/xjyutping/__init__.py) come from Visual Jyutping
|
|
47
|
+
(https://github.com/VincentTam/visual-jyutping), MIT License,
|
|
48
|
+
Copyright (c) 2024 Vincent Tam.
|
xjyutping-1.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,245 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: xjyutping
|
|
3
|
+
Version: 1.1.0
|
|
4
|
+
Summary: Convert Traditional Chinese text to Cantonese Jyutping (粵拼), with pronunciation chosen from context
|
|
5
|
+
License-Expression: MIT AND CC-BY-SA-4.0
|
|
6
|
+
Project-URL: Homepage, https://github.com/Beyond3345/xjyutping-py
|
|
7
|
+
Project-URL: Repository, https://github.com/Beyond3345/xjyutping-py
|
|
8
|
+
Project-URL: Issues, https://github.com/Beyond3345/xjyutping-py/issues
|
|
9
|
+
Project-URL: Changelog, https://github.com/Beyond3345/xjyutping-py/blob/main/CHANGELOG.md
|
|
10
|
+
Project-URL: LaTeX package, https://github.com/Beyond3345/xjyutping-tex
|
|
11
|
+
Keywords: jyutping,cantonese,romanization,romanisation,粵拼,粵語,chinese
|
|
12
|
+
Classifier: Development Status :: 4 - Beta
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: Intended Audience :: Education
|
|
15
|
+
Classifier: Natural Language :: Chinese (Traditional)
|
|
16
|
+
Classifier: Operating System :: OS Independent
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
19
|
+
Classifier: Topic :: Text Processing :: Linguistic
|
|
20
|
+
Requires-Python: >=3.8
|
|
21
|
+
Description-Content-Type: text/markdown
|
|
22
|
+
License-File: LICENSE
|
|
23
|
+
Provides-Extra: test
|
|
24
|
+
Requires-Dist: pytest>=6.2; extra == "test"
|
|
25
|
+
Dynamic: license-file
|
|
26
|
+
|
|
27
|
+
# xjyutping (Python)
|
|
28
|
+
|
|
29
|
+
Version 1.1.0 (2026-09-28). Versions follow
|
|
30
|
+
[Semantic Versioning](https://semver.org), and the history is in
|
|
31
|
+
[`CHANGELOG.md`](https://github.com/Beyond3345/xjyutping-py/blob/main/CHANGELOG.md).
|
|
32
|
+
|
|
33
|
+
xjyutping converts Traditional Chinese text to Cantonese Jyutping (粵拼).
|
|
34
|
+
Pronunciation is determined by looking at the context and cross-referencing
|
|
35
|
+
it with a list of about 104 000 words, so 行 is *hong4* in 銀行 but *haang4*
|
|
36
|
+
in 行路. It is the Python version of the LaTeX package
|
|
37
|
+
[xjyutping](https://github.com/Beyond3345/xjyutping-tex) and gives the same
|
|
38
|
+
readings. It was inspired by [xpinyin](https://github.com/lxneng/xpinyin),
|
|
39
|
+
which converts Chinese characters to Mandarin pinyin, and follows its API.
|
|
40
|
+
|
|
41
|
+
## Install
|
|
42
|
+
|
|
43
|
+
Python 3.8 or later, no dependencies.
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
pip install xjyutping
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
or from GitHub with `pip install git+https://github.com/Beyond3345/xjyutping-py`.
|
|
50
|
+
For development (editable installs need pip 21.3 or later):
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
python3 -m pip install --upgrade pip && pip install -e '.[test]' && pytest tests
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Usage
|
|
57
|
+
|
|
58
|
+
```pycon
|
|
59
|
+
>>> from xjyutping import Jyutping
|
|
60
|
+
>>> j = Jyutping()
|
|
61
|
+
>>> j.get_jyutping('我哋去銀行') # default splitter is '-'
|
|
62
|
+
'ngo5-dei6-heoi3-ngan4-hong4'
|
|
63
|
+
>>> j.get_jyutping('行路返屋企', ' ')
|
|
64
|
+
'haang4 lou6 faan1 uk1 kei2'
|
|
65
|
+
>>> j.get_jyutping('香港', '')
|
|
66
|
+
'hoeng1gong2'
|
|
67
|
+
>>> j.get_jyutping('香港', convert='capitalize') # or 'upper'; default 'lower'
|
|
68
|
+
'Hoeng1-Gong2'
|
|
69
|
+
>>> j.get_jyutping('Apple發布iOS7') # other text is kept as it is
|
|
70
|
+
'Apple-faat3-bou3-iOS7'
|
|
71
|
+
>>> j.get_jyutping('你好 世界') # spaces between characters are dropped
|
|
72
|
+
'nei5-hou2-sai3-gaai3'
|
|
73
|
+
>>> j.annotate('銀行 ATM') # one pair per input character
|
|
74
|
+
[('銀', 'ngan4'), ('行', 'hong4'), (' ', None), ('A', None), ('T', None), ('M', None)]
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
### Tone styles
|
|
78
|
+
|
|
79
|
+
`tone_marks` works with `get_jyutping`, `get_jyutpings` and `annotate`:
|
|
80
|
+
|
|
81
|
+
| `tone_marks` | 詩史試時市事 |
|
|
82
|
+
| --- | --- |
|
|
83
|
+
| `'numbers'` (default) | si1 si2 si3 si4 si5 si6 |
|
|
84
|
+
| `None` | si si si si si si |
|
|
85
|
+
| `'fancy'` | siˉ¹ siˊ² si˗₃ siˎ₄ siˏ₅ siˍ₆ |
|
|
86
|
+
| `'discord'` | siˉ¹ si⸍² si-₃ si⸜₄ si⸝₅ siˍ₆ |
|
|
87
|
+
|
|
88
|
+
`'fancy'` and `'discord'` use the tone symbols of
|
|
89
|
+
[Visual Jyutping](https://github.com/VincentTam/visual-jyutping), from its web
|
|
90
|
+
and Discord versions. Any other value raises `ValueError`.
|
|
91
|
+
|
|
92
|
+
```pycon
|
|
93
|
+
>>> j.get_jyutping('行路', ' ', tone_marks='fancy')
|
|
94
|
+
'haangˎ₄ louˍ₆'
|
|
95
|
+
>>> Jyutping.decode_jyutping('hou2', 'fancy')
|
|
96
|
+
'houˊ²'
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
### Context and your own readings
|
|
100
|
+
|
|
101
|
+
Text is split into runs of Chinese characters, and each run is split into
|
|
102
|
+
the fewest words from the word list. A word takes its reading from the list,
|
|
103
|
+
and a character on its own takes your reading or else its default. 呢 reads
|
|
104
|
+
*ni1* inside a run but *ne1* at its end. Variant shapes such as 為/爲 and
|
|
105
|
+
裡/裏 find the same words.
|
|
106
|
+
|
|
107
|
+
```pycon
|
|
108
|
+
>>> j.get_jyutping('校長喺學校長大')
|
|
109
|
+
'haau6-zoeng2-hai2-hok6-haau6-zoeng2-daai6'
|
|
110
|
+
>>> j.get_jyutping('呢張床好平,好麻煩呢?', ' ')
|
|
111
|
+
'ni1 zoeng1 cong4 hou2 peng4 , hou2 maa4 faan4 ne1 ?'
|
|
112
|
+
>>> j.segment('佢重話聽日嚟')
|
|
113
|
+
[Segment(text='佢', readings=['keoi5'], type='s'), Segment(text='重', readings=['cung5'], type='m'), Segment(text='話', readings=['waa6'], type='s'), Segment(text='聽日', readings=['ting1', 'jat6'], type='w'), Segment(text='嚟', readings=['lai4'], type='m')]
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
`segment` shows how a text was read, like the LaTeX package's `debug`
|
|
117
|
+
option. The types are `w` for a word from the list, `u` for your setting, `m`
|
|
118
|
+
for a guessed reading of a character with several, and `s` for a character
|
|
119
|
+
with one reading.
|
|
120
|
+
|
|
121
|
+
`set_jyutping` works like `\setjyutping` in LaTeX. One character gets a new
|
|
122
|
+
default reading, and several characters (one syllable each) become a word. A
|
|
123
|
+
syllable count that doesn't match raises `ValueError`. Settings belong to the
|
|
124
|
+
`Jyutping` instance.
|
|
125
|
+
|
|
126
|
+
```pycon
|
|
127
|
+
>>> j.set_jyutping('重話', 'zung6 waa6')
|
|
128
|
+
>>> j.get_jyutping('佢重話聽日嚟')
|
|
129
|
+
'keoi5-zung6-waa6-ting1-jat6-lai4'
|
|
130
|
+
>>> j.set_jyutping('重', 'zung6')
|
|
131
|
+
>>> j.get_jyutping('佢重未嚟')
|
|
132
|
+
'keoi5-zung6-mei6-lai4'
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
### Combinations
|
|
136
|
+
|
|
137
|
+
`get_jyutpings` lists up to `n` (default 10) combinations of readings, with
|
|
138
|
+
the reading in context first.
|
|
139
|
+
|
|
140
|
+
```pycon
|
|
141
|
+
>>> j.get_jyutpings('行')
|
|
142
|
+
['haang4', 'hang4', 'hong4', 'hong2', 'hang6']
|
|
143
|
+
>>> j.get_jyutpings('銀行', n=3)
|
|
144
|
+
['ngan4-hong4', 'ngan4-haang4', 'ngan4-hang4']
|
|
145
|
+
>>> j.get_jyutpings('行', tone_marks=None)
|
|
146
|
+
['haang', 'hang', 'hong']
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
### Initials
|
|
150
|
+
|
|
151
|
+
`get_initial` and `get_initials` give the first letter of each reading in
|
|
152
|
+
capitals, and leave other text unchanged. With `full=True` they give the
|
|
153
|
+
whole initial (`NG`, `GW`, `KW` ...), or an empty string for a syllable
|
|
154
|
+
without one, such as 屋 *uk1* or 唔 *m4*.
|
|
155
|
+
|
|
156
|
+
```pycon
|
|
157
|
+
>>> j.get_initials('我哋去廣州')
|
|
158
|
+
'N-D-H-G-Z'
|
|
159
|
+
>>> j.get_initials('我哋去廣州', full=True)
|
|
160
|
+
'NG-D-H-GW-Z'
|
|
161
|
+
>>> j.get_initials('唔五屋'), j.get_initials('唔五屋', full=True)
|
|
162
|
+
('M-N-U', '--')
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
## API
|
|
166
|
+
|
|
167
|
+
| | |
|
|
168
|
+
| --- | --- |
|
|
169
|
+
| `Jyutping(data_dir=None)` | Loads the data (from the package unless `data_dir` is given). |
|
|
170
|
+
| `get_jyutping(chars, splitter='-', tone_marks='numbers', convert='lower')` | Jyutping of a text. |
|
|
171
|
+
| `get_jyutpings(chars, splitter='-', tone_marks='numbers', convert='lower', n=10)` | Combinations of readings. |
|
|
172
|
+
| `annotate(text, tone_marks='numbers')` | `[(char, reading or None), ...]`, one per character. |
|
|
173
|
+
| `segment(text)` | `[Segment(text, readings, type), ...]` for the Chinese text. |
|
|
174
|
+
| `set_jyutping(text, reading)` | Your reading for a character or a word. |
|
|
175
|
+
| `get_initial(char, full=False)`, `get_initials(chars, splitter='-', full=False)` | Initials. |
|
|
176
|
+
| `Jyutping.decode_jyutping(syllable, tone_marks)`, `Jyutping.convert_jyutping(syllable, convert)` | Format one syllable. |
|
|
177
|
+
|
|
178
|
+
## Data
|
|
179
|
+
|
|
180
|
+
The data in `src/xjyutping/data/` (30 089 characters, about 104 000 words) is
|
|
181
|
+
generated by `tools/build-data.py` in
|
|
182
|
+
[xjyutping-tex](https://github.com/Beyond3345/xjyutping-tex), from the
|
|
183
|
+
sources credited below. Fix readings there, not in the TSV files.
|
|
184
|
+
|
|
185
|
+
## Tests
|
|
186
|
+
|
|
187
|
+
`pytest tests` runs the tests. One of them checks that the readings match
|
|
188
|
+
the LaTeX package's debug log (`tests/parity_expected.txt`), which
|
|
189
|
+
`tests/parity.tex` regenerates.
|
|
190
|
+
|
|
191
|
+
## Acknowledgements and attributions
|
|
192
|
+
|
|
193
|
+
This project was inspired by [xpinyin](https://github.com/lxneng/xpinyin) by
|
|
194
|
+
Eric Lo ([lxneng](https://lxneng.com)), a Python package that converts
|
|
195
|
+
Chinese characters to pinyin. Its API is the model for this one. The LaTeX
|
|
196
|
+
version of xjyutping was inspired by the LaTeX package
|
|
197
|
+
[xpinyin](https://ctan.org/pkg/xpinyin) by Qing Lee (李清).
|
|
198
|
+
|
|
199
|
+
The `'fancy'` and `'discord'` styles use the tone symbols of
|
|
200
|
+
[Visual Jyutping](https://github.com/VincentTam/visual-jyutping) by Vincent
|
|
201
|
+
Tam, which was itself inspired by the Visual Cantonese Fonts (粵語字體) by Jon
|
|
202
|
+
Chui / A3I Ltd. ([canto.hk](https://canto.hk),
|
|
203
|
+
[documentation](https://docs.visual-fonts.com),
|
|
204
|
+
[source](https://github.com/jkwchui/visual-fonts-starlight-docs)). Thank you
|
|
205
|
+
both.
|
|
206
|
+
|
|
207
|
+
Thanks also to the authors of the data sources:
|
|
208
|
+
|
|
209
|
+
* the Jyutping Workgroup of the Linguistic Society of Hong Kong, for the
|
|
210
|
+
*Cantonese Pronunciation List of Characters for Computers*
|
|
211
|
+
(電腦用漢字粵語拼音表,
|
|
212
|
+
[lshk-org/jyutping-table](https://github.com/lshk-org/jyutping-table),
|
|
213
|
+
CC BY 4.0), and to Prof Lu Qin and Dr Cheung Kwan Hin of the Hong Kong
|
|
214
|
+
Polytechnic University and Nathan Hammond, who are thanked there;
|
|
215
|
+
* the Cantonese Computational Linguistics Infrastructure Development
|
|
216
|
+
Workgroup (CanCLID) and contributors, for
|
|
217
|
+
[rime-cantonese](https://github.com/rime/rime-cantonese) (粵語拼音輸入方案,
|
|
218
|
+
CC BY 4.0), which gives the default readings and most of the words;
|
|
219
|
+
* 石見田, for 粵音資料集叢 ([jyut.net](https://jyut.net/about), data at
|
|
220
|
+
[jyutnet/cantonese-books-data](https://github.com/jyutnet/cantonese-books-data)),
|
|
221
|
+
and the authors and editors of the dictionaries it digitises: 廣州話正音字典
|
|
222
|
+
(2004), 廣州話標準音字彙 (1988), 粵語同音字典 (1974/1996), 粵語查音識字字典
|
|
223
|
+
(1985), 同音字彙 (1971), 部身字典 (1967), *The Student's Cantonese-English
|
|
224
|
+
Dictionary* (1947), 粵音韻彙 (1941), 道字典 (1941), 道漢字音 (1939),
|
|
225
|
+
民眾識字粵語拼音字彙 (1931), 廣話國語一貫未定稿 (1916) and
|
|
226
|
+
分部分音廣話九聲字宗 (1914);
|
|
227
|
+
* Aaron Tan, for [Jyut Dictionary](https://github.com/aaronhktan/jyut-dict),
|
|
228
|
+
which distributes CC-Canto (© 2015–17 Pleco Inc.,
|
|
229
|
+
[cantonese.org](https://cantonese.org), CC BY-SA 3.0) and the Cantonese
|
|
230
|
+
readings for CC-CEDICT (© 2015 Pleco Software Inc., CC BY-SA 3.0), and MDBG
|
|
231
|
+
and the contributors of [CC-CEDICT](https://cc-cedict.org);
|
|
232
|
+
* Carbo Kuo (BYVoid) and contributors, for
|
|
233
|
+
[OpenCC](https://github.com/BYVoid/OpenCC) (Apache-2.0), whose variant
|
|
234
|
+
tables let variant shapes find the same words.
|
|
235
|
+
|
|
236
|
+
## Licence
|
|
237
|
+
|
|
238
|
+
The code is under the MIT licence
|
|
239
|
+
([`LICENSE`](https://github.com/Beyond3345/xjyutping-py/blob/main/LICENSE)),
|
|
240
|
+
and the tone symbols in `TONE_MARKS` come from Visual Jyutping (MIT). The data
|
|
241
|
+
files in `src/xjyutping/data/` are under
|
|
242
|
+
[CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/), because they
|
|
243
|
+
adapt the CC BY-SA 3.0 word lists above. The readings of the 640 characters
|
|
244
|
+
from 粵音資料集叢 come from data published without a licence and are used with
|
|
245
|
+
attribution.
|