cpp-pinyin2 2.0.0__py3-none-win_amd64.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- cpp_pinyin2/__init__.py +860 -0
- cpp_pinyin2/_errors.py +64 -0
- cpp_pinyin2/_ffi.py +411 -0
- cpp_pinyin2/_native/libcpp-pinyin2.dll +0 -0
- cpp_pinyin2/_native/name.txt +1 -0
- cpp_pinyin2/_native.py +80 -0
- cpp_pinyin2/_version.py +4 -0
- cpp_pinyin2/data/dict/cantonese/License.txt +13 -0
- cpp_pinyin2/data/dict/cantonese/dialect.json +15 -0
- cpp_pinyin2/data/dict/cantonese/format.txt +1 -0
- cpp_pinyin2/data/dict/cantonese/phrases_dict.txt +1477 -0
- cpp_pinyin2/data/dict/cantonese/phrases_map.txt +461 -0
- cpp_pinyin2/data/dict/cantonese/trans_word.txt +1799 -0
- cpp_pinyin2/data/dict/cantonese/user_dict.txt +0 -0
- cpp_pinyin2/data/dict/cantonese/word.txt +19483 -0
- cpp_pinyin2/data/dict/mandarin/License.txt +32 -0
- cpp_pinyin2/data/dict/mandarin/dialect.json +15 -0
- cpp_pinyin2/data/dict/mandarin/format.txt +1 -0
- cpp_pinyin2/data/dict/mandarin/phrases_dict.txt +6975 -0
- cpp_pinyin2/data/dict/mandarin/phrases_map.txt +1003 -0
- cpp_pinyin2/data/dict/mandarin/trans_word.txt +6110 -0
- cpp_pinyin2/data/dict/mandarin/user_dict.txt +101 -0
- cpp_pinyin2/data/dict/mandarin/word.txt +17525 -0
- cpp_pinyin2-2.0.0.dist-info/METADATA +509 -0
- cpp_pinyin2-2.0.0.dist-info/RECORD +29 -0
- cpp_pinyin2-2.0.0.dist-info/WHEEL +5 -0
- cpp_pinyin2-2.0.0.dist-info/licenses/LICENSE +202 -0
- cpp_pinyin2-2.0.0.dist-info/licenses/THIRD_PARTY_NOTICES.md +44 -0
- cpp_pinyin2-2.0.0.dist-info/top_level.txt +1 -0
|
@@ -0,0 +1,509 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: cpp-pinyin2
|
|
3
|
+
Version: 2.0.0
|
|
4
|
+
Summary: Chinese (Mandarin + Cantonese) pinyin conversion driven by a native C++ engine
|
|
5
|
+
License: Apache-2.0 AND CC-BY-SA-3.0 AND CC-BY-SA-4.0
|
|
6
|
+
Project-URL: Homepage, https://github.com/wolfgitpr/cpp-pinyin
|
|
7
|
+
Project-URL: Documentation, https://github.com/wolfgitpr/cpp-pinyin/blob/v2/docs/QUICKSTART.md
|
|
8
|
+
Project-URL: Changelog, https://github.com/wolfgitpr/cpp-pinyin/blob/v2/CHANGELOG.md
|
|
9
|
+
Project-URL: Source, https://github.com/wolfgitpr/cpp-pinyin
|
|
10
|
+
Project-URL: Tracker, https://github.com/wolfgitpr/cpp-pinyin/issues
|
|
11
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Operating System :: Microsoft :: Windows
|
|
14
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
15
|
+
Classifier: Programming Language :: C++
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
18
|
+
Classifier: Topic :: Text Processing :: Linguistic
|
|
19
|
+
Requires-Python: >=3.9
|
|
20
|
+
Description-Content-Type: text/markdown
|
|
21
|
+
License-File: LICENSE
|
|
22
|
+
License-File: THIRD_PARTY_NOTICES.md
|
|
23
|
+
Dynamic: classifier
|
|
24
|
+
Dynamic: description
|
|
25
|
+
Dynamic: description-content-type
|
|
26
|
+
Dynamic: license
|
|
27
|
+
Dynamic: license-file
|
|
28
|
+
Dynamic: project-url
|
|
29
|
+
Dynamic: requires-python
|
|
30
|
+
Dynamic: summary
|
|
31
|
+
|
|
32
|
+
# cpp-pinyin2
|
|
33
|
+
|
|
34
|
+
A C++17 library for converting Chinese characters to Pinyin (Mandarin) and Jyutping (Cantonese). Zero external dependencies, fast, and accurate.
|
|
35
|
+
|
|
36
|
+
**Version: `project(... VERSION)` in the root `CMakeLists.txt` is the only copy** | **License: Apache-2.0 (code) · CC BY-SA (dictionary data, see [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md))** | **C++17** | **CMake >= 3.17**
|
|
37
|
+
|
|
38
|
+
**What changed, and where the measured numbers live: [CHANGELOG.md](CHANGELOG.md).** The changelog describes user-visible changes and points at the documents that own the numbers; it does not quote them, so the two cannot drift apart.
|
|
39
|
+
|
|
40
|
+
**New here? [`docs/QUICKSTART.md`](docs/QUICKSTART.md)** is the shortest path for all three surfaces (C++ / Python / C#): what to install, five lines of code, the expected output, and the traps people hit first. The section below is the same material in the repository's own voice, with the full examples.
|
|
41
|
+
|
|
42
|
+
## Features
|
|
43
|
+
|
|
44
|
+
- Mandarin Pinyin and Cantonese Jyutping conversion
|
|
45
|
+
- Polyphonic character disambiguation via phrase dictionary (sliding window, max length 4)
|
|
46
|
+
- Traditional to Simplified Chinese conversion
|
|
47
|
+
- Custom user dictionaries and fan-jian (Traditional/Simplified) dictionaries
|
|
48
|
+
- Custom dialect support through `ToneConverter` inheritance
|
|
49
|
+
- Multiple tone styles (Unicode marks, numbered, plain)
|
|
50
|
+
- Initials (声母) and finals (韵母) styles, aligned with python-pinyin
|
|
51
|
+
- Optional Mandarin tone-sandhi post-pass (三声连读 / 不 / 一) for TONE3 output
|
|
52
|
+
- Unicode range: \[0x4E00, 0x9FFF\] (CJK Unified Ideographs basic block)
|
|
53
|
+
- Performance and accuracy figures are deliberately **not** duplicated here — numbers drift out of sync when copied.
|
|
54
|
+
The single authority is [`docs/PERFORMANCE.md`](docs/PERFORMANCE.md) for measured values, and `ctest` for what is
|
|
55
|
+
actually enforced on every build. (The figures once printed here — throughput, no-tone accuracy on a lyrics set, and
|
|
56
|
+
with-tone accuracy on `CPP_Dataset` — are kept, with their dated comparison against re-measurement, in that document.)
|
|
57
|
+
|
|
58
|
+
## Install
|
|
59
|
+
|
|
60
|
+
### CMake
|
|
61
|
+
|
|
62
|
+
No vcpkg port is provided: this repository ships no portfile, and there is no `cpp-pinyin2` port in the
|
|
63
|
+
registry. Build from source and install with CMake as below.
|
|
64
|
+
|
|
65
|
+
```cmake
|
|
66
|
+
find_package(cpp-pinyin2)
|
|
67
|
+
target_link_libraries(${PROJECT_NAME} PRIVATE cpp-pinyin2::cpp-pinyin2)
|
|
68
|
+
|
|
69
|
+
# Copy dictionary files to the output directory
|
|
70
|
+
#
|
|
71
|
+
# The variable is `CPP_PINYIN2_DICT_DIR`, which cpp-pinyin2Config.cmake sets to the installed
|
|
72
|
+
# dictionary location (`share/cpp-pinyin2/dict`, see the migration table below). The 1.x line put
|
|
73
|
+
# them under `lib/cmake/cpp-pinyin2/dict` instead, which is why an older recipe may say that.
|
|
74
|
+
add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD
|
|
75
|
+
COMMAND "${CMAKE_COMMAND}" -E copy_directory
|
|
76
|
+
"${CPP_PINYIN2_DICT_DIR}"
|
|
77
|
+
"$<TARGET_FILE_DIR:${PROJECT_NAME}>/dict"
|
|
78
|
+
)
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
### CMake FetchContent
|
|
82
|
+
|
|
83
|
+
```cmake
|
|
84
|
+
include(FetchContent)
|
|
85
|
+
FetchContent_Declare(
|
|
86
|
+
cpp-pinyin2
|
|
87
|
+
GIT_REPOSITORY https://github.com/wolfgitpr/cpp-pinyin.git
|
|
88
|
+
GIT_TAG <tag-or-commit> # use a release tag, or a commit hash
|
|
89
|
+
)
|
|
90
|
+
FetchContent_MakeAvailable(cpp-pinyin2)
|
|
91
|
+
|
|
92
|
+
target_link_libraries(${PROJECT_NAME} PRIVATE cpp-pinyin2::cpp-pinyin2)
|
|
93
|
+
|
|
94
|
+
# Copy dictionary files to the output directory
|
|
95
|
+
add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD
|
|
96
|
+
COMMAND "${CMAKE_COMMAND}" -E copy_directory
|
|
97
|
+
"${CPP_PINYIN2_DICT_DIR}"
|
|
98
|
+
"$<TARGET_FILE_DIR:${PROJECT_NAME}>/dict"
|
|
99
|
+
)
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
### Build from source
|
|
103
|
+
|
|
104
|
+
```bash
|
|
105
|
+
mkdir build && cd build
|
|
106
|
+
cmake .. -DCPP_PINYIN2_BUILD_TESTS=ON
|
|
107
|
+
cmake --build .
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
A checked-in `CMakePresets.json` reproduces the development configuration (it differs from the CI one in two
|
|
111
|
+
options on purpose: examples OFF, WERROR not forced), and keeps the
|
|
112
|
+
build tree under `.tmp/` so that nothing grows in the repository root:
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
cmake --preset default
|
|
116
|
+
cmake --build --preset default
|
|
117
|
+
ctest --preset default
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
The root of this repository is a working directory as much as a source tree: `.tmp/` holds scratch builds and
|
|
121
|
+
research output, `.tmp-firefly-disk/` holds board cleanup scripts, and both are git-ignored. Anything an IDE
|
|
122
|
+
wants to generate — build directories above all — belongs under `.tmp/` as well, which is what the preset is
|
|
123
|
+
for. Only what `git ls-files` tracks at the top level is project content.
|
|
124
|
+
|
|
125
|
+
For static linking, define `CPP_PINYIN2_STATIC` before including headers.
|
|
126
|
+
|
|
127
|
+
`-DCPP_PINYIN2_BUILD_TESTS=ON` builds the C++ suite (`ctest` runs it). The two language shells have their own
|
|
128
|
+
tests. Neither is a single command on a fresh checkout: both need files that are generated and not tracked.
|
|
129
|
+
|
|
130
|
+
> **Windows hosts: give the suite a writable `TEMP`.** Some C++ tests write a scratch file under
|
|
131
|
+
> `std::filesystem::temp_directory_path()`. On this development host, binaries built by the MinGW toolchain
|
|
132
|
+
> **cannot create files under the default `%TEMP%`** (`C:\Users\<user>\AppData\Local\Temp\`) while Python
|
|
133
|
+
> can -- measured with a 16-line standalone probe compiled by the same `g++`: `ofstream.is_open()` is `false`
|
|
134
|
+
> there and `true` when `TEMP`/`TMP` point at a writable directory (the build tree works). The suite then fails
|
|
135
|
+
> three tests with `REQUIRE( out.is_open() )` in `tests/cpp/test_user_dict.cpp`, which reads as a repository
|
|
136
|
+
> failure but is not one: **this is an environment fact, not a repo fact**, exactly like the missing-`pypinyin`
|
|
137
|
+
> note in the dictionary repository's README. Point `TEMP`/`TMP` somewhere writable before `ctest` and the
|
|
138
|
+
> suite is green (exit 0). The **registered test count is a single authority** — `docs/STABILITY.md` §0 — and is
|
|
139
|
+
> deliberately not quoted here: the number in this line once said 33/33 long after the suite had 36 and then 41.
|
|
140
|
+
|
|
141
|
+
```bash
|
|
142
|
+
python python/tools/build_native.py # stages the native library, the dictionaries, and _version.py
|
|
143
|
+
(cd python && python -m pytest tests -q) # the Python shell
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
```bash
|
|
147
|
+
# the C# shell. type into the placeholders the built library under build/ and the directory holding it;
|
|
148
|
+
# emit_build_info.py writes version.props and BuildInfo.g.cs, which are not tracked either.
|
|
149
|
+
python csharp/tools/emit_build_info.py --native <built cpp-pinyin2 library>
|
|
150
|
+
python csharp/tools/run_tests.py --native-dir <directory holding it>
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
The order that is actually executed for both shells lives in [`.github/workflows/`](./.github/workflows).
|
|
154
|
+
`csharp/tools/run_tests.py` is not a convenience wrapper: on some hardened hosts `dotnet test` aborts before
|
|
155
|
+
the first test runs, with `Win32Exception (5): Access denied` raised by VSTest's testhost, and this runner
|
|
156
|
+
drives xunit's own engine over the same assemblies instead, proving itself with `--self-test` first. CI runs
|
|
157
|
+
`dotnet test` on three operating systems; whether that job passes is for the job's own result to say, not for
|
|
158
|
+
this note. Measurements and the two traps this runner codes around:
|
|
159
|
+
[`docs/DECISIONS.md` §2.1](./`docs/DECISIONS.md` §2.1) V30.
|
|
160
|
+
|
|
161
|
+
## Quick Start
|
|
162
|
+
|
|
163
|
+
### Mandarin Pinyin
|
|
164
|
+
|
|
165
|
+
```cpp
|
|
166
|
+
#include <cpp-pinyin2/core/Engine.h>
|
|
167
|
+
#include <cpp-pinyin2/ext/Registry.h>
|
|
168
|
+
#include <iostream>
|
|
169
|
+
|
|
170
|
+
cpp_pinyin::v2::Registry registry; // 方言注册表,比引擎活得久
|
|
171
|
+
auto opened = cpp_pinyin::v2::Engine::open(registry, "dict", "mandarin");
|
|
172
|
+
cpp_pinyin::v2::EngineConfig cfg;
|
|
173
|
+
cfg.style = cpp_pinyin::v2::MandarinStyle::TONE3; // "zhong1 guo2 ren2 min2"
|
|
174
|
+
std::cout << opened.value().query("中国人民", cfg).value().join() << "\n";
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
这一版只展示 API 主干(注册表 + 打开 + 配置 + 查询)。**完整示例**——逐字结果、候选读音、**打开失败的错误详情**——见
|
|
178
|
+
`examples/mandarin_example.cpp`(可构建、随门禁编译)与 `docs/API_en.md` / `docs/API_zh.md`
|
|
179
|
+
(此处不再复制一份,避免同一段代码三处维护)。
|
|
180
|
+
|
|
181
|
+
### Cantonese Jyutping
|
|
182
|
+
|
|
183
|
+
```cpp
|
|
184
|
+
#include <cpp-pinyin2/core/Engine.h>
|
|
185
|
+
#include <cpp-pinyin2/ext/Registry.h>
|
|
186
|
+
#include <iostream>
|
|
187
|
+
|
|
188
|
+
cpp_pinyin::v2::Registry registry; // 方言注册表,比引擎活得久
|
|
189
|
+
auto opened = cpp_pinyin::v2::Engine::open(registry, "dict", "cantonese");
|
|
190
|
+
cpp_pinyin::v2::EngineConfig cfg;
|
|
191
|
+
cfg.style = cpp_pinyin::v2::CantoneseStyle::TONE3; // "zung1 gwok3 jan4 man4"
|
|
192
|
+
std::cout << opened.value().query("中國人民", cfg).value().join() << "\n";
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
与上面的普通话例子**同形**:注册表 + 打开(失败即带原因返回)+ 查询。完整示例与错误处理见
|
|
196
|
+
`examples/`(随门禁构建)与 `docs/API_en.md` / `docs/API_zh.md`。
|
|
197
|
+
|
|
198
|
+
### Loading custom dictionaries
|
|
199
|
+
|
|
200
|
+
```cpp
|
|
201
|
+
// 沿用上面打开的引擎 m(方言 id 决定它读哪套词典)
|
|
202
|
+
m.loadUserDict("path/to/my_dict.txt"); // 自定义词典:词 -> 读音
|
|
203
|
+
m.loadTraditionalDict("path/to/my_fanjian.txt"); // 简繁映射
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
## Consuming from another project
|
|
207
|
+
|
|
208
|
+
**CMake.** The installed package exports one target and one variable, and **there is no implicit
|
|
209
|
+
dictionary path** — `dict_dir` is always an argument (`include/cpp-pinyin2/ext/c_api.h`, rule 7):
|
|
210
|
+
|
|
211
|
+
```cmake
|
|
212
|
+
find_package(cpp-pinyin2 CONFIG REQUIRED)
|
|
213
|
+
target_link_libraries(your_app PRIVATE cpp-pinyin2::cpp-pinyin2)
|
|
214
|
+
|
|
215
|
+
# The dictionaries this package installed; copy or point at them yourself -- the engine
|
|
216
|
+
# never guesses a location. Exported by cpp-pinyin2Config.cmake.
|
|
217
|
+
add_custom_command(TARGET your_app POST_BUILD
|
|
218
|
+
COMMAND "${CMAKE_COMMAND}" -E copy_directory
|
|
219
|
+
"${CPP_PINYIN2_DICT_DIR}" "$<TARGET_FILE_DIR:your_app>/dict")
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
The package refuses a **cross-major** request on purpose (`COMPATIBILITY SameMajorVersion`), so a
|
|
223
|
+
1.x `find_package(cpp-pinyin2 1.x)` does not silently resolve to a 2.x install.
|
|
224
|
+
|
|
225
|
+
**Per-character alignment.** Downstream morphology/TTS code usually needs "which reading belongs to
|
|
226
|
+
which input character". One row of a `Result` covers **one input span**, and the spans are a
|
|
227
|
+
partition of your input — the contract is that **concatenating every row's `text` in order reproduces
|
|
228
|
+
the input byte string** (enforced by `tests/cpp/test_c_api.cpp`, `CHECK(joined == one.text)`).
|
|
229
|
+
So a byte cursor over the input is the whole recipe:
|
|
230
|
+
|
|
231
|
+
```cpp
|
|
232
|
+
std::string input = "银行ABC";
|
|
233
|
+
auto rows = engine.query(input).value(); // one row per span, spans partition `input`
|
|
234
|
+
|
|
235
|
+
std::size_t byteOffset = 0; // stays aligned: Σ row.text == input
|
|
236
|
+
for (const cpp_pinyin::v2::Segment &row : rows) {
|
|
237
|
+
// This row's reading is `row.rendered` (bytes), not `row.candidates[0]`:
|
|
238
|
+
// a phrase hit may pick a reading that the character's own candidate list
|
|
239
|
+
// does not even contain (docs/REQUIREMENTS.md §1.2 K9).
|
|
240
|
+
use(byteOffset, row.text.size(), row.rendered);
|
|
241
|
+
|
|
242
|
+
// Two deliberate exceptions to "one row == one character":
|
|
243
|
+
// * a run of ASCII letters/digits is ONE span, so a 3-byte Hanzi is never
|
|
244
|
+
// the unit of `text.size()` -- always advance by the byte count;
|
|
245
|
+
// * a row the library could not read keeps `rendered == text` and carries
|
|
246
|
+
// `row.failure_kind != Code::Ok` (punctuation, digits, unknown characters).
|
|
247
|
+
byteOffset += row.text.size();
|
|
248
|
+
}
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
If you control phrase-level disambiguation yourself, `Engine::setDisambiguationCallback` hands you
|
|
252
|
+
the **UTF-8 byte offset** of the character inside your text (`core/Types.h`), which is the same
|
|
253
|
+
alignment expressed from the callback side.
|
|
254
|
+
|
|
255
|
+
## Migrating from 1.x (and from an earlier 2.0 revision)
|
|
256
|
+
|
|
257
|
+
**Nothing in the 1.x surface survives as an alias**, so a 1.x call site has to be rewritten — the
|
|
258
|
+
names below have **zero** hits under `include/` and `src/` on this branch.
|
|
259
|
+
|
|
260
|
+
| 1.x (tag `1.0.2`) | this branch | notes |
|
|
261
|
+
|---|---|---|
|
|
262
|
+
| `namespace Pinyin` | `namespace cpp_pinyin` | every public type |
|
|
263
|
+
| `Pinyin::Pinyin` / `Pinyin::Jyutping` | `cpp_pinyin::v2::Engine::open(registry, dictRoot, "mandarin"/"cantonese")` | one class instead of two; the dialect is an argument |
|
|
264
|
+
| `Pinyin::setDictionaryPath(root)` / `dictionaryPath()` (process-global) | the `dictRoot` argument of `open` / the description constructors | **no process-global path at all** — two engines with two roots coexist (`core/Engine.h:21-22`) |
|
|
265
|
+
| `hanziToPinyin(hans, style, error, candidates, v_to_u, neutral_tone_with_five)` → `PinyinResVector` | `query(text, cfg)` → `Outcome<Result>` | `cfg` carries style/`includeCandidates`/`vToU`/`markNeutralAs5`; a failure is an `Outcome` with no value plus `error().code`/`detail` |
|
|
266
|
+
| row fields `character` / `reading` / `failed` | `Segment::text` / `rendered` / `failure_kind` | the mapping is recorded in `core/Types.h` (the `Segment` block header) |
|
|
267
|
+
| `ManTone::Style` / `CanTone::Style` | `MandarinStyle` / `CantoneseStyle` | **the numbers are unchanged** (`NORMAL=0`, `TONE=1`, `TONE2=2`, `TONE3=8`, …) — only the enum's name and namespace moved |
|
|
268
|
+
| `include/cpp-pinyin2/Pinyin.h` etc. | `include/cpp-pinyin2/core/…`, `ext/…`, `dialects/…` | 15 public headers, frozen list in `tests/abi/expected_public_headers.txt` |
|
|
269
|
+
| tone sandhi as a side effect of a style | `cpp_pinyin::v2::applyMandarinToneSandhi(result, opts)` | an explicit post-pass (`core/ToneSandhi.h`) |
|
|
270
|
+
|
|
271
|
+
**If you were consuming the Python shell of an earlier 2.0 revision** (`Segment.failed` /
|
|
272
|
+
`Segment.reading`), two field names changed on 2026-10-07:
|
|
273
|
+
|
|
274
|
+
| earlier 2.0 shell | now |
|
|
275
|
+
|---|---|
|
|
276
|
+
| `Segment.failed` (bool) | `Segment.failure_kind` (int; `0` = OK, `code_text()` is the wording authority) |
|
|
277
|
+
| `Segment.reading` (str) | `Segment.rendered` (str) — the structured reading is `Segment.syllables` |
|
|
278
|
+
| — | `Segment.text` is a **span**, not a character: a run of ASCII letters is one span |
|
|
279
|
+
|
|
280
|
+
The shell's class/constructor names did **not** change: `Engine(dialect_id=..., dict_dir=...)`,
|
|
281
|
+
`engine.query(text, style=..., include_candidates=...)`, `engine.readings(...)`.
|
|
282
|
+
|
|
283
|
+
**Build/packaging changes** that a port or CI job has to absorb:
|
|
284
|
+
|
|
285
|
+
| 1.x | this branch |
|
|
286
|
+
|---|---|
|
|
287
|
+
| dictionaries installed under `lib/cmake/cpp-pinyin2/dict` | `share/cpp-pinyin2/dict` (`CMAKE_INSTALL_DATADIR`), exported as `CPP_PINYIN2_DICT_DIR` |
|
|
288
|
+
| `-DCPP_PINYIN_VCPKG_DICT_DIR=…` | **removed** (no equivalent option) |
|
|
289
|
+
| `-DCPP_PINYIN_BUILD_STATIC` | `-DBUILD_SHARED_LIBS=OFF` (which defines `CPP_PINYIN2_STATIC` for consumers) |
|
|
290
|
+
| a 1.x `find_package(cpp-pinyin2 1.x)` request | refused on purpose: `COMPATIBILITY SameMajorVersion` ⇒ request `2.x` |
|
|
291
|
+
|
|
292
|
+
## API Reference
|
|
293
|
+
|
|
294
|
+
All public types live in the `cpp_pinyin::v2` namespace. This page does not restate the signatures --
|
|
295
|
+
the headers are the authority, and `docs/API_zh.md` / `docs/API_en.md` enumerate every type and member
|
|
296
|
+
with examples.
|
|
297
|
+
|
|
298
|
+
| You are looking for | Authority |
|
|
299
|
+
|---|---|
|
|
300
|
+
| `Engine`, `EngineConfig`, `Outcome`, `Code`, `Error` | `include/cpp-pinyin2/core/Engine.h`, `core/Types.h` |
|
|
301
|
+
| `Segment`, `Result`, `Syllable`, `Style`, `ToneKind` | `include/cpp-pinyin2/core/Types.h`, `core/Syllable.h` |
|
|
302
|
+
| `Dialect`, `ToneConverter`, `StyleDecl`, `Registry` (the plugin surface) | `include/cpp-pinyin2/ext/Dialect.h`, `core/ToneConverter.h`, `ext/Registry.h` |
|
|
303
|
+
| Character predicates (`isHanzi`, `isSpace`, ...) | `include/cpp-pinyin2/core/G2pglobal.h` |
|
|
304
|
+
| Mandarin tone sandhi | `include/cpp-pinyin2/core/ToneSandhi.h` |
|
|
305
|
+
| UTF-8 / UTF-16 conversion | `include/cpp-pinyin2/core/U16Str.h` |
|
|
306
|
+
| The C ABI (`cppp2_*`) | `include/cpp-pinyin2/ext/c_api.h` |
|
|
307
|
+
|
|
308
|
+
Cantonese declares three styles (`NORMAL` 0, `FIRST_LETTER` 3, `TONE3` 8 -- `zung gwok`, `z g`,
|
|
309
|
+
`zung1 gwok3`); it has no `INITIALS`/`FINALS` styles yet. Mandarin's style table is in `core/Types.h`.
|
|
310
|
+
|
|
311
|
+
### The three calls most code needs
|
|
312
|
+
|
|
313
|
+
```cpp
|
|
314
|
+
cpp_pinyin::v2::Engine engine(dialect, dictRoot); // or Engine::open(registry, dictRoot, "mandarin")
|
|
315
|
+
auto result = engine.query("中国"); // Outcome<Result>; check it before reading
|
|
316
|
+
for (const auto &seg : result.value()) std::cout << seg.rendered; // Outcome has value()/error(),
|
|
317
|
+
// no operator* -- value() throws on failure
|
|
318
|
+
```
|
|
319
|
+
|
|
320
|
+
```cpp
|
|
321
|
+
// Structured readings: Segment::reading carries initial / fin / tone / tone_kind / source.
|
|
322
|
+
// `candidates` stays a column of STRINGS on purpose -- seven shipped readings cannot be
|
|
323
|
+
// rendered back from (initial, fin), so moving it to structures needs the renderer first.
|
|
324
|
+
config.includeReading = true;
|
|
325
|
+
```
|
|
326
|
+
|
|
327
|
+
```cpp
|
|
328
|
+
cpp_pinyin::v2::ToneSandhiOptions opts; // all three Mandarin rules on by default
|
|
329
|
+
const size_t changed = cpp_pinyin::v2::applyMandarinToneSandhi(result.value(), opts);
|
|
330
|
+
```
|
|
331
|
+
|
|
332
|
+
## Custom Dialect Support
|
|
333
|
+
|
|
334
|
+
**Adding a dialect does not require touching this library** — you supply dictionary files, a
|
|
335
|
+
`ToneConverter` subclass, and one `Dialect`, then hand it to `Engine` (directly, or through a
|
|
336
|
+
`Registry` so it can be addressed by id). Authority: `include/cpp-pinyin2/ext/Dialect.h`.
|
|
337
|
+
|
|
338
|
+
```cpp
|
|
339
|
+
#include <cpp-pinyin2/core/Engine.h>
|
|
340
|
+
#include <cpp-pinyin2/ext/Dialect.h>
|
|
341
|
+
#include <cpp-pinyin2/ext/Registry.h>
|
|
342
|
+
|
|
343
|
+
// 1. Your own ToneConverter subclass. It carries its OWN style numbering: an out-of-tree
|
|
344
|
+
// dialect cannot extend the built-in enums, which is why the numbering lives here.
|
|
345
|
+
class MyTone final : public cpp_pinyin::v2::ToneConverter {
|
|
346
|
+
public:
|
|
347
|
+
enum Style { NORMAL = 0, TONE3 = 8 };
|
|
348
|
+
MyTone() { m_converts[static_cast<int>(TONE3)] = &myTone3; } // static ConvertFunc, defined by you
|
|
349
|
+
};
|
|
350
|
+
|
|
351
|
+
// 2. One description. The registry stores a CREATOR, so it can hand out a fresh immutable
|
|
352
|
+
// description each time the dialect is resolved.
|
|
353
|
+
auto makeDialect = [] {
|
|
354
|
+
auto d = std::make_shared<cpp_pinyin::v2::Dialect>();
|
|
355
|
+
d->id = "wuu"; // the id `Engine::open(registry, root, "wuu")` resolves
|
|
356
|
+
d->dictSubDir = "wuu"; // subdirectory of the dictionary root; empty means "use id"
|
|
357
|
+
d->tone = MyTone{}; // your converter, held BY VALUE
|
|
358
|
+
d->supportsVToU = true; // gates EngineConfig::vToU
|
|
359
|
+
d->supportsNeutralTone5 = true; // gates EngineConfig::markNeutralAs5
|
|
360
|
+
d->styles = {{"normal", 0, false}, {"tone3", 8, true}}; // {name, alias, isDefault}
|
|
361
|
+
return d;
|
|
362
|
+
};
|
|
363
|
+
|
|
364
|
+
// 3. A registry the caller owns -- registering the built-ins, then yours next to them.
|
|
365
|
+
cpp_pinyin::v2::Registry registry;
|
|
366
|
+
if (!registry.add("wuu", makeDialect))
|
|
367
|
+
/* the id was already taken and `replace` was false, or the id/creator was empty */;
|
|
368
|
+
|
|
369
|
+
auto opened = cpp_pinyin::v2::Engine::open(registry, "path/to/dict", "wuu"); // reports WHY on failure
|
|
370
|
+
```
|
|
371
|
+
|
|
372
|
+
A dialect that is **not** registered needs no creator: keep the description and hand it straight to
|
|
373
|
+
the constructor — `cpp_pinyin::v2::Engine engine(*makeDialect(), "path/to/dict");`.
|
|
374
|
+
|
|
375
|
+
Two properties are deliberate and load-bearing (`ext/Dialect.h:20-32`):
|
|
376
|
+
|
|
377
|
+
- **`tone` is held by value, not by pointer**, because `EngineImpl` copies the converter it is given.
|
|
378
|
+
A value copy slices to the `ToneConverter` base subobject — which is harmless because every dialect
|
|
379
|
+
behaviour lives in that base's **function pointer table** (`m_converts`, `m_normalize`) and never in
|
|
380
|
+
a virtual. Do not "improve" this into a virtual.
|
|
381
|
+
- **A `Dialect` carries no dictionary path and no loaded state.** The root is an `Engine` constructor
|
|
382
|
+
argument, so two instances of the same dialect can read two different directories at once.
|
|
383
|
+
|
|
384
|
+
`Dialect::postProcess` is the supported place for a caller-side post-pass over a finished `Result`
|
|
385
|
+
(`ext/Dialect.h:68-72`); the engine's own disambiguation window is frozen and must not be
|
|
386
|
+
reimplemented there.
|
|
387
|
+
|
|
388
|
+
**The executable form of everything above is [`examples/plugin-dialect/plugin_dialect.cpp`](./examples/plugin-dialect/plugin_dialect.cpp)**
|
|
389
|
+
— one translation unit outside the library, reached through the installed package, registered into a
|
|
390
|
+
registry the caller owns, running next to a built-in dialect over the same dictionary root. It is
|
|
391
|
+
built by the gates, so it cannot rot.
|
|
392
|
+
|
|
393
|
+
[pinyin-makedict](https://github.com/wolfgitpr/pinyin-makedict) is the companion tool for creating custom dialect dictionaries.
|
|
394
|
+
|
|
395
|
+
See [Custom Chinese Dialect](./docs/Custom%20Chinese%20Dialect.md) for details.
|
|
396
|
+
|
|
397
|
+
## Dictionary Structure
|
|
398
|
+
|
|
399
|
+
Dictionary files live under `data/dialects/` with separate directories per language. Each dialect states the
|
|
400
|
+
role of every file it ships in its own `dialect.json`, so a missing table is an error rather than a partial
|
|
401
|
+
dictionary that still loads:
|
|
402
|
+
|
|
403
|
+
```
|
|
404
|
+
data/dialects/
|
|
405
|
+
mandarin/
|
|
406
|
+
dialect.json # file roles: required tables, optional tables, assets
|
|
407
|
+
word.txt
|
|
408
|
+
phrases_map.txt
|
|
409
|
+
phrases_dict.txt
|
|
410
|
+
user_dict.txt
|
|
411
|
+
trans_word.txt
|
|
412
|
+
License.txt
|
|
413
|
+
cantonese/
|
|
414
|
+
dialect.json
|
|
415
|
+
word.txt
|
|
416
|
+
phrases_map.txt
|
|
417
|
+
phrases_dict.txt
|
|
418
|
+
user_dict.txt
|
|
419
|
+
trans_word.txt
|
|
420
|
+
License.txt
|
|
421
|
+
```
|
|
422
|
+
|
|
423
|
+
These two are the dialects this repository ships and claims; CI fails if a third one appears here. Your own
|
|
424
|
+
dialect's dictionaries live in a directory of your choice outside the repository and are passed to the engine
|
|
425
|
+
-- see [Custom Chinese Dialect](./docs/Custom%20Chinese%20Dialect.md).
|
|
426
|
+
|
|
427
|
+
## Build Options
|
|
428
|
+
|
|
429
|
+
| Option | Default | Description |
|
|
430
|
+
|--------|---------|-------------|
|
|
431
|
+
| `BUILD_SHARED_LIBS` | ON | Build as shared library (dynamic linking) |
|
|
432
|
+
| `CPP_PINYIN2_BUILD_TESTS` | OFF | Build the test suite |
|
|
433
|
+
| `CPP_PINYIN2_INSTALL` | ON | Generate install targets |
|
|
434
|
+
| `CPP_PINYIN2_EMBED_DICTS` | OFF | Embed the dictionary tables into the library: `ON`, or a `;`-separated list of dialect names. Generated by `tools/embed_dialects.py`; a build that embeds answers table requests from itself when the requested path does not exist, which is what a single-file deployment needs |
|
|
435
|
+
|
|
436
|
+
For static linking in your own project, define `CPP_PINYIN2_STATIC` before including any cpp-pinyin2 headers.
|
|
437
|
+
|
|
438
|
+
## Contributing
|
|
439
|
+
|
|
440
|
+
Contributions are welcome. [`CONTRIBUTING.md`](CONTRIBUTING.md) lists the gates to run before opening a pull request, and [`SECURITY.md`](SECURITY.md) is the private channel for a vulnerability.
|
|
441
|
+
|
|
442
|
+
## Documentation
|
|
443
|
+
|
|
444
|
+
Three lists. **Start here** if you are using the library; **reference** when you need one specific
|
|
445
|
+
answer; **maintainer ledgers** record how this tree got where it is — useful when you are *changing*
|
|
446
|
+
the repository, not when you are using it. [`docs/README.md`](./docs/README.md) is the full index,
|
|
447
|
+
including which document owns which fact.
|
|
448
|
+
|
|
449
|
+
### Start here
|
|
450
|
+
|
|
451
|
+
| Document | Contents |
|
|
452
|
+
|---|---|
|
|
453
|
+
| [`docs/QUICKSTART.md`](./docs/QUICKSTART.md) | **起手最快的一份**:三种壳(C++ / Python / C#)各自的最短可用路径、预期输出与常见坑(中英对照) |
|
|
454
|
+
| [`docs/API_en.md`](./docs/API_en.md) | Full C++ API reference (English) |
|
|
455
|
+
| [`docs/API_zh.md`](./docs/API_zh.md) | 完整 C++ API 文档(中文) |
|
|
456
|
+
| [`docs/Custom Chinese Dialect.md`](./docs/Custom%20Chinese%20Dialect.md) | Adding a custom dialect |
|
|
457
|
+
| [`docs/PERFORMANCE.md`](./docs/PERFORMANCE.md) | 性能简报:**性能数字的唯一真相源**、测量协议、准入判据 |
|
|
458
|
+
|
|
459
|
+
### Reference
|
|
460
|
+
|
|
461
|
+
| Document | Contents |
|
|
462
|
+
|---|---|
|
|
463
|
+
| [`docs/CONTRACT.md`](./docs/CONTRACT.md) | 契约与不变量(对齐、失败码、线程安全边界、C ABI 增长规则) |
|
|
464
|
+
| [`docs/STABILITY.md`](./docs/STABILITY.md) | **跨代兼容政策 §4.5(R1–R16 的唯一权威)** + 长期稳定性评估(sanitizer / 线程 / 多实例 / ABI) |
|
|
465
|
+
| [`docs/UPGRADING_A_GENERATION.md`](./docs/UPGRADING_A_GENERATION.md) | **换代/迁移清单**:发布前与落地时两张表(替换半径、冻结清单、版本链、数据面 `format.txt`、默认风格迁移) |
|
|
466
|
+
|
|
467
|
+
### Maintainer ledgers (internal)
|
|
468
|
+
|
|
469
|
+
| Document | Contents |
|
|
470
|
+
|---|---|
|
|
471
|
+
| [`docs/DECISIONS.md`](./docs/DECISIONS.md) | **决策号唯一登记处 + 现行裁定**:各系列记号的含义、D 号分段、算法冻结区 K1–K8、验收线、性能与测量纪律、明确不做清单 |
|
|
472
|
+
| [`docs/DECISIONS.md` §2.1](./`docs/DECISIONS.md` §2.1) | 打包与发布:轮子/nupkg 矩阵、版本链、插件/打包决策台账 |
|
|
473
|
+
| [`docs/REQUIREMENTS.md`](./docs/REQUIREMENTS.md) | 需求契约:跨轮硬约束索引(算法冻结区、准确率门禁、跨平台) |
|
|
474
|
+
| [`docs/CONTRACT.md`](./docs/CONTRACT.md) | 用户面契约:C ABI 增长规则、失败值域、结构化读数开关 |
|
|
475
|
+
| [`docs/STABILITY.md`](./docs/STABILITY.md) | 稳定性政策 R1–R16、跨代兼容窗口、注册用例数的唯一权威 |
|
|
476
|
+
| [`docs/NEW_DIALECT_PLAYBOOK.md`](./docs/NEW_DIALECT_PLAYBOOK.md) | 新语种拓展 playbook(本仓只做中粤) |
|
|
477
|
+
| [`docs/UPGRADING_A_GENERATION.md`](./docs/UPGRADING_A_GENERATION.md) | 换代流程与故障对照表(判词 → 原因 → 修法) |
|
|
478
|
+
| [`docs/PERFORMANCE.md`](./docs/PERFORMANCE.md) | 性能测量协议与基线读数 |
|
|
479
|
+
| [`docs/RELEASE_CHECKLIST.md`](./docs/RELEASE_CHECKLIST.md) · [`docs/RELEASE_RUNBOOK.md`](./docs/RELEASE_RUNBOOK.md) | 发布一版的落点与当天要敲的命令 |
|
|
480
|
+
|
|
481
|
+
## Open-source references
|
|
482
|
+
|
|
483
|
+
- [zh_CN](https://github.com/ZiQiangWang/zh_CN): Core algorithm source, further tailored to this project's dictionaries.
|
|
484
|
+
- [opencpop](http://wenet.org.cn/opencpop/): Test data source.
|
|
485
|
+
- [M4Singer](https://github.com/M4Singer/M4Singer): Test data source.
|
|
486
|
+
- [cc-edict](https://cc-cedict.org/wiki/): Dictionary source.
|
|
487
|
+
- [pinyin (kfcd)](https://github.com/kfcd/pinyin): Fan-jian dictionary source.
|
|
488
|
+
- [cpp_dataset](https://github.com/kakaobrain/g2pm/tree/master/data): CPP_Dataset source.
|
|
489
|
+
|
|
490
|
+
## Related projects
|
|
491
|
+
|
|
492
|
+
- [pinyin](https://github.com/hotoo/pinyin): Chinese to Pinyin, Node.js/JavaScript.
|
|
493
|
+
- [python-pinyin](https://github.com/mozillazg/python-pinyin): Chinese to Pinyin, Python.
|
|
494
|
+
- [go-pinyin](https://github.com/mozillazg/go-pinyin): Chinese to Pinyin, Go.
|
|
495
|
+
- [rust-pinyin](https://github.com/mozillazg/rust-pinyin): Chinese to Pinyin, Rust.
|
|
496
|
+
- [csharp-pinyin](https://github.com/wolfgitpr/csharp-pinyin): Chinese to Pinyin, C#.
|
|
497
|
+
|
|
498
|
+
## License
|
|
499
|
+
|
|
500
|
+
**The code is Apache-2.0** — see [`LICENSE`](./LICENSE).
|
|
501
|
+
|
|
502
|
+
**The dictionary tables are not**: Mandarin ships under **CC BY-SA 4.0** and Cantonese under
|
|
503
|
+
**CC BY-SA 3.0**, and Catch2 (tests only, never installed) under BSL-1.0. Every package carries the
|
|
504
|
+
per-directory `License.txt` files next to the tables; the one-page summary of what is under which
|
|
505
|
+
license, where it ends up, and what a redistributor has to keep is
|
|
506
|
+
[`THIRD_PARTY_NOTICES.md`](./THIRD_PARTY_NOTICES.md).
|
|
507
|
+
|
|
508
|
+
If your product embeds this library, the share-alike obligation applies to the dictionary data, not to
|
|
509
|
+
your own source code.
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
cpp_pinyin2/__init__.py,sha256=ICTybG0Ffwpoza57d5vMrYOs0zaHjh1nRMc7J347UVE,40249
|
|
2
|
+
cpp_pinyin2/_errors.py,sha256=vFIcBFzyTh9R5N_57-3_pq4uf-jac6tE3QfcRMr6K0Y,2933
|
|
3
|
+
cpp_pinyin2/_ffi.py,sha256=SXg9ydZyXbSAYLLwlerL3URcmhfUBW-4OGjXBZVLe3M,17726
|
|
4
|
+
cpp_pinyin2/_native.py,sha256=OU9WWxMaOODB2LNL4wucQGkJhQgo2Est2aWUgJXxChM,3324
|
|
5
|
+
cpp_pinyin2/_version.py,sha256=19x9d1DtwFIMJSXpQhtPpHMtPK_Cf0voeHqerpS24GE,125
|
|
6
|
+
cpp_pinyin2/_native/libcpp-pinyin2.dll,sha256=sf9q7U8vtxQ8O2YZQfdKlJ_xH-CcN_Tu_K3apR-0GnA,3247186
|
|
7
|
+
cpp_pinyin2/_native/name.txt,sha256=MzahRmrBP6kxj467ZMhwNj4BxXzvRZdBz_qyh2Cv634,19
|
|
8
|
+
cpp_pinyin2/data/dict/cantonese/License.txt,sha256=KDu8_IH-3TxVXFRDFBkqh_4wto9vshGqEmPIU3JV_Jc,526
|
|
9
|
+
cpp_pinyin2/data/dict/cantonese/dialect.json,sha256=_F_w0shSIzB6OObulMRUqOPCL54fRRs5XO_8Sw6esf4,452
|
|
10
|
+
cpp_pinyin2/data/dict/cantonese/format.txt,sha256=U8I05ehHK2rFHBrhyrP-BvrQU7646_2Jd7AQZVv908M,2
|
|
11
|
+
cpp_pinyin2/data/dict/cantonese/phrases_dict.txt,sha256=fvlNETLtw3k3UKrD2cmwYZtXbgsvXY2Id4EJoApItoc,29369
|
|
12
|
+
cpp_pinyin2/data/dict/cantonese/phrases_map.txt,sha256=XSNQImSti2Miwef2TVYMorsLiteRb8mDOhLFfIaRmGk,2995
|
|
13
|
+
cpp_pinyin2/data/dict/cantonese/trans_word.txt,sha256=XWym6pAZWhg0xVKCPjp4OpjkTlCiuO7VKmveobQZa6c,14426
|
|
14
|
+
cpp_pinyin2/data/dict/cantonese/user_dict.txt,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
|
15
|
+
cpp_pinyin2/data/dict/cantonese/word.txt,sha256=HIPprd9jy6WZUG9xEQq8oRIqmi2EgQCvUnozW6Do0HI,238117
|
|
16
|
+
cpp_pinyin2/data/dict/mandarin/License.txt,sha256=Z06jTMwXp90opdGJtMNWIuCsNpLDY6zt8fkTMiGp6Yo,1128
|
|
17
|
+
cpp_pinyin2/data/dict/mandarin/dialect.json,sha256=laG9WwP53cyWXm92SDO-O05dGpAIMttqPPHzJGxuaCA,451
|
|
18
|
+
cpp_pinyin2/data/dict/mandarin/format.txt,sha256=U8I05ehHK2rFHBrhyrP-BvrQU7646_2Jd7AQZVv908M,2
|
|
19
|
+
cpp_pinyin2/data/dict/mandarin/phrases_dict.txt,sha256=d_mG-TtpJ6CQ7TGWPP1dKQUxH7OaOr3lfbCvJ4qTXXk,146439
|
|
20
|
+
cpp_pinyin2/data/dict/mandarin/phrases_map.txt,sha256=f8Tb8Z53CmorKOT1dWD1TBtNBx-r3tcUEBc9g4GJxyE,7087
|
|
21
|
+
cpp_pinyin2/data/dict/mandarin/trans_word.txt,sha256=YgRjkiDMx7bnQ4ALpJaMba8VpjrVepdoQCdzIPqta-4,49542
|
|
22
|
+
cpp_pinyin2/data/dict/mandarin/user_dict.txt,sha256=MVLl7M5a4-PjuYCevwYf_K1g291xQxM2RYSgwYf8qq0,2125
|
|
23
|
+
cpp_pinyin2/data/dict/mandarin/word.txt,sha256=q1iYz42JuYX9TtobXEZoDtz9AW9KDK0vQk8fpwsJD78,164968
|
|
24
|
+
cpp_pinyin2-2.0.0.dist-info/licenses/LICENSE,sha256=W6dIfTMOlfv3NPOOmhc_s2A-us9_aX2i2QR2y4fed88,11646
|
|
25
|
+
cpp_pinyin2-2.0.0.dist-info/licenses/THIRD_PARTY_NOTICES.md,sha256=k5uNhEvpz6tJaNkofNW78c_7vnQ62HpDCWNWsiyyFlw,2992
|
|
26
|
+
cpp_pinyin2-2.0.0.dist-info/METADATA,sha256=0bJNehGupE0aRLWKysDNm43QIxTPb3JGFskTIauPVTY,27976
|
|
27
|
+
cpp_pinyin2-2.0.0.dist-info/WHEEL,sha256=bhMFZwp6VMtrbl2Ck6PlXryKaCggO1NzYn_HcFbBF1I,97
|
|
28
|
+
cpp_pinyin2-2.0.0.dist-info/top_level.txt,sha256=BeJlT1zyXBtjq4ptA6RvnIyONr9B4LXAucc_0G3bhRQ,12
|
|
29
|
+
cpp_pinyin2-2.0.0.dist-info/RECORD,,
|