@kensio/pinyinjs 1.6.0 → 1.7.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (88) hide show
  1. package/README.md +211 -5
  2. package/data/core.entries +393 -392
  3. package/data/core.freq +0 -0
  4. package/data/core.keys +1 -0
  5. package/data/full.entries +10220 -10220
  6. package/data/manifest.json +12 -12
  7. package/data/script.map +7 -7
  8. package/data/standard.entries +957 -956
  9. package/data/standard.freq +0 -0
  10. package/data/standard.keys +1 -0
  11. package/dist/cli/arguments.d.ts +38 -0
  12. package/dist/cli/arguments.d.ts.map +1 -1
  13. package/dist/cli/arguments.js +46 -0
  14. package/dist/cli/arguments.js.map +1 -1
  15. package/dist/cli/commands.d.ts.map +1 -1
  16. package/dist/cli/commands.js +207 -1
  17. package/dist/cli/commands.js.map +1 -1
  18. package/dist/cli/help.d.ts.map +1 -1
  19. package/dist/cli/help.js +5 -0
  20. package/dist/cli/help.js.map +1 -1
  21. package/dist/decode/convert.d.ts +16 -0
  22. package/dist/decode/convert.d.ts.map +1 -1
  23. package/dist/decode/convert.js +3 -28
  24. package/dist/decode/convert.js.map +1 -1
  25. package/dist/decode/runs.d.ts +11 -0
  26. package/dist/decode/runs.d.ts.map +1 -1
  27. package/dist/decode/runs.js +19 -0
  28. package/dist/decode/runs.js.map +1 -1
  29. package/dist/decode/segment.d.ts +66 -0
  30. package/dist/decode/segment.d.ts.map +1 -0
  31. package/dist/decode/segment.js +79 -0
  32. package/dist/decode/segment.js.map +1 -0
  33. package/dist/dictionary/artifact.d.ts +10 -0
  34. package/dist/dictionary/artifact.d.ts.map +1 -1
  35. package/dist/dictionary/artifact.js +17 -0
  36. package/dist/dictionary/artifact.js.map +1 -1
  37. package/dist/dictionary/assertions.d.ts.map +1 -1
  38. package/dist/dictionary/assertions.js +38 -0
  39. package/dist/dictionary/assertions.js.map +1 -1
  40. package/dist/dictionary/corpus-mass.d.ts +68 -0
  41. package/dist/dictionary/corpus-mass.d.ts.map +1 -0
  42. package/dist/dictionary/corpus-mass.js +163 -0
  43. package/dist/dictionary/corpus-mass.js.map +1 -0
  44. package/dist/dictionary/dictionary.d.ts +56 -0
  45. package/dist/dictionary/dictionary.d.ts.map +1 -1
  46. package/dist/dictionary/dictionary.js +116 -1
  47. package/dist/dictionary/dictionary.js.map +1 -1
  48. package/dist/dictionary/erhua.d.ts +4 -0
  49. package/dist/dictionary/erhua.d.ts.map +1 -1
  50. package/dist/dictionary/erhua.js +6 -0
  51. package/dist/dictionary/erhua.js.map +1 -1
  52. package/dist/dictionary/merge.d.ts.map +1 -1
  53. package/dist/dictionary/merge.js +16 -8
  54. package/dist/dictionary/merge.js.map +1 -1
  55. package/dist/dictionary/tiers.d.ts +2 -1
  56. package/dist/dictionary/tiers.d.ts.map +1 -1
  57. package/dist/dictionary/tiers.js +77 -11
  58. package/dist/dictionary/tiers.js.map +1 -1
  59. package/dist/grade/check.d.ts +156 -0
  60. package/dist/grade/check.d.ts.map +1 -0
  61. package/dist/grade/check.js +356 -0
  62. package/dist/grade/check.js.map +1 -0
  63. package/dist/index.d.ts +11 -1
  64. package/dist/index.d.ts.map +1 -1
  65. package/dist/index.js +5 -0
  66. package/dist/index.js.map +1 -1
  67. package/dist/search/candidates.d.ts +84 -0
  68. package/dist/search/candidates.d.ts.map +1 -0
  69. package/dist/search/candidates.js +290 -0
  70. package/dist/search/candidates.js.map +1 -0
  71. package/dist/search/match.d.ts +72 -0
  72. package/dist/search/match.d.ts.map +1 -0
  73. package/dist/search/match.js +419 -0
  74. package/dist/search/match.js.map +1 -0
  75. package/dist/search/query.d.ts +49 -0
  76. package/dist/search/query.d.ts.map +1 -0
  77. package/dist/search/query.js +115 -0
  78. package/dist/search/query.js.map +1 -0
  79. package/dist/search/reverse-index.d.ts +141 -0
  80. package/dist/search/reverse-index.d.ts.map +1 -0
  81. package/dist/search/reverse-index.js +356 -0
  82. package/dist/search/reverse-index.js.map +1 -0
  83. package/dist/sources/unihan.d.ts +9 -0
  84. package/dist/sources/unihan.d.ts.map +1 -1
  85. package/dist/sources/unihan.js +12 -1
  86. package/dist/sources/unihan.js.map +1 -1
  87. package/package.json +2 -3
  88. package/CHANGELOG.md +0 -634
package/CHANGELOG.md DELETED
@@ -1,634 +0,0 @@
1
- # Changelog
2
-
3
- Notable changes to `@kensio/pinyinjs`. Versions follow
4
- [semantic versioning](https://semver.org) from 1.0.0 onward.
5
-
6
- Two things are outside the version contract, for the reasons
7
- [the API page](docs/api/#stability) gives: the readings a conversion produces,
8
- which a dictionary rebuild can change, and the artifact format under `data/`.
9
-
10
- ## Unreleased
11
-
12
- ## 1.6.0
13
-
14
- ### Added
15
-
16
- - **Hanzi and pinyin together, rather than one instead of the other.**
17
- `convertToAnnotatedHtml` writes the characters with their reading above them,
18
- as `<ruby>` markup — the output every other mode could not give, and what a
19
- learner's text, a subtitle and a dictionary entry all want. `convert` and
20
- `convertToHtml` write the reading _instead of_ the characters.
21
- `toAnnotatedHtml(pieces)` renders pieces you already have, and the command
22
- line has `pinyinjs annotate`.
23
-
24
- Each `<rt>` holds exactly what `convertToHtml` would have written, so tone
25
- classes and uncertainty marking work inside an annotation as they do outside
26
- one. `<rp>` gives the parenthesised fallback, the `<ruby>` carries `lang="zh"`
27
- and the syllables inside it keep `zh-Latn-CN-pinyin`, since on an English page
28
- a screen reader needs to be told both.
29
-
30
- **A base is not always one character**, which is what makes this harder than
31
- it looks and what every naive per-character annotation gets wrong. 玩儿 is two
32
- characters and the one syllable `wánr`; 95% is `bǎifēnzhījiǔshíwǔ` over three
33
- written characters in the other order; 1988 is `yī jiǔ bā bā` over one. All
34
- are annotated whole. `ConvertedPiece` gains a `source` — the characters a
35
- piece reads, or undefined where it reads on into the ones named before it —
36
- which is what makes that expressible.
37
-
38
- Pinyin orthography stays out of the hanzi it annotates: no word spaces, since
39
- Chinese is not written with them; the hyphen of `gāngān-jìngjìng` sits in the
40
- reading, being a boundary inside one word rather than a gap between two; and
41
- the base keeps 。 where the conversion writes a full stop.
42
-
43
- - **`readings`: a conversion takes readings the caller asserts.** No rule
44
- settles every polyphone, and some texts are genuinely ambiguous —
45
- 孩子越长越漂亮 grows where 头发越长越漂亮 lengthens, and nothing in the
46
- characters says which. An application that knows its own content can now say
47
- so, rather than the library guessing on its behalf.
48
-
49
- ```ts
50
- convert(dictionary, "这篇文章不太长。", { readings: { 太长: "tài cháng" } });
51
- ```
52
-
53
- A **word** hint asserts something about the text it names and rewrites exactly
54
- those characters, so a bare `长` leaves 校长 as `xiàozhǎng` — the dictionary
55
- knowing that word is better evidence about the stretch than a remark about one
56
- of its characters, and that is what makes a corrections table safe to
57
- accumulate. Naming the whole word does reach it, and keeps it whole:
58
- `{ 银行: "yín xíng" }` gives `yínxíng`.
59
-
60
- A **positional** hint asserts something about one character of one text, and
61
- nothing outranks it, the enclosing word included. Positions count code points
62
- from the start of the text, across any non-Han runs in it. The list form takes
63
- both kinds together.
64
-
65
- Hints enter as lattice edges rather than as rules, because the rules layer
66
- cannot invent a reading no source attests — which is exactly what a hint is,
67
- the difference being that a caller attests it. Spacing is untouched, an
68
- unmarked syllable is 轻声, and a hint that cannot be parsed throws rather than
69
- being skipped: a correction that silently does nothing is worse than one that
70
- fails.
71
-
72
- ### Fixed
73
-
74
- - **Polyphones defaulted to whichever reading a dictionary happened to print
75
- first.** Unihan's `kTGHZ2013` ranked above `kMandarin`, and that field
76
- prefixes every reading with its entry number in 《現代漢語規範詞典》 — 勒 is
77
- `212.050:lè 212.100:lēi` — so its order is a page order, and taking the first
78
- reading meant taking whichever one sorted earlier by pinyin. 殷 read `yān` and
79
- 楷 `jiē`. The dictionary-indexed fields now rank last: they are a good
80
- _source_ of readings and a meaningless _ranking_ of them.
81
-
82
- `kHanyuPinlu` still leads where it lists more than one reading, because there
83
- it has counted them against each other, but where it lists one it counted
84
- nothing — it is saying "this occurred", not "this leads" — and a lone reading
85
- now ranks last too. Only ten characters have one that disagrees with
86
- `kMandarin`, and eight of those are simply pre-1985: the corpus behind the
87
- field, 《現代漢語頻率詞典》, predates the 普通话异读词审音表, and the 132
88
- occurrences carrying 绩 as `jī` are the old standard rather than a case for
89
- it.
90
-
91
- 354 readings fixed against 94 broken over 25 characters, the largest being 贾
92
- `gǔ`→`jiǎ`, 苔 `tāi`→`tái`, 町 `dīng`→`tǐng`, 殷 `yān`→`yīn` and 芥
93
- `gài`→`jiè`. CPP's polyphone score is 90.34%, up from 89.05%, and the gold
94
- corpus does not move. What broke is mostly `kMandarin` being wrong with
95
- nothing behind it — 媛 `yuán`→`yuàn`, 桔 `jú`→`jié`, 耶 `yē`→`yé`, 缪
96
- `miào`→`móu` — which needs a third opinion with corpus evidence behind it
97
- rather than an override, since here the sources disagree. 䘚 is now paired
98
- with 卒, so `core` and `standard` each gained a key.
99
-
100
- - **`full` read 特徵 as `tèzhǐ` where `standard` read it right.**
101
- `large_pinyin.txt` is a 简体 corpus, and where it carries a 繁體 headword
102
- anyway it has read it as though the characters were 简体, taking each at face
103
- value instead of as the 简体 character it stands for: 徵 as itself rather than
104
- as 征, 沈 as the surname rather than as 沉, so 沈溺 came out `shěnnì`. Of the
105
- 2,854 corpus headwords carrying a 繁體-only character, 72 are ones CC-CEDICT
106
- pairs with a 简体 form and 13 of those read wrongly.
107
-
108
- The reading was only half of it. A corpus headword becomes an entry of its
109
- own, and an entry outranks the 繁體 key that the traditional derivation hangs
110
- on the 简体 entry — so the word was in the dictionary twice, and which copy
111
- answered depended on the tier. `standard` excludes the phrase tail and was
112
- right; `full`, the default, was wrong. Loading more dictionary made the answer
113
- worse, which is also the upgrade path the docs describe, of converting with
114
- `standard` and reloading as `full`.
115
-
116
- A headword CC-CEDICT knows _only_ as another word's 繁體 spelling now
117
- contributes no entry, and the word keeps the reading CC-CEDICT pairs it with.
118
- 68 entries drop out and their keys are re-claimed from the 简体 entry, so
119
- `full` loses 2 keys; 简→繁→简 round trips gain 38 against 5 lost on 繁→简→繁,
120
- both unchanged as percentages. The 2,782 headwords CC-CEDICT does not pair are
121
- rare or mixed-script words whose characters do read them, and are left alone.
122
- Three of the words are build assertions now, this being the kind of defect
123
- that comes back when a rolling source is refreshed and nothing is watching.
124
-
125
- - **长 read `zhǎng` wherever no word covered it**, so 这篇文章不太长 came out
126
- `bú tài zhǎng` and 要多长时间 `duō zhǎng shíjiān`. The stored default is not
127
- the defect — Unihan ranks the readings `zhǎng(1879) cháng(1179)` and names
128
- `zhǎng` in `kMandarin`, and that ranking is what reads 署长, 团长, 公安局长 and
129
- 总会长 correctly off a bare 长 at the end of a title. What was missing is that
130
- nothing could prefer the alternate at a position no word covers.
131
-
132
- A degree adverb settles it, and only from the left: a growing 长 is a verb and
133
- no 很, 太, 最 or 多 can modify one, while what follows an adjectival 长 is a
134
- noun, a particle or the end of the sentence — which is what follows half the
135
- verbs too. 得, 着 and the 越…越 correlative are the exceptions, and are guarded;
136
- 了 and 的 are deliberately not, since after an adverb they are the sentence
137
- particle and the attributive, making 时间太长了 and 很长的道路 both `cháng`.
138
-
139
- Measured over the same 88,866 lines of Tatoeba and zh.wikipedia the other
140
- edge rules were sized against, 260 长 decode as a word of their own and this
141
- moves 75 to `cháng`, all 75 correctly. On CPP's 40 hand-labelled 长 the
142
- character goes 85.00% to 87.50%; overall CPP is unmoved at 90.34%, as expected
143
- of one character in 20,139.
144
-
145
- - **越长越高 read `yuè cháng yuè gāo`.** 越长 is a key read `yuè cháng` and the
146
- only one of its shape — 越大, 越高, 越好 and 越快 are all absent, so 越高越好
147
- decodes as two words while 越长越高 reached for a word nothing else in the
148
- paradigm has. It carries no part of speech, which is how a reading somebody
149
- asserted is held rather than a word anybody counted, and only
150
- `large_pinyin.txt` has it. Where the far half of the correlative names
151
- something growing produces — 高, 大, 胖, 壮, 结实 — the edge is dropped and the
152
- character's own `zhǎng` stands.
153
-
154
- Unlike everything else here this is a heuristic with no corpus behind it: 越长
155
- occurs three times in the 88,866 lines and all three are 越来越长 or 说的越长.
156
- It forbids rather than forces, so `cháng` stays on as a rival one bucket
157
- dearer and the syllable still reports itself uncertain — 越长越X is genuinely
158
- ambiguous, 孩子越长越漂亮 growing where 头发越长越漂亮 lengthens, which is why
159
- 漂亮 is not in the list. 时间越长越好 and 头发越长 are untouched.
160
-
161
- ## 1.5.3
162
-
163
- ### Fixed
164
-
165
- - **The changelog inside the package described five releases as unreleased.**
166
- Every tarball from 1.1.0 to 1.5.2 shipped a `## Unreleased` section holding
167
- everything since 1.0.0, so a reader who opened the copy they installed —
168
- which [the API page](docs/api/#stability) points them at — had no way to tell
169
- which version any of it arrived in. The entries are filed under the versions
170
- that shipped them now. No code changes; this release exists to put the
171
- corrected file on the registry.
172
-
173
- ## 1.5.2
174
-
175
- ### Fixed
176
-
177
- - **Third-tone sandhi ran over the syllables rather than over the feet.** The
178
- rule is usually stated as "a third tone before another third tone is said as
179
- a second", and a left-to-right scan of a reading applying exactly that is
180
- wrong about as often as it is right: sandhi's domain is the prosodic foot,
181
- which is built out of structure ([Shih 1986][1]). 這家銀行的行長很喜歡旅行
182
- came out `hángzháng hén xǐhuan`, running the subject into the predicate, and
183
- 紙老虎 came out `zhíláohǔ`.
184
-
185
- Three passes now, innermost first, each reading what the one before it left.
186
- Inside a constituent, every third tone but the last lowers. Between the
187
- constituents of a word, so that 展覽館 is 展覽 + 館 and stays `zhánlánguǎn`
188
- while 紙老虎 is 紙 + 老虎 and becomes `zhǐláohǔ` — the dictionary is asked
189
- where a word divides, and only proposes one where both halves are words. And
190
- between a monosyllabic word and the word after it, which it leans on: 我也很好
191
- is `wó yé hén hǎo`, but 行長 and 老闆 keep their second syllable.
192
-
193
- `applySandhi` takes a `SandhiGrouping` for this, and without one still treats
194
- the whole reading as a single word. `convert` passes what the decoder found;
195
- the `sandhi` command splits its argument on whitespace.
196
-
197
- What is given up is the monosyllable leaning backwards — 保管好 is
198
- `báoguán hǎo` and comes out `báoguǎn hǎo` — which needs to know which way it
199
- attaches, a question about syntax rather than about the words.
200
-
201
- [1]: https://www.researchgate.net/publication/36071823_The_Prosodic_Domain_of_Tone_Sandhi_in_Chinese
202
-
203
- - **14 characters read wrongly under `zh-TW`.** CC-CEDICT states a `Taiwan pr.`
204
- note either as a definition of its own, where it qualifies the headword's
205
- reading, or parenthesised inside one sense, where it qualifies that sense —
206
- and the merge took both as facts about the headword. 從's `zòng` is the 侍從
207
- and 從兄弟 senses, all of them bound forms, so `我從北京來` read
208
- `wǒ zòng Běijīng lái`; 會 took `huǐ` from 一會兒 and `很會說話` followed it.
209
- A note now has to cover the sense the entry leads with, which is what a
210
- character means with nothing to narrow it — 和's `hàn` is the conjunction, and
211
- it stays. Also fixed for 勞, 燥, 行, 勝, 匹, 多, 抵, 枕, 比, 玩, 署 and 聽.
212
- Compounds keep their own notes, so 肉燥麵 is still `ròusào miàn`.
213
-
214
- Only a character is tested this way; a word is reached only where it is
215
- written, so 相親 keeps the `xiàngqīn` its dominant sense carries. CC-CEDICT
216
- now supplies 335 deltas rather than 349.
217
-
218
- ## 1.5.1
219
-
220
- Tooling and test-suite work only; nothing a consumer installs changed.
221
-
222
- ## 1.5.0
223
-
224
- ### Added
225
-
226
- - **`convertToHtml` declares what language it is emitting.** Every syllable
227
- element now carries `lang="zh-Latn-CN-pinyin"`, or `zh-Latn-TW-pinyin` where
228
- the conversion read `zh-TW`. Nothing about `yín` on its own says it is
229
- Mandarin in the Latin alphabet, so without the tag a screen reader pronounces
230
- it as whatever the surrounding page claims to be — English, usually — and the
231
- browser hyphenates it that way too. The subtags are all registered, and the
232
- `pinyin` variant's prefix in the IANA registry is `zh-Latn`.
233
-
234
- **This changes the markup existing callers get**, which is why it is worth
235
- saying plainly: the classes are unchanged, and `lang` is a new attribute
236
- between `class` and `data-alternatives`. `lang: false` — `--no-lang` at the
237
- command line — leaves it off, for a caller who would rather declare the
238
- language once on a wrapper of their own than repeat it on every syllable.
239
-
240
- ## 1.4.0
241
-
242
- ### Added
243
-
244
- - **`pinyinjs script`,** script conversion at the command line, with `--to` for
245
- the target and `--from-script` where the text is too short for detection to
246
- tell. The lines carry the converted text and nothing else so it pipes;
247
- `--json` carries the per-character evidence and what was uncertain. Also a
248
- [script conversion](docs/script-conversion/) page, which the README and the
249
- docs index now point at — `toScript` shipped with one row in the API table
250
- and nothing a reader would find.
251
-
252
- - **`toScript`,** 简体 ↔ 繁體 conversion. 我们后来发现了头发问题 is
253
- 我們後來發現了頭髮問題. The reading is the evidence, in both directions, and it
254
- is what nothing else in this space has: 发 is 發 or 髮 and only `fà` tells them
255
- apart; 干 is 幹, 乾 or 干; 乾 stays 乾 in 乾隆 and simplifies to 干 in 乾燥.
256
- Every other converter resolves these from a phrase table alone. This one
257
- segments with the lattice and reads the result, so it generalises to words no
258
- phrase table lists.
259
-
260
- Evidence is taken strongest first — a word some source wrote in both scripts,
261
- then the character at the reading it was decoded with, then the character's
262
- commonest form — and `toScriptPieces` reports which of those settled every
263
- character. `isUncertainChoice` marks the last one, the only case that is a
264
- guess: 下面 is a surface or a bowl of noodles and nothing here can tell, so it
265
- says so rather than picking silently. Over the gold corpus 97.7% of characters
266
- are locked and 1.5% are guesses.
267
-
268
- Targets are BCP-47: `zh-Hans`, `zh-Hant`, `zh-Hant-TW`, `zh-Hant-HK`, with a
269
- bare `zh-Hant` taking Taiwan. The input script is detected unless `from` names
270
- it, which matters more than it sounds — plenty of characters are current in
271
- both scripts, so running 繁體 through the 简→繁 tables would turn 准將 into
272
- 準將.
273
-
274
- **This is orthography, not translation.** 軟體 and 软件 are different words
275
- rather than two spellings of one. OpenCC folds that into script conversion
276
- with `s2twp`; pinyinjs does not.
277
-
278
- - **`data/script.map`,** the conversion tables, as their own fetchable file:
279
- 282 KB, 96 KB brotli, loaded by `loadScriptTables` only when something asks
280
- to convert. A caller converting hanzi to pinyin pays nothing for it. The
281
- character tables alone are right for 99.13% of the 267,639 entries whose two
282
- scripts differ, and the word exception lists cover the rest.
283
-
284
- - **`hansOnly` and `hantOnly`,** the character sets `detectScript` has always
285
- taken and nothing shipped. They ride in the same file.
286
-
287
- - **Round-trip scoring in `pnpm accuracy`.** 繁→简 is near-deterministic, so
288
- 简→繁→简 has to be the identity for essentially every key — half a million
289
- cases with no hand-labelling, and every failure a real defect. It reports
290
- 99.6% over the words in use and 99.3% over every key. The reverse trip is
291
- lossy by design and is reported rather than targeted.
292
-
293
- - **Hong Kong 繁體 glyph forms are recognised on the lookup path.** Taiwan and
294
- Hong Kong write the same characters with the same readings and disagree about
295
- the standard form of 58 of them — 裡 against 裏, 群 against 羣, 麵 against 麪.
296
- Hong Kong spellings were missing dictionary keys and falling back to
297
- converting character by character, so 羣眾 and 麪包 read worse than 群眾 and
298
- 麵包 did. The lookup path now normalises 繁體 glyph forms to the Taiwan
299
- standard the keys are already built from: measured over the 9,833 keys with a
300
- distinct Hong Kong spelling, 2,358 were already keys and normalisation reaches
301
- a further 5,439. Only the reading is taken from the normalised form — the
302
- characters a caller gets back are the ones they wrote. `toCanonicalGlyphs`,
303
- `toRegionalGlyphs` and the `Region` type are exported for it, and
304
- SCRIPTS-AND-LOCALES.md carries the reasoning.
305
-
306
- Nineteen of OpenCC's mappings are deliberately not applied, because the
307
- variant is a live spelling rather than a regional one. Six are current 繁體 —
308
- 台 appears 164 times in CC-CEDICT against 臺's 208 — and thirteen are current
309
- **简体**, because the PRC simplification adopted the same 新字形 conventions
310
- Hong Kong did: 着, 温, 脱, 户, 税, 卧, 悦, 兑, 葱, 幺, 棱, 檐, 痹. Normalising
311
- those would have rewritten 简体 text, and 走着 `zǒuzhe` is the case that
312
- caught it.
313
-
314
- - **OpenCC as a data source.** Its 简体 ↔ 繁體 character tables and the Taiwan
315
- and Hong Kong variant tables, Apache-2.0, attributed in `NOTICE`. The glyph
316
- tables are held in `src/script/glyphs.ts` as code rather than shipped as an
317
- artifact, since 120 mappings do not justify a download and they have to work
318
- before any dictionary has loaded; `pnpm build:data` re-derives them from
319
- OpenCC and fails the build if the two have drifted.
320
-
321
- ### Fixed
322
-
323
- - **240 dictionary entries could not be looked up at all.** Recognising Hong
324
- Kong glyph forms on the lookup path means a key written in a form the path
325
- normalises _away from_ can never be found, and the merge derived 繁體 forms
326
- per word from Unihan's variant lists with no single standard behind them — so
327
- it wrote 裏 where the corpus overwhelmingly writes 裡, and 上樑, 中峯 and 义藴
328
- became keys nothing could reach. The merge now derives canonical glyph forms,
329
- and keys a headword's canonical spelling alongside it where the phrase corpus
330
- wrote one with a 繁體 variant. **No key is now unreachable**, and Hong Kong
331
- spellings resolve slightly better than before: 5,499 of the 9,978 keys with a
332
- distinct Hong Kong form, against 5,439.
333
-
334
- Two things had to be got right alongside it. The evidence for "简体 writes
335
- this character" comes from CC-CEDICT entries whose two columns differ, not
336
- from the phrase corpus, which is nominally 简体 but carries 峯, 藴 and 枱 of
337
- its own — counting it cut the table from 50 mappings to 32 and stopped Hong
338
- Kong text converting at all. And a 繁體 form differing from its 简体 one only
339
- by glyph normalisation is not evidence about script, or the alias key would
340
- put that entry's 简体 characters into the 繁體-only set, which is how 发
341
- briefly stopped being 简体-only.
342
-
343
- `toScript` drops the pass that was papering over this. Key counts move:
344
- 723,149 for `full`, 97,997 for `standard`, 16,975 for `core`.
345
-
346
- ## 1.3.0
347
-
348
- ### Added
349
-
350
- - **`slug`,** hanzi to a URL-safe slug: 我想学中文。 is
351
- `wo3-xiang3-xue2-zhong1wen2`. Built on the decode rather than on a finished
352
- string, which is the whole point of it — 银行 is `yin2hang2` and not
353
- `yin2xing2`, the separators fall on the word boundaries 分词连写 found, and
354
- 西安 keeps its 隔音符号 as a break so a toneless slug is `xi-an` rather than
355
- `xian`, which is a different word. Tones are written by default because
356
- dropping them collides, and `hash` puts a short hash of the _hanzi_ on the end
357
- for the homophones tones cannot separate: 权利 and 权力 are both `quan2li4`.
358
- The options generalise it past URLs to search keys and identifiers, which the
359
- [slug page](docs/slug/) tabulates. `pinyinjs slug` at the command line.
360
-
361
- ## 1.2.2
362
-
363
- ### Added
364
-
365
- - **`TEACHING_JIAO` and `COUNTED_MEASURE`,** two more lattice rules, and
366
- `QUANTITY_CHARACTERS`, the 汉字 a quantity is written with that the second of
367
- them asks about. See the fixes below for what they do and what they measure.
368
-
369
- - **`counts` on the numeral options.** Whether the number stands immediately in
370
- front of something it counts, which is what makes a lone 2 两:
371
- `numeralHanzi(2, { counts: true })` is `两` and `numeralHanzi(2)` is `二`.
372
- Unlike `liang` it is not a variable choice — 二个 is simply wrong — but
373
- nothing about the number says whether it is counting, so the caller says.
374
- `convert` sets it from the character after the digits.
375
-
376
- ### Fixed
377
-
378
- - **教 was `jiào` even where it was teaching.** 他在北京大学教了三年书 came out
379
- `jiàole sān nián shū`, and 我教英语, 谁教你法语 and 她教我如何游泳 with it.
380
- The dictionary stores 教 as `jiào` with `jiāo` as an alternate, and `jiào` is
381
- right for the compounds — 教育, 教师, 宗教, 主教 — which are words and reach
382
- their reading through the word; what is left is the 教 standing as a word of
383
- its own, and that one is the verb. The object is what says so: a pronoun, a
384
- noun or a name after it, or an aspect particle, since only a verb takes 了,
385
- 过 or 得. Forcing the character's own reading was not enough on its own,
386
- because a two-character reading carries its 教 into the position from
387
- outside — 王老师教我们汉语 read `jiào` off 师教 and 来教我 off 来教 — so an
388
- untagged pair ending in 教 goes too, while every tagged word keeps its
389
- reading: 任教, 宗教, 主教, 佛教, 传教, 执教, 请教, 家教. Over 88,866 lines
390
- 181 教 decode as a word of their own and every one read `jiào`; 158 are now
391
- `jiāo`, three of them wrongly — 统一教创始人, 方法教深思 and 做到了教政分离,
392
- where a nominal compound takes an object's shape. CPP's polyphone score is
393
- 89.05%, up from 89.04%.
394
-
395
- - **A 量词 was swallowed by the word behind it.** 三个人 was `sān gèrén`, three
396
- _personals_, because 个人 is a common noun and nothing weighed it against the
397
- 个 belonging to the 三 in front of it. It is now `sān gè rén`, and so are
398
- 两家人, 三杯水, 四匹马, 十八层楼 and 一段长时间. What makes it decidable is
399
- that the dictionary tags the 量词: 个, 次, 天 and 杯 are `q`, while the
400
- characters that only look like measure words in this position are not — 分 is
401
- a verb, 部 and 成 are nouns, 年 and 点 are numerals — so 五分钟, 三部分,
402
- 五成分, 五年级 and 三点钟 are untouched, and those are exactly the words a
403
- blanket rule would break. Nor does an ordinal count anything (第三集团军), nor
404
- a numeral inside a longer word (唯一道路), and a number and its measure
405
- written solid survive whole where the dictionary has them: 一辈子, 一会儿,
406
- 一口气 and 两口子 are unchanged. Over 88,866 lines the rule forbids 561 edges
407
- and moves 53 decodes, three of them wrongly — 一批评, 这一名词 and 六七股灾,
408
- where the 一 and the 六七 count nothing. Exactly one reading changes in the
409
- whole corpus and it is a fix: 已经下了两天雨了 read 天雨 as `tiān yù`.
410
-
411
- - **The Han after a number was decoded without it.** 2个人 was `liǎng gèrén`
412
- while 两个人 written out was already `liǎng gè rén`, because a Han run was
413
- decoded on its own and had no idea a number preceded it. A run is now decoded
414
- with the 汉字 the digits stand for in front of it, and reported without them,
415
- so digits and 汉字 reach the same decode. `decodeRun` and `decodeRunScored`
416
- take that context as a fourth argument; where a reading would straddle the
417
- join — 1点儿事, whose 一点儿 is one `yìdiǎnr` over both sides of it — the run
418
- is decoded alone, as before.
419
-
420
- - **A lone 2 in front of a 量词 was read 二.** 我们买了2个西瓜 came out as
421
- `wǒmen mǎile èr gè xīguā`, and a 2 standing immediately in front of what it
422
- counts is 两: it is now `liǎng gè`, and 2岁 is `liǎng suì`, 2点 `liǎng diǎn`
423
- and 2万人 `liǎng wàn rén`, the same 两 that 2,000 and 2:00 already got. The
424
- measure words a 2 can count with are an open list, so 两 is the default and
425
- the exceptions are named — 月, 日, 号, 楼, 路, 班 and 期, where the digit
426
- labels a position rather than counting (2月 is `èr yuè`), an ordinal, which
427
- the 第 in front of it marks (第2次 is `dì èr cì`), and 十 and 百, which follow
428
- 二 as 20 and 200 already do. A 2 inside a larger number is untouched: 12个 is
429
- still `shí'èr gè`.
430
-
431
- ## 1.2.1
432
-
433
- ### Fixed
434
-
435
- - **A decimal lost the grouping of its counted part.** 一共75.5元 was
436
- `yígòng qī shí wǔ diǎn wǔ yuán`, with the 75 in loose syllables, and is now
437
- `yígòng qīshíwǔ diǎn wǔ yuán`, the same word 一共75元 already got. Only what
438
- follows the 点 is read a digit at a time, and that part is unchanged: 3.14 is
439
- still `sān diǎn yī sì`, and sandhi still stops at the point.
440
-
441
- - **A decimal point was read as a full stop.** 一共75.5元 came back as
442
- `Yígòng …`, capitalised as though the text were a sentence, and a conversion
443
- keeping its digits capitalised again after the point: 我有75.5个。was
444
- `Wǒ yǒu75.5Gè.` A stop between two digits now ends no sentence.
445
-
446
- - **A compound lost the zh-TW reading its own parts carry.** 垃圾分類 read
447
- `lājīfēnlèi` under `zh-TW`, because CC-CEDICT marks `Taiwan pr.` on 垃圾 and
448
- 垃圾桶 and on no other compound, and the decoder prefers the longer word, so
449
- the entry that had the delta was never consulted. 101 compounds now compose
450
- one from their constituents, 垃圾 accounting for 14 of them and 比肩, 玳瑁,
451
- 从容, 骨头, 蜗牛 and 说服 for most of the rest. Three guards keep it honest:
452
- the constituent has to survive segmentation (運行狀況 contains 行狀 but reads
453
- 運行 + 狀況), it has to be read in the compound exactly as its own entry reads
454
- it, and it has to be a whole word rather than a bare character, since 會's `huǐ`
455
- would rewrite 三合會 as `sānhéhuǐ` and 3,743 entries would compose that way
456
- against the 101 that whole words compose.
457
-
458
- - **A character's zh-TW delta was often a choice between its 普通话 senses.**
459
- 从容地 read `cōngróng dì` and 他都知道 read `tā dū zhīdào` under `zh-TW`,
460
- because 地 was stored with a Taiwan reading of `dì` and 都 with `dū`. Neither
461
- is a locale shift: CC-CEDICT lists 地[de5] and 地[di4] as separate entries, so
462
- `dì` is what 地 reads in 普通话 when it means the ground, and the adverbial
463
- particle is `de` in Taipei too. A delta the word already has as a 普通话 sense
464
- is now refused, so 71 characters and 3 words lose one, 都, 着, 应, 差, 称, 斗,
465
- 舍, 薄 and 万 among them, while 和 `hàn`, 期 `qí`, 垃 `lè` and 髮 `fǎ` stay,
466
- no 普通话 sense reading any of those that way. A `Taiwan pr.` note is also
467
- only read off a sense matching the reading the entry settled on, which drops
468
- 16 more: the note on 著's chess-move sense, which reads `zhāo`, had been
469
- giving the aspect particle 着 a 國語 reading of `zhuó`.
470
-
471
- - **A locale delta overwrote a reading it was not measured against.** The delta
472
- describes the entry's own 普通话 reading, so a polyphone the decode read some
473
- other way, such as 得 forced to the modal `děi` rather than the particle `de`
474
- its entry stores, now keeps what the decode worked out instead of being
475
- rewritten from the entry.
476
-
477
- ## 1.2.0
478
-
479
- ### Fixed
480
-
481
- - **`--system ipa` capitalised the IPA.** 我去银行。他姓王。 came out as
482
- `Uo˨˩˦ ... Tʰa˥ ɕiŋ˥˩ Uaŋ˧˥`, with the sentence capital and the proper noun
483
- carried over from the pinyin as though a transcription spelled words. IPA
484
- letters are symbols rather than an alphabet, since `[T]` is not `[t]` in a
485
- larger size but a symbol the IPA does not have, so the capitals are now dropped for
486
- IPA and bopomofo and kept for the three romanisations. `toTranscription`
487
- takes `{ capitals: false }` for callers writing their own.
488
-
489
- ## 1.1.0
490
-
491
- ### Added
492
-
493
- - **`SYLLABLE_TONES` and `isAttestedTone`.** Which tones each of the 424
494
- syllables is actually written in, extracted from the merged dictionary and
495
- held to it by a build assertion. 424 syllables in five tones would be 2,120
496
- combinations and only 1,708 of them occur, so a fifth of that grid is empty.
497
-
498
- - **`Syllable.originalTone`,** the tone a neutral syllable had before it was
499
- reduced. Gwoyeu Romatzyh writes the neutral tone as a dot in front of the
500
- syllable _in its original tonal spelling_, so 没有 méiyou is `mei.yeou`, and
501
- nothing else in a pinyin syllable records what that tone was. Optional, never
502
- inferred, and used by GR alone.
503
-
504
- - **Two more outside sources for the transcription tables.** 148 words from
505
- en.wiktionary's Mandarin pronunciation blocks, in bopomofo, Wade-Giles, Yale
506
- and GR with their tones, and all 50 rows of Wikipedia's _Help:IPA/Mandarin_
507
- key. The 417-row syllabary already checked was toneless, so it could say
508
- nothing about tone marks, the neutral tone, 儿化 or how a word is joined,
509
- which is where the four fixes below were found.
510
-
511
- ### Fixed
512
-
513
- - **Bopomofo wrote the tone mark after the 儿化 ㄦ.** 哪儿 nǎr was `ㄋㄚㄦˇ` and
514
- is now `ㄋㄚˇㄦ`: the mark belongs to the nucleus and the suffix is not part of
515
- what it marks. A mark written after the ㄦ is still read.
516
-
517
- - **Wade-Giles put the tone digit on the 儿化 suffix,** so 花儿 huār was
518
- `hua-êrh¹`, a first-tone 兒. It is now `hua¹-'rh`, with the digit on the
519
- syllable the tone belongs to and the suffix written as the reduced `'rh` that
520
- keeps it apart from 女儿 nǚ'ér's `nü³-êrh²`.
521
-
522
- - **Gwoyeu Romatzyh wrote 儿化 as a plain `-l` suffix.** GR fuses it into the
523
- rime instead, and the rules are now implemented as _Spelling in Gwoyeu
524
- Romatzyh_ gives them: 玩儿 wánr is `wal` rather than `wanl`, 事儿 shìr is
525
- `shell`, 今儿 jīnr is `jiel`. The fusion is many-to-one, so `jiel` is 今儿 and
526
- 鸡儿 both and 128 forms no longer round-trip, every one of them a form the
527
- language does not write.
528
-
529
- - **Gwoyeu Romatzyh wrote `.mhe` for a neutral 么.** The `-h-` is the first tone
530
- of a sonorant initial, and a neutral syllable is in no tone at all; the basic
531
- form goes behind the dot, so 什么 shénme is `shern.me`.
532
-
533
- - **The IPA module said it followed _Help:IPA/Mandarin_ and did not.** It
534
- follows the broader IPA column of _Comparison of Standard Chinese
535
- transcription systems_, which is where its ground truth comes from; the two
536
- pages differ at the medials, -ang, the empty rhyme and the diphthongs. No
537
- output changed, only the claim.
538
-
539
- ### Changed
540
-
541
- - **Reading a romanisation narrows on the tone that was written.**
542
- `readWadeGiles("lo²")` was `[luó, ló]` and is now `[luó]`: 咯 is a
543
- sentence-final particle and is only ever neutral, so ló is not a syllable
544
- Mandarin has. The same goes for `readWadeGilesLoosely`, where `pan²` is pán
545
- alone, and for `readYale`, `readIpa` and `readGwoyeu`, where it settles the 儿化
546
- collisions the same way (`ell` is 二 èr, there being no first-tone 兒).
547
- Narrowing never empties a list: a tone no candidate is written in leaves the
548
- candidates alone, since that says the tone is wrong rather than the spelling.
549
- Over the phrase corpus, taking the first candidate recovers 82.66% of
550
- syllables when the tone digit is written, against 79.05% without it.
551
-
552
- - Eight Wade-Giles forms no longer round-trip, and one to three in each of the
553
- other systems, because they are forms the language does not write: `lo` in the
554
- four contour tones, a first-tone 兒, and a neutral 誒. Every attested
555
- syllable-and-tone combination still comes back.
556
-
557
- ## 1.0.0
558
-
559
- The first stable release. Everything the
560
- [API page](docs/api/) lists is now covered by semantic versioning, including the
561
- decoder internals and the build pipeline.
562
-
563
- ### Added since 0.1.0-beta.0
564
-
565
- - **Transcription systems, all five, in both directions.** Bopomofo, Wade-Giles,
566
- Yale, Gwoyeu Romatzyh and IPA at the syllable level, plus hanzi → any of them
567
- end to end via `convert --system` and `convertToWadeGiles`. Checked against
568
- 417 rows of an outside syllabary in every system, which is 3,336 cells.
569
- - **Reading ambiguous Wade-Giles.** `readWadeGilesLoosely` allows for the
570
- apostrophes and diacritics real texts drop, and `splitWadeGiles` puts back the
571
- hyphens they lose. Measured: 52.07% of written syllables lose their identity
572
- without their marks, and taking the first candidate recovers 79.05%.
573
- - **Numbers read aloud** (`src/numerals/`, the `number` command): counting and
574
- spelling out, 两 against 二, decimals, percentages, fractions, `yāo`, and
575
- times. `convert` reads the digits it meets and takes the style from what
576
- follows them.
577
- - **Orthography rules.** The 重叠 hyphen for AABB and repeated words, the 成语
578
- hyphen from a curated list of 117, 老王 → `Lǎo Wáng`, and GB/T 16159 5.1 in
579
- full, so 毛泽东 is `Máo Zédōng` and 北京大学 is `Běijīng Dàxué`, on boundaries
580
- CC-CEDICT's own capitalisation states.
581
- - **Typed rule overrides on the lattice** (`READING_RULES`), with the modal 得
582
- and attested 儿化 as the first two.
583
- - **Tone colour in the terminal**, in MDBG's palette, on every command that
584
- writes a syllable.
585
-
586
- ### Fixed since 0.1.0-beta.0
587
-
588
- - **一 sandhi is about the 一 that counts.** 十一月 is `shíyīyuè` and 第一次 is
589
- `dìyīcì`; the citation tone survives a final digit and an ordinal. 561
590
- conversions change over 88,866 lines, 520 of them right.
591
- - **A colon between digits is a time**, not an identifier: 6:30 is
592
- `liù diǎn sānshí fēn`.
593
- - **The documented proper-noun mechanism was wrong.** The capitals page
594
- described a 500-entry surname list that has never existed; what produces
595
- `Lǐ Huá` is two single-character dictionary entries each flagged a proper
596
- noun.
597
-
598
- ### Changed
599
-
600
- - **`engines` is now `>=22.0.0`**, down from `>=24.0.0`. The suite and the
601
- packaged tarball are both exercised on Node 22 and 24 in CI.
602
- - **No more `beta` dist-tag.** `pnpm add @kensio/pinyinjs` finds it.
603
- - `src/romanization/` is `src/transcription/`, since bopomofo has a script of
604
- its own and IPA is a transcription rather than a spelling. The published docs
605
- path keeps the older name.
606
- - The build cleans `dist/` before compiling, and `pnpm pack:check` asserts the
607
- tarball's contents. The beta tarball shipped 24 stale files from a directory
608
- that had been renamed away.
609
-
610
- ### Accuracy at 1.0.0
611
-
612
- Over the 130-case gold corpus, which is committed to the repository rather than
613
- to the published package, and scored by `pnpm accuracy`:
614
-
615
- | | lattice | greedy baseline |
616
- | ---------------- | --------: | --------------: |
617
- | exact match | **97.7%** | 86.9% |
618
- | reading accuracy | 99.7% | 97.9% |
619
- | spacing (F1) | **99.8%** | 95.7% |
620
-
621
- Every figure in that table is asserted against the scorer by
622
- `src/changelog.test.ts`, because a number nothing executes goes stale, which is
623
- what happened to this project's README table across three releases.
624
-
625
- `pnpm polyphones` reports 89.04% for the lattice against greedy's 88.82% over
626
- CPP's 20,139 hand-labelled polyphones. That corpus is fetched rather than
627
- committed, so unlike the table above it cannot be asserted here, and it is
628
- reported rather than guarded.
629
-
630
- ## 0.1.0-beta.0
631
-
632
- First published release: the lattice decoder, the tiered dictionaries, the
633
- syllable layer, sandhi, GB/T 16159 spacing and capitals, HTML output and the
634
- `pinyinjs` command.