@kensio/pinyinjs 1.3.0 → 1.5.0

package/CHANGELOG.md

@@ -9,8 +9,124 @@ which a dictionary rebuild can change, and the artifact format under `data/`.
 
 ## Unreleased
 
+### Fixed
+
+- **240 dictionary entries could not be looked up at all.** Recognising Hong
+  Kong glyph forms on the lookup path means a key written in a form the path
+  normalises _away from_ can never be found, and the merge derived 繁體 forms
+  per word from Unihan's variant lists with no single standard behind them — so
+  it wrote 裏 where the corpus overwhelmingly writes 裡, and 上樑, 中峯 and 义藴
+  became keys nothing could reach. The merge now derives canonical glyph forms,
+  and keys a headword's canonical spelling alongside it where the phrase corpus
+  wrote one with a 繁體 variant. **No key is now unreachable**, and Hong Kong
+  spellings resolve slightly better than before: 5,499 of the 9,978 keys with a
+  distinct Hong Kong form, against 5,439.
+
+  Two things had to be got right alongside it. The evidence for "简体 writes
+  this character" comes from CC-CEDICT entries whose two columns differ, not
+  from the phrase corpus, which is nominally 简体 but carries 峯, 藴 and 枱 of
+  its own — counting it cut the table from 50 mappings to 32 and stopped Hong
+  Kong text converting at all. And a 繁體 form differing from its 简体 one only
+  by glyph normalisation is not evidence about script, or the alias key would
+  put that entry's 简体 characters into the 繁體-only set, which is how 发
+  briefly stopped being 简体-only.
+
+  `toScript` drops the pass that was papering over this. Key counts move:
+  723,149 for `full`, 97,997 for `standard`, 16,975 for `core`.
+
 ### Added
 
+- **`convertToHtml` declares what language it is emitting.** Every syllable
+  element now carries `lang="zh-Latn-CN-pinyin"`, or `zh-Latn-TW-pinyin` where
+  the conversion read `zh-TW`. Nothing about `yín` on its own says it is
+  Mandarin in the Latin alphabet, so without the tag a screen reader pronounces
+  it as whatever the surrounding page claims to be — English, usually — and the
+  browser hyphenates it that way too. The subtags are all registered, and the
+  `pinyin` variant's prefix in the IANA registry is `zh-Latn`.
+
+  **This changes the markup existing callers get**, which is why it is worth
+  saying plainly: the classes are unchanged, and `lang` is a new attribute
+  between `class` and `data-alternatives`. `lang: false` — `--no-lang` at the
+  command line — leaves it off, for a caller who would rather declare the
+  language once on a wrapper of their own than repeat it on every syllable.
+
+- **`pinyinjs script`,** script conversion at the command line, with `--to` for
+  the target and `--from-script` where the text is too short for detection to
+  tell. The lines carry the converted text and nothing else so it pipes;
+  `--json` carries the per-character evidence and what was uncertain. Also a
+  [script conversion](docs/script-conversion/) page, which the README and the
+  docs index now point at — `toScript` shipped with one row in the API table
+  and nothing a reader would find.
+
+- **`toScript`,** 简体 ↔ 繁體 conversion. 我们后来发现了头发问题 is
+  我們後來發現了頭髮問題. The reading is the evidence, in both directions, and it
+  is what nothing else in this space has: 发 is 發 or 髮 and only `fà` tells them
+  apart; 干 is 幹, 乾 or 干; 乾 stays 乾 in 乾隆 and simplifies to 干 in 乾燥.
+  Every other converter resolves these from a phrase table alone. This one
+  segments with the lattice and reads the result, so it generalises to words no
+  phrase table lists.
+
+  Evidence is taken strongest first — a word some source wrote in both scripts,
+  then the character at the reading it was decoded with, then the character's
+  commonest form — and `toScriptPieces` reports which of those settled every
+  character. `isUncertainChoice` marks the last one, the only case that is a
+  guess: 下面 is a surface or a bowl of noodles and nothing here can tell, so it
+  says so rather than picking silently. Over the gold corpus 97.7% of characters
+  are locked and 1.5% are guesses.
+
+  Targets are BCP-47: `zh-Hans`, `zh-Hant`, `zh-Hant-TW`, `zh-Hant-HK`, with a
+  bare `zh-Hant` taking Taiwan. The input script is detected unless `from` names
+  it, which matters more than it sounds — plenty of characters are current in
+  both scripts, so running 繁體 through the 简→繁 tables would turn 准將 into
+  準將.
+
+  **This is orthography, not translation.** 軟體 and 软件 are different words
+  rather than two spellings of one. OpenCC folds that into script conversion
+  with `s2twp`; pinyinjs does not.
+
+- **`data/script.map`,** the conversion tables, as their own fetchable file:
+  282 KB, 96 KB brotli, loaded by `loadScriptTables` only when something asks
+  to convert. A caller converting hanzi to pinyin pays nothing for it. The
+  character tables alone are right for 99.13% of the 267,639 entries whose two
+  scripts differ, and the word exception lists cover the rest.
+
+- **`hansOnly` and `hantOnly`,** the character sets `detectScript` has always
+  taken and nothing shipped. They ride in the same file.
+
+- **Round-trip scoring in `pnpm accuracy`.** 繁→简 is near-deterministic, so
+  简→繁→简 has to be the identity for essentially every key — half a million
+  cases with no hand-labelling, and every failure a real defect. It reports
+  99.6% over the words in use and 99.3% over every key. The reverse trip is
+  lossy by design and is reported rather than targeted.
+
+- **Hong Kong 繁體 glyph forms are recognised on the lookup path.** Taiwan and
+  Hong Kong write the same characters with the same readings and disagree about
+  the standard form of 58 of them — 裡 against 裏, 群 against 羣, 麵 against 麪.
+  Hong Kong spellings were missing dictionary keys and falling back to
+  converting character by character, so 羣眾 and 麪包 read worse than 群眾 and
+  麵包 did. The lookup path now normalises 繁體 glyph forms to the Taiwan
+  standard the keys are already built from: measured over the 9,833 keys with a
+  distinct Hong Kong spelling, 2,358 were already keys and normalisation reaches
+  a further 5,439. Only the reading is taken from the normalised form — the
+  characters a caller gets back are the ones they wrote. `toCanonicalGlyphs`,
+  `toRegionalGlyphs` and the `Region` type are exported for it, and
+  SCRIPTS-AND-LOCALES.md carries the reasoning.
+
+  Nineteen of OpenCC's mappings are deliberately not applied, because the
+  variant is a live spelling rather than a regional one. Six are current 繁體 —
+  台 appears 164 times in CC-CEDICT against 臺's 208 — and thirteen are current
+  **简体**, because the PRC simplification adopted the same 新字形 conventions
+  Hong Kong did: 着, 温, 脱, 户, 税, 卧, 悦, 兑, 葱, 幺, 棱, 檐, 痹. Normalising
+  those would have rewritten 简体 text, and 走着 `zǒuzhe` is the case that
+  caught it.
+
+- **OpenCC as a data source.** Its 简体 ↔ 繁體 character tables and the Taiwan
+  and Hong Kong variant tables, Apache-2.0, attributed in `NOTICE`. The glyph
+  tables are held in `src/script/glyphs.ts` as code rather than shipped as an
+  artifact, since 120 mappings do not justify a download and they have to work
+  before any dictionary has loaded; `pnpm build:data` re-derives them from
+  OpenCC and fails the build if the two have drifted.
+
 - **`slug`,** hanzi to a URL-safe slug: 我想学中文。 is
   `wo3-xiang3-xue2-zhong1wen2`. Built on the decode rather than on a finished
   string, which is the whole point of it — 银行 is `yin2hang2` and not
@@ -236,14 +352,14 @@ decoder internals and the build pipeline.
 
 ### Accuracy at 1.0.0
 
-Over the 118-case gold corpus, which is committed to the repository rather than
+Over the 124-case gold corpus, which is committed to the repository rather than
 to the published package, and scored by `pnpm accuracy`:
 
 |                  |   lattice | greedy baseline |
 | ---------------- | --------: | --------------: |
-| exact match      | **97.5%** |           88.1% |
+| exact match      | **97.6%** |           88.7% |
 | reading accuracy |     99.7% |           98.3% |
-| spacing (F1)     | **99.8%** |           96.3% |
+| spacing (F1)     | **99.8%** |           96.4% |
 
 Every figure in that table is asserted against the scorer by
 `src/changelog.test.ts`, because a number nothing executes goes stale, which is

package/NOTICE

@@ -32,6 +32,15 @@ Source:   https://raw.githubusercontent.com/fxsjy/jieba/master/jieba/dict.txt
 Licence:  MIT, though the dictionary's own provenance is not documented upstream; recorded here as an accepted unknown
 Provides: word frequencies and the part-of-speech tags that mark proper nouns
 
+## OpenCC
+
+Source:   https://raw.githubusercontent.com/BYVoid/OpenCC/master/data/dictionary/STCharacters.txt
+          https://raw.githubusercontent.com/BYVoid/OpenCC/master/data/dictionary/TSCharacters.txt
+          https://raw.githubusercontent.com/BYVoid/OpenCC/master/data/dictionary/TWVariants.txt
+          https://raw.githubusercontent.com/BYVoid/OpenCC/master/data/dictionary/HKVariants.txt
+Licence:  Apache-2.0
+Provides: the 简体 ↔ 繁體 character tables, and the Taiwan and Hong Kong glyph form tables that separate 裡 from 裏 and 群 from 羣
+
 ## Comparison of Standard Chinese transcription systems (Wikipedia)
 
 Source:   https://en.wikipedia.org/wiki/Comparison_of_Standard_Chinese_transcription_systems

package/README.md

@@ -32,6 +32,9 @@ yin2hang2
 $ pinyinjs slug 我想学中文。
 wo3-xiang3-xue2-zhong1wen2
 
+$ pinyinjs script 我们后来发现了头发问题 --to zh-Hant
+我們後來發現了頭髮問題
+
 $ pinyinjs explain 银行
 银行  yínháng
   yín     locked
@@ -57,6 +60,7 @@ běijīng     běijīng   ㄅㄟˇ ㄐㄧㄥ     pei³-ching¹ běijīng   beeij
 | `convert`    | hanzi to pinyin                                     |
 | `html`       | the same, as HTML                                   |
 | `slug`       | hanzi to a URL-safe slug                            |
+| `script`     | 简体 ↔ 繁體 conversion                              |
 | `explain`    | each syllable, how settled it was, and what it beat |
 | `lookup`     | what the dictionary holds for a word                |
 | `syllable`   | take written pinyin apart                           |
@@ -181,7 +185,6 @@ convert(dictionary, text, { notation: "numbers", capitals: "none" });
 | `punctuation` | `"latin"`                          | `"latin"`, `"keep"`                               |
 | `grouping`    | `true`                             | `false` turns off GB/T 16159 word spacing         |
 | `numbers`     | `"read"`                           | `"keep"` leaves every digit as it was written     |
-| `numbers`     | `"read"`                           | `"keep"` leaves every digit as it was written     |
 | `sandhi`      | `{ yiBu: true, thirdTone: false }` | `{ yiBu?: boolean; thirdTone?: boolean }`         |
 
 ```ts
@@ -323,13 +326,15 @@ evidence there was, not as a probability.
 import { convertToHtml } from "@kensio/pinyinjs";
 
 convertToHtml(dictionary, "行");
-// <span class="py-syllable py-tone-2 py-uncertain"
+// <span class="py-syllable py-tone-2 py-uncertain" lang="zh-Latn-CN-pinyin"
 //       data-alternatives="háng héng hàng">xíng</span>
 ```
 
 One element per syllable, with `py-tone-1` to `py-tone-5` (5 is the neutral
 tone), and `py-uncertain` plus the rejected readings where the decoder was
-guessing. Text that is not Han is escaped, not marked up. No styles are
+guessing. Each one declares itself pinyin — `zh-Latn-TW-pinyin` for a `zh-TW`
+conversion — so that a screen reader does not read it as the language of the
+page around it. Text that is not Han is escaped, not marked up. No styles are
 included, so write your own:
 
 ```css
@@ -341,9 +346,8 @@ included, so write your own:
 }
 ```
 
-Takes any `convert` option, plus `toneClasses: false` and
-`markUncertain: false`. `toHtml(pieces, options)` renders pieces you already
-have.
+Takes any `convert` option, plus `toneClasses: false`, `markUncertain: false`
+and `lang: false`. `toHtml(pieces, options)` renders pieces you already have.
 
 ## Look words up
 
@@ -531,16 +535,57 @@ Third-tone sandhi is off by default: standard orthography writes 你好 as
 
 ## Scripts and locales
 
-Script and locale are separate options: Taiwan writes 繁體 with `zh-TW`
-readings, but mainland editions of classical texts use 繁體 with `zh-CN`
-readings, and Singapore uses 简体.
+Three axes, not one. Taiwan writes 繁體 with `zh-TW` readings, but mainland
+editions of classical texts use 繁體 with `zh-CN` readings, and Singapore uses
+简体.
+
+| Axis        | Values            | What differs                    |
+| ----------- | ----------------- | ------------------------------- |
+| Script      | `Hans` / `Hant`   | which characters are written    |
+| Glyph forms | `TW` / `HK`       | which 繁體 standard writes them |
+| Locale      | `zh-CN` / `zh-TW` | how they are read               |
+
+Both scripts are dictionary keys, so only the locale is an option to pass. Hong
+Kong's 繁體 forms are recognised on the way in, so 羣眾 and 麪包 read exactly as
+群眾 and 麵包 do.
+
+## Simplified and traditional
+
+`toScript` converts between the scripts, and the reading is what makes it
+accurate. Simplification merged distinct characters; the reading un-merges them.
+
+```ts
+import { loadScriptTables, toScript } from "@kensio/pinyinjs";
+
+const tables = await loadScriptTables(source);
+
+toScript(dictionary, tables, "我们后来发现了头发问题", { to: "zh-Hant" });
+// "我們後來發現了頭髮問題"
+
+toScript(dictionary, tables, "头发", { to: "zh-Hant" }); // "頭髮", tóufà
+toScript(dictionary, tables, "出发", { to: "zh-Hant" }); // "出發", chūfā
+```
+
+发 is 發 or 髮 and nothing about the character says which. Every other converter
+works from phrase tables alone, so it is right about the words on the list and
+guessing past it.
+
+It runs both ways — 乾燥 is `gānzào` and simplifies to 干燥, while 乾隆 is
+`Qiánlóng` and stays 乾隆 — and it targets a region, because there is no
+region-free 繁體:
+
+```ts
+toScript(dictionary, tables, "面包", { to: "zh-Hant-TW" }); // "麵包"
+toScript(dictionary, tables, "面包", { to: "zh-Hant-HK" }); // "麪包"
+```
 
-| Axis   | Values            | What differs                 |
-| ------ | ----------------- | ---------------------------- |
-| Script | `Hans` / `Hant`   | which characters are written |
-| Locale | `zh-CN` / `zh-TW` | how they are read            |
+`toScriptPieces` reports what settled each character, and which were guesses:
+下面 is a surface or a bowl of noodles, both `xiàmiàn`, and it says so rather
+than picking silently. The tables load separately from the dictionary, so
+converting hanzi to pinyin costs nothing for them.
 
-Both scripts are dictionary keys, so only the locale is an option to pass.
+This is orthography and not translation — 软件 becomes 軟件, never 軟體. See
+[script conversion](docs/script-conversion/).
 
 ## API
 

package/data/core.entries

@@ -714,7 +714,6 @@ liao2		n
 qi1		zg
 cheng1		zg
 chan2		zg
-wei3	wei4	j
 ji1
 bo1
 hui4		zg
@@ -1771,7 +1770,6 @@ nie4		zg
 xuan1		v
 liang4		zg
 sang4				sang1
-chi1		v
 qiao2
 yan4
 dan1				chan2,shan4
@@ -5081,7 +5079,6 @@ mi3		g
 chu4		zg
 ge2		zg
 di2		ng
-xu4		ns	p
 jiao4		zg
 min3		ag
 jiu4		v
@@ -7133,6 +7130,7 @@ cui3		zg
 ti2		zg
 yang4		zg
 jiang1
+cong2
 ying3
 xiu3		zg
 shu4		zg
@@ -10024,7 +10022,6 @@ quan3
 qi2		nr
 zhun3
 qian4
-xian4		n
 shou4
 wei2		zg
 qing4		zg		qi3,qing3
@@ -10179,7 +10176,6 @@ hui4
 xi4
 se4		zg
 jian3
-jiang1		n
 huan2		zg
 zao3
 xie4		zg
@@ -10756,7 +10752,6 @@ lie4
 shan1
 ting3
 mei2
-chun2
 shen4
 qian3		zg
 de5				te5
@@ -12201,7 +12196,6 @@ jie2
 na4		g
 yi4
 zhong1		d
-zhi3		d		zhi1
 xie2
 ren4		zg
 qin1		zg
@@ -12589,7 +12583,6 @@ gao4
 song4
 hui4
 shuo1		zg
-shuo1
 chang4		zg
 shui2		zg
 ke4
@@ -14155,7 +14148,6 @@ na4
 e2		zg
 jue2
 dun4
-gou1
 qian2
 ban3
 sa4		zg
@@ -14206,7 +14198,6 @@ fu2
 yue4
 pi1
 yang1		zg
-bo1		ng
 gou1		zg
 shu4
 zheng1
@@ -14403,7 +14394,6 @@ jian4
 jian4
 si1		ng
 zhe3		n
-zhen1		p
 zhong1		zg
 zi1		zg
 xiu1		zg
@@ -16107,7 +16097,6 @@ ba4		zg
 you2		zg
 gun3		zg
 pi1
-nian2
 tai2
 bao4
 fu4
@@ -16722,7 +16711,6 @@ mai4		zg
 mai4		nr	p
 chao3
 fu1		zg
-mian4		n		mian5
 mian4
 qu4
 qu1	qu2	q		qu3

package/data/core.keys

@@ -714,7 +714,6 @@ B
 僛
 僜
 僝
-僞
 僟
 僠
 僡
@@ -1771,7 +1770,6 @@ B
 喧
 喨
 喪
-喫
 喬
 喭
 單
@@ -5081,7 +5079,6 @@ B
 敊
 敋
 敌
-敍
 敎
 敏
 救
@@ -7133,6 +7130,7 @@ B
 漽
 漾
 漿
+潀
 潁
 潃
 潄
@@ -10024,7 +10022,6 @@ B
 綦
 綧
 綪
-綫
 綬
 維
 綮
@@ -10179,7 +10176,6 @@ B
 繫
 繬
 繭
-繮
 繯
 繰
 繲
@@ -10756,7 +10752,6 @@ B
 脠
 脡
 脢
-脣
 脤
 脥
 脦
@@ -12201,7 +12196,6 @@ B
 衲
 衵
 衷
-衹
 衺
 衽
 衾
@@ -12589,7 +12583,6 @@ B
 誦
 誨
 說
-説
 誯
 誰
 課
@@ -14155,7 +14148,6 @@ B
 鈋
 鈌
 鈍
-鈎
 鈐
 鈑
 鈒
@@ -14206,7 +14198,6 @@ B
 鉞
 鉟
 鉠
-鉢
 鉤
 鉥
 鉦
@@ -14403,7 +14394,6 @@ B
 鍵
 鍶
 鍺
-鍼
 鍾
 鍿
 鎀
@@ -16107,7 +16097,6 @@ B
 鮋
 鮌
 鮍
-鮎
 鮐
 鮑
 鮒
@@ -16722,7 +16711,6 @@ B
 麦
 麨
 麩
-麪
 麫
 麮
 麯