@kensio/pinyinjs 1.5.3 → 1.6.0

package/CHANGELOG.md

@@ -9,6 +9,155 @@ which a dictionary rebuild can change, and the artifact format under `data/`.
 
 ## Unreleased
 
+## 1.6.0
+
+### Added
+
+- **Hanzi and pinyin together, rather than one instead of the other.**
+  `convertToAnnotatedHtml` writes the characters with their reading above them,
+  as `<ruby>` markup — the output every other mode could not give, and what a
+  learner's text, a subtitle and a dictionary entry all want. `convert` and
+  `convertToHtml` write the reading _instead of_ the characters.
+  `toAnnotatedHtml(pieces)` renders pieces you already have, and the command
+  line has `pinyinjs annotate`.
+
+  Each `<rt>` holds exactly what `convertToHtml` would have written, so tone
+  classes and uncertainty marking work inside an annotation as they do outside
+  one. `<rp>` gives the parenthesised fallback, the `<ruby>` carries `lang="zh"`
+  and the syllables inside it keep `zh-Latn-CN-pinyin`, since on an English page
+  a screen reader needs to be told both.
+
+  **A base is not always one character**, which is what makes this harder than
+  it looks and what every naive per-character annotation gets wrong. 玩儿 is two
+  characters and the one syllable `wánr`; 95% is `bǎifēnzhījiǔshíwǔ` over three
+  written characters in the other order; 1988 is `yī jiǔ bā bā` over one. All
+  are annotated whole. `ConvertedPiece` gains a `source` — the characters a
+  piece reads, or undefined where it reads on into the ones named before it —
+  which is what makes that expressible.
+
+  Pinyin orthography stays out of the hanzi it annotates: no word spaces, since
+  Chinese is not written with them; the hyphen of `gāngān-jìngjìng` sits in the
+  reading, being a boundary inside one word rather than a gap between two; and
+  the base keeps 。 where the conversion writes a full stop.
+
+- **`readings`: a conversion takes readings the caller asserts.** No rule
+  settles every polyphone, and some texts are genuinely ambiguous —
+  孩子越长越漂亮 grows where 头发越长越漂亮 lengthens, and nothing in the
+  characters says which. An application that knows its own content can now say
+  so, rather than the library guessing on its behalf.
+
+  ```ts
+  convert(dictionary, "这篇文章不太长。", { readings: { 太长: "tài cháng" } });
+  ```
+
+  A **word** hint asserts something about the text it names and rewrites exactly
+  those characters, so a bare `长` leaves 校长 as `xiàozhǎng` — the dictionary
+  knowing that word is better evidence about the stretch than a remark about one
+  of its characters, and that is what makes a corrections table safe to
+  accumulate. Naming the whole word does reach it, and keeps it whole:
+  `{ 银行: "yín xíng" }` gives `yínxíng`.
+
+  A **positional** hint asserts something about one character of one text, and
+  nothing outranks it, the enclosing word included. Positions count code points
+  from the start of the text, across any non-Han runs in it. The list form takes
+  both kinds together.
+
+  Hints enter as lattice edges rather than as rules, because the rules layer
+  cannot invent a reading no source attests — which is exactly what a hint is,
+  the difference being that a caller attests it. Spacing is untouched, an
+  unmarked syllable is 轻声, and a hint that cannot be parsed throws rather than
+  being skipped: a correction that silently does nothing is worse than one that
+  fails.
+
+### Fixed
+
+- **Polyphones defaulted to whichever reading a dictionary happened to print
+  first.** Unihan's `kTGHZ2013` ranked above `kMandarin`, and that field
+  prefixes every reading with its entry number in 《現代漢語規範詞典》 — 勒 is
+  `212.050:lè 212.100:lēi` — so its order is a page order, and taking the first
+  reading meant taking whichever one sorted earlier by pinyin. 殷 read `yān` and
+  楷 `jiē`. The dictionary-indexed fields now rank last: they are a good
+  _source_ of readings and a meaningless _ranking_ of them.
+
+  `kHanyuPinlu` still leads where it lists more than one reading, because there
+  it has counted them against each other, but where it lists one it counted
+  nothing — it is saying "this occurred", not "this leads" — and a lone reading
+  now ranks last too. Only ten characters have one that disagrees with
+  `kMandarin`, and eight of those are simply pre-1985: the corpus behind the
+  field, 《現代漢語頻率詞典》, predates the 普通话异读词审音表, and the 132
+  occurrences carrying 绩 as `jī` are the old standard rather than a case for
+  it.
+
+  354 readings fixed against 94 broken over 25 characters, the largest being 贾
+  `gǔ`→`jiǎ`, 苔 `tāi`→`tái`, 町 `dīng`→`tǐng`, 殷 `yān`→`yīn` and 芥
+  `gài`→`jiè`. CPP's polyphone score is 90.34%, up from 89.05%, and the gold
+  corpus does not move. What broke is mostly `kMandarin` being wrong with
+  nothing behind it — 媛 `yuán`→`yuàn`, 桔 `jú`→`jié`, 耶 `yē`→`yé`, 缪
+  `miào`→`móu` — which needs a third opinion with corpus evidence behind it
+  rather than an override, since here the sources disagree. 䘚 is now paired
+  with 卒, so `core` and `standard` each gained a key.
+
+- **`full` read 特徵 as `tèzhǐ` where `standard` read it right.**
+  `large_pinyin.txt` is a 简体 corpus, and where it carries a 繁體 headword
+  anyway it has read it as though the characters were 简体, taking each at face
+  value instead of as the 简体 character it stands for: 徵 as itself rather than
+  as 征, 沈 as the surname rather than as 沉, so 沈溺 came out `shěnnì`. Of the
+  2,854 corpus headwords carrying a 繁體-only character, 72 are ones CC-CEDICT
+  pairs with a 简体 form and 13 of those read wrongly.
+
+  The reading was only half of it. A corpus headword becomes an entry of its
+  own, and an entry outranks the 繁體 key that the traditional derivation hangs
+  on the 简体 entry — so the word was in the dictionary twice, and which copy
+  answered depended on the tier. `standard` excludes the phrase tail and was
+  right; `full`, the default, was wrong. Loading more dictionary made the answer
+  worse, which is also the upgrade path the docs describe, of converting with
+  `standard` and reloading as `full`.
+
+  A headword CC-CEDICT knows _only_ as another word's 繁體 spelling now
+  contributes no entry, and the word keeps the reading CC-CEDICT pairs it with.
+  68 entries drop out and their keys are re-claimed from the 简体 entry, so
+  `full` loses 2 keys; 简→繁→简 round trips gain 38 against 5 lost on 繁→简→繁,
+  both unchanged as percentages. The 2,782 headwords CC-CEDICT does not pair are
+  rare or mixed-script words whose characters do read them, and are left alone.
+  Three of the words are build assertions now, this being the kind of defect
+  that comes back when a rolling source is refreshed and nothing is watching.
+
+- **长 read `zhǎng` wherever no word covered it**, so 这篇文章不太长 came out
+  `bú tài zhǎng` and 要多长时间 `duō zhǎng shíjiān`. The stored default is not
+  the defect — Unihan ranks the readings `zhǎng(1879) cháng(1179)` and names
+  `zhǎng` in `kMandarin`, and that ranking is what reads 署长, 团长, 公安局长 and
+  总会长 correctly off a bare 长 at the end of a title. What was missing is that
+  nothing could prefer the alternate at a position no word covers.
+
+  A degree adverb settles it, and only from the left: a growing 长 is a verb and
+  no 很, 太, 最 or 多 can modify one, while what follows an adjectival 长 is a
+  noun, a particle or the end of the sentence — which is what follows half the
+  verbs too. 得, 着 and the 越…越 correlative are the exceptions, and are guarded;
+  了 and 的 are deliberately not, since after an adverb they are the sentence
+  particle and the attributive, making 时间太长了 and 很长的道路 both `cháng`.
+
+  Measured over the same 88,866 lines of Tatoeba and zh.wikipedia the other
+  edge rules were sized against, 260 长 decode as a word of their own and this
+  moves 75 to `cháng`, all 75 correctly. On CPP's 40 hand-labelled 长 the
+  character goes 85.00% to 87.50%; overall CPP is unmoved at 90.34%, as expected
+  of one character in 20,139.
+
+- **越长越高 read `yuè cháng yuè gāo`.** 越长 is a key read `yuè cháng` and the
+  only one of its shape — 越大, 越高, 越好 and 越快 are all absent, so 越高越好
+  decodes as two words while 越长越高 reached for a word nothing else in the
+  paradigm has. It carries no part of speech, which is how a reading somebody
+  asserted is held rather than a word anybody counted, and only
+  `large_pinyin.txt` has it. Where the far half of the correlative names
+  something growing produces — 高, 大, 胖, 壮, 结实 — the edge is dropped and the
+  character's own `zhǎng` stands.
+
+  Unlike everything else here this is a heuristic with no corpus behind it: 越长
+  occurs three times in the 88,866 lines and all three are 越来越长 or 说的越长.
+  It forbids rather than forces, so `cháng` stays on as a rival one bucket
+  dearer and the syllable still reports itself uncertain — 越长越X is genuinely
+  ambiguous, 孩子越长越漂亮 growing where 头发越长越漂亮 lengthens, which is why
+  漂亮 is not in the list. 时间越长越好 and 头发越长 are untouched.
+
 ## 1.5.3
 
 ### Fixed
@@ -460,14 +609,14 @@ decoder internals and the build pipeline.
 
 ### Accuracy at 1.0.0
 
-Over the 126-case gold corpus, which is committed to the repository rather than
+Over the 130-case gold corpus, which is committed to the repository rather than
 to the published package, and scored by `pnpm accuracy`:
 
 |                  |   lattice | greedy baseline |
 | ---------------- | --------: | --------------: |
-| exact match      | **97.6%** |           88.9% |
-| reading accuracy |     99.7% |           98.4% |
-| spacing (F1)     | **99.8%** |           96.5% |
+| exact match      | **97.7%** |           86.9% |
+| reading accuracy |     99.7% |           97.9% |
+| spacing (F1)     | **99.8%** |           95.7% |
 
 Every figure in that table is asserted against the scorer by
 `src/changelog.test.ts`, because a number nothing executes goes stale, which is

package/README.md

@@ -59,6 +59,7 @@ běijīng     běijīng   ㄅㄟˇ ㄐㄧㄥ     pei³-ching¹ běijīng   beeij
 | ------------ | --------------------------------------------------- |
 | `convert`    | hanzi to pinyin                                     |
 | `html`       | the same, as HTML                                   |
+| `annotate`   | hanzi with its pinyin above, as ruby HTML           |
 | `slug`       | hanzi to a URL-safe slug                            |
 | `script`     | 简体 ↔ 繁體 conversion                              |
 | `explain`    | each syllable, how settled it was, and what it beat |
@@ -143,9 +144,9 @@ Load the dictionary once and keep it; it is immutable and safe to share.
 
 | Tier       | Entries | Download (brotli) | Contains               |
 | ---------- | ------: | ----------------: | ---------------------- |
-| `core`     |  16,730 |             70 KB | single characters only |
-| `standard` |  66,730 |            376 KB | the most common words  |
-| `full`     | 461,623 |          2,381 KB | every word             |
+| `core`     |  16,727 |             70 KB | single characters only |
+| `standard` |  66,727 |            376 KB | the most common words  |
+| `full`     | 461,555 |          2,381 KB | every word             |
 
 `full` is the default. The tiers are nested, so a page can load `standard`
 first, convert with it, and reload as `full` arrives.
@@ -349,6 +350,45 @@ included, so write your own:
 Takes any `convert` option, plus `toneClasses: false`, `markUncertain: false`
 and `lang: false`. `toHtml(pieces, options)` renders pieces you already have.
 
+## Annotated output: hanzi and pinyin together
+
+`convertToHtml` writes the pinyin _instead of_ the hanzi. To keep both — a
+learner's text, a subtitle, a dictionary entry — annotate it instead:
+
+```ts
+import { convertToAnnotatedHtml } from "@kensio/pinyinjs";
+
+convertToAnnotatedHtml(dictionary, "银行");
+// <ruby lang="zh">银<rp>(</rp><rt><span class="py-syllable py-tone-2"
+//       lang="zh-Latn-CN-pinyin">yín</span></rt><rp>)</rp></ruby>…
+```
+
+The markup is `<ruby>`, which browsers lay out natively — the reading sits
+above the characters, and `<rp>` puts it in parentheses where it does not. Each
+`<rt>` holds exactly what `convertToHtml` would have written, so tone colours
+and uncertainty marking work inside an annotation as they do outside one.
+
+**A base is not always one character**, and this is where per-character
+annotation goes wrong. 玩儿 is two characters and the one syllable `wánr`, and a
+read number reverses on the way, so both are annotated whole:
+
+```ts
+convertToAnnotatedHtml(dictionary, "玩儿"); // one <ruby>, 玩儿 over wánr
+convertToAnnotatedHtml(dictionary, "95%"); // one <ruby>, 95% over bǎifēnzhījiǔshíwǔ
+```
+
+Ruby needs no styling to work, but the reading is small by default:
+
+```css
+ruby rt {
+  font-size: 0.5em;
+}
+```
+
+`toAnnotatedHtml(pieces, options)` renders pieces you already have, and
+`ConvertedPiece.source` is what makes that possible: the characters each piece
+reads, or undefined where it reads on into the ones before it.
+
 ## Look words up
 
 ```ts
@@ -598,6 +638,7 @@ This is orthography and not translation — 软件 becomes 軟件, never 軟體.
 | `convert(dictionary, text, options?)`                | hanzi → pinyin                                    |
 | `convertPieces(dictionary, text, ...)`               | the same, per syllable, with confidence           |
 | `convertToHtml(dictionary, text, ...)`               | the same, as HTML                                 |
+| `convertToAnnotatedHtml(dictionary, text, ...)`      | hanzi and pinyin together, as ruby HTML           |
 | `slug(dictionary, text, options?)`                   | hanzi → a URL-safe slug                           |
 | `joinPieces(pieces)` / `toHtml(pieces)`              | render pieces                                     |
 | `isUncertain(confidence)`                            | was this syllable a guess?                        |