@kensio/pinyinjs 0.1.0-beta.0 → 1.0.0

package/CHANGELOG.md

@@ -0,0 +1,87 @@
+# Changelog
+
+Notable changes to `@kensio/pinyinjs`. Versions follow
+[semantic versioning](https://semver.org) from 1.0.0 onward.
+
+Two things are outside the version contract, for the reasons
+[the API page](docs/api/#stability) gives: the readings a conversion produces,
+which a dictionary rebuild can change, and the artifact format under `data/`.
+
+## 1.0.0
+
+The first stable release. Everything the
+[API page](docs/api/) lists is now covered by semantic versioning, including the
+decoder internals and the build pipeline.
+
+### Added since 0.1.0-beta.0
+
+- **Transcription systems, all five, in both directions.** Bopomofo, Wade-Giles,
+  Yale, Gwoyeu Romatzyh and IPA at the syllable level, plus hanzi → any of them
+  end to end via `convert --system` and `convertToWadeGiles`. Checked against
+  417 rows of an outside syllabary in every system — 3,336 cells.
+- **Reading ambiguous Wade-Giles.** `readWadeGilesLoosely` allows for the
+  apostrophes and diacritics real texts drop, and `splitWadeGiles` puts back the
+  hyphens they lose. Measured: 52.07% of written syllables lose their identity
+  without their marks, and taking the first candidate recovers 79.05%.
+- **Numbers read aloud** (`src/numerals/`, the `number` command): counting and
+  spelling out, 两 against 二, decimals, percentages, fractions, `yāo`, and
+  times. `convert` reads the digits it meets and takes the style from what
+  follows them.
+- **Orthography rules.** The 重叠 hyphen for AABB and repeated words, the 成语
+  hyphen from a curated list of 117, 老王 → `Lǎo Wáng`, and GB/T 16159 5.1 in
+  full — 毛泽东 is `Máo Zédōng`, 北京大学 is `Běijīng Dàxué`, on boundaries
+  CC-CEDICT's own capitalisation states.
+- **Typed rule overrides on the lattice** (`READING_RULES`), with the modal 得
+  and attested 儿化 as the first two.
+- **Tone colour in the terminal**, in MDBG's palette, on every command that
+  writes a syllable.
+
+### Fixed since 0.1.0-beta.0
+
+- **一 sandhi is about the 一 that counts.** 十一月 is `shíyīyuè` and 第一次 is
+  `dìyīcì`; the citation tone survives a final digit and an ordinal. 561
+  conversions change over 88,866 lines, 520 of them right.
+- **A colon between digits is a time**, not an identifier: 6:30 is
+  `liù diǎn sānshí fēn`.
+- **The documented proper-noun mechanism was wrong.** The capitals page
+  described a 500-entry surname list that has never existed; what produces
+  `Lǐ Huá` is two single-character dictionary entries each flagged a proper
+  noun.
+
+### Changed
+
+- **`engines` is now `>=22.0.0`**, down from `>=24.0.0`. The suite and the
+  packaged tarball are both exercised on Node 22 and 24 in CI.
+- **No more `beta` dist-tag.** `pnpm add @kensio/pinyinjs` finds it.
+- `src/romanization/` is `src/transcription/`, since bopomofo has a script of
+  its own and IPA is a transcription rather than a spelling. The published docs
+  path keeps the older name.
+- The build cleans `dist/` before compiling, and `pnpm pack:check` asserts the
+  tarball's contents. The beta tarball shipped 24 stale files from a directory
+  that had been renamed away.
+
+### Accuracy at 1.0.0
+
+Over the 105-case gold corpus, which is committed to the repository — not to
+the published package — and scored by `pnpm accuracy`:
+
+|                  |   lattice | greedy baseline |
+| ---------------- | --------: | --------------: |
+| exact match      | **97.1%** |           92.4% |
+| reading accuracy |     99.7% |           98.6% |
+| spacing (F1)     | **99.7%** |           97.0% |
+
+Every figure in that table is asserted against the scorer by
+`src/changelog.test.ts`, because a number nothing executes goes stale — which is
+what happened to this project's README table across three releases.
+
+`pnpm polyphones` reports 89.04% for the lattice against greedy's 88.82% over
+CPP's 20,139 hand-labelled polyphones. That corpus is fetched rather than
+committed, so unlike the table above it cannot be asserted here, and it is
+reported rather than guarded.
+
+## 0.1.0-beta.0
+
+First published release: the lattice decoder, the tiered dictionaries, the
+syllable layer, sandhi, GB/T 16159 spacing and capitals, HTML output and the
+`pinyinjs` command.

package/NOTICE

@@ -31,3 +31,10 @@ Provides: the bulk of the multi-character word readings
 Source:   https://raw.githubusercontent.com/fxsjy/jieba/master/jieba/dict.txt
 Licence:  MIT, though the dictionary's own provenance is not documented upstream; recorded here as an accepted unknown
 Provides: word frequencies and the part-of-speech tags that mark proper nouns
+
+## Comparison of Standard Chinese transcription systems (Wikipedia)
+
+Source:   https://en.wikipedia.org/wiki/Comparison_of_Standard_Chinese_transcription_systems
+Licence:  CC BY-SA 4.0
+Provides: the 417-row syllabary in test/fixtures/syllabary.ts, which the bopomofo, Wade-Giles, Yale, IPA and Gwoyeu Romatzyh tables are checked against, GR having four columns of its own
+Note:     used by the tests only; nothing in data/ is derived from it

package/README.md

@@ -5,18 +5,13 @@ validates and writes pinyin syllables on their own, with no dictionary.
 
 [https://pinyinjs.dev](https://pinyinjs.dev "PinyinJS docs website")
 
-> **Beta.** Published under the `beta` tag, so a plain `pnpm add
-@kensio/pinyinjs` will not find it until 1.0. Wade-Giles, Bopomofo, Yale and
-> IPA, and reading numbers aloud, are not built. Everything below works and is
-> tested, but the API will change before 1.0.
-
 ## Install
 
 ```bash
-pnpm add @kensio/pinyinjs@beta
+pnpm add @kensio/pinyinjs
 ```
 
-Node 24+, or any browser. The core imports no Node built-ins, and the package is
+Node 22+, or any browser. The core imports no Node built-ins, and the package is
 ESM only.
 
 The dictionaries ship inside the package, which is what makes it a 4 MB
@@ -46,17 +41,25 @@ $ pinyinjs syllable nǐhǎo
 nǐhǎo  nǐ hǎo
   nǐ        n + i, tone 3         nǐ  ni3  ni³
   hǎo       h + ao, tone 3        hǎo  hao3  hao³
+
+$ pinyinjs convert --system wade-giles --notation none 我要去北京。
+Wo yao ch'ü Pei-ching.
+
+$ pinyinjs transcribe běijīng
+běijīng     běijīng   ㄅㄟˇ ㄐㄧㄥ     pei³-ching¹ běijīng   beeijing  pei˨˩˦tɕiŋ˥
 ```
 
-| Command    | Does                                                |
-| ---------- | --------------------------------------------------- |
-| `convert`  | hanzi to pinyin                                     |
-| `html`     | the same, as HTML                                   |
-| `explain`  | each syllable, how settled it was, and what it beat |
-| `lookup`   | what the dictionary holds for a word                |
-| `syllable` | take written pinyin apart                           |
-| `sandhi`   | apply tone sandhi to written pinyin                 |
-| `info`     | which dictionary is loaded, and how big it is       |
+| Command      | Does                                                |
+| ------------ | --------------------------------------------------- |
+| `convert`    | hanzi to pinyin                                     |
+| `html`       | the same, as HTML                                   |
+| `explain`    | each syllable, how settled it was, and what it beat |
+| `lookup`     | what the dictionary holds for a word                |
+| `syllable`   | take written pinyin apart                           |
+| `sandhi`     | apply tone sandhi to written pinyin                 |
+| `number`     | read a number as Chinese numerals                   |
+| `transcribe` | pinyin to bopomofo, Wade-Giles, Yale, GR and IPA    |
+| `info`       | which dictionary is loaded, and how big it is       |
 
 Every conversion option below is a flag: `--notation`, `--locale`,
 `--apostrophe`, `--capitals`, `--punctuation`, `--no-grouping`, `--third-tone`,
@@ -66,6 +69,12 @@ A command given no arguments reads standard input, one text per line, so
 `cat article.txt | pinyinjs convert` works. `syllable` and `sandhi` need no
 dictionary at all and start without loading one.
 
+At a terminal each syllable is written in its tone's colour, in
+[MDBG](https://www.mdbg.net)'s palette — red, yellow, green, blue, and the
+terminal's own colour for the neutral tone. It is off for a pipe, `NO_COLOR` is
+honoured, `--colour` and `--no-colour` force it either way, and `--json` is
+never coloured. See [the command line](docs/cli/#colour).
+
 ### Calling it from something else
 
 `convert` writes the pinyin and nothing else, so it drops straight into a
@@ -140,9 +149,20 @@ first, convert with it, and reload as `full` arrives.
 convert(dictionary, "银行"); // "yínháng"
 convert(dictionary, "行长"); // "hángzhǎng"
 convert(dictionary, "我要去北京。"); // "Wǒ yào qù Běijīng."
-convert(dictionary, "3D银行"); // "3Dyínháng" — non-Han text is left as written
+convert(dictionary, "3D银行"); // "sān D yínháng" — the digit is read, the letter is not
 ```
 
+A reading the dictionary cannot settle on its own is settled by context, with
+typed rules over the lattice rather than tweaks to the output:
+
+```ts
+convert(dictionary, "我得走了"); // "wǒ děi zǒule" — modal 得
+convert(dictionary, "他跑得很快"); // "tā pǎo de hěn kuài" — the particle
+convert(dictionary, "那边儿"); // "nà biānr" — 儿 does not stand on its own
+```
+
+See [converting](docs/converting/#rules-where-the-cost-model-cannot-reach).
+
 ### Options
 
 ```ts
@@ -157,6 +177,8 @@ convert(dictionary, text, { notation: "numbers", capitals: "none" });
 | `capitals`    | `"auto"`                           | `"auto"`, `"proper"`, `"none"`                    |
 | `punctuation` | `"latin"`                          | `"latin"`, `"keep"`                               |
 | `grouping`    | `true`                             | `false` turns off GB/T 16159 word spacing         |
+| `numbers`     | `"read"`                           | `"keep"` leaves every digit as it was written     |
+| `numbers`     | `"read"`                           | `"keep"` leaves every digit as it was written     |
 | `sandhi`      | `{ yiBu: true, thirdTone: false }` | `{ yiBu?: boolean; thirdTone?: boolean }`         |
 
 ```ts
@@ -218,8 +240,36 @@ The list is not a complete 正词法 implementation, so some words it does not
 cover are written differently: 不但 is `búdàn`, 大米 is `dàmǐ`, 青海 is
 `Qīnghǎi`.
 
-**Non-Han text** — digits, Latin letters, punctuation — passes through exactly
-as written. Reading numbers aloud is a separate package, not yet built.
+**Reduplication** takes a hyphen rather than a space, since it is one word with
+a boundary inside it.
+
+```ts
+convert(dictionary, "干干净净"); // "gāngān-jìngjìng"
+convert(dictionary, "研究研究"); // "yánjiū-yánjiū"
+convert(dictionary, "爸爸妈妈"); // "bàba māma" — that shape, but two words
+```
+
+**A 成语 that can be read as two disyllables** takes the same hyphen, from a
+curated list of 117; the rest are written solid, as the standard writes them.
+
+```ts
+convert(dictionary, "风平浪静"); // "fēngpíng-làngjìng"
+convert(dictionary, "不亦乐乎"); // "búyìlèhū" — cannot be halved
+```
+
+**Digits are read**, and the rest of a non-Han run passes through exactly as
+written:
+
+```ts
+convert(dictionary, "我有3个苹果。"); // "Wǒ yǒu sān gè píngguǒ."
+convert(dictionary, "1988年之后"); // "yī jiǔ bā bā nián zhīhòu"
+convert(dictionary, "95%的人"); // "bǎifēnzhījiǔshíwǔ de rén"
+convert(dictionary, "3D打印"); // "sān D dǎyìn"
+convert(dictionary, "6:30起床"); // "liù diǎn sānshí fēn qǐchuáng"
+convert(dictionary, "16:9的"); // "16:9de" — a ratio is not a quantity
+```
+
+`numbers: "keep"` leaves every digit alone. See [numbers](docs/numerals/).
 
 ## Syllable by syllable
 
@@ -384,6 +434,77 @@ toneFromMarks("hǎo"); // 3
 `de` in 我的 is neutral (5), whereas the `bei` in a typed `beijing` has no tone
 written at all.
 
+## Numbers
+
+Reading a number needs no dictionary — arithmetic and about twenty readings —
+so this works with nothing loaded.
+
+```ts
+import { numeralHanzi, percentHanzi, readNumeral } from "@kensio/pinyinjs";
+
+numeralHanzi(12345); // "一万两千三百四十五"
+numeralHanzi(1005); // "一千零五" — a skipped place is spoken
+numeralHanzi(2000); // "两千" — a leading lone 2 before a big unit
+percentHanzi(95); // "百分之九十五" — the order reverses
+```
+
+The same digits are read two ways and nothing in the number says which — 2026年
+is spelled out and 2026个 is counted — so the style is the caller's:
+
+```ts
+numeralHanzi(2026); // "两千零二十六"
+numeralHanzi(2026, { style: "digits" }); // "二〇二六"
+readNumeral(110, { style: "digits", yao: true }); // yāo yāo líng
+```
+
+More in [numbers](docs/numerals/).
+
+## Bopomofo, Wade-Giles, Yale, Gwoyeu Romatzyh and IPA
+
+Also dictionary-free: a romanisation is a mapping over about 420 syllables, so
+hanzi → Wade-Giles is hanzi → pinyin → Wade-Giles.
+
+```ts
+import {
+  readSyllable,
+  readWadeGilesLoosely,
+  writeBopomofo,
+  writeGwoyeu,
+  writeIpa,
+  writeWadeGiles,
+  writeYale,
+} from "@kensio/pinyinjs";
+
+const jiu = readSyllable("jiù");
+writeBopomofo(jiu); // "ㄐㄧㄡˋ"
+writeWadeGiles(jiu); // "chiu⁴"
+writeYale(jiu); // "jyòu"
+writeGwoyeu(jiu); // "jiow"
+writeIpa(jiu); // "tɕiou˥˩"
+```
+
+Gwoyeu Romatzyh is the odd one, and it needs no tone mark because the tone is
+spelled into the syllable — which is why 陝西 is Shaanxi in English and 山西 is
+Shanxi:
+
+```ts
+const gr = (pinyin: string) => writeGwoyeu(readSyllable(pinyin));
+[gr("shān"), gr("shán"), gr("shǎn"), gr("shàn")]; // shan, sharn, shaan, shann
+```
+
+Reading Wade-Giles back gives an **array**, because real text drops the
+apostrophes and diacritics that carry the distinctions:
+
+```ts
+readWadeGilesLoosely("chi¹"); // [jī, qī] — chi is jī, ch'i is qī
+readWadeGilesLoosely("chu¹"); // [zhū, chū, jū, qū]
+```
+
+Measured over the phrase corpus, 52.07% of written syllables have a Wade-Giles
+spelling that merges with another once its marks are dropped, and taking the
+first candidate recovers 79.05% of them. More in
+[romanisation](docs/romanization/).
+
 ## Sandhi
 
 The dictionary stores underlying tones, and sandhi is applied to the syllable
@@ -431,6 +552,10 @@ Both scripts are dictionary keys, so only the locale is an option to pass.
 | `readSyllable` / `writeSyllable` / `isSyllable`      | one syllable, no dictionary                       |
 | `splitSyllables` / `readWord`                        | split written pinyin                              |
 | `applySandhi`                                        | 一, 不 and optional third-tone sandhi             |
+| `writeBopomofo` / `writeWadeGiles`                   | one syllable, romanised                           |
+| `writeYale` / `writeGwoyeu` / `writeIpa`             | the same, in Yale, GR and IPA                     |
+| `readBopomofo` / `readWadeGilesLoosely`              | and back again                                    |
+| `readYale` / `readGwoyeu` / `readIpa`                | and back from those three                         |
 | `applyToneMark` / `stripToneMarks` / `toneFromMarks` | tone marks                                        |
 | `convertGreedily(...)`                               | the old longest-match decoder, kept as a baseline |