@kensio/pinyinjs 1.6.0 → 1.7.1

package/README.md

@@ -35,6 +35,11 @@ wo3-xiang3-xue2-zhong1wen2
 $ pinyinjs script 我们后来发现了头发问题 --to zh-Hant
 我們後來發現了頭髮問題
 
+$ pinyinjs check 银行 yínxíng
+银行  yínháng  50%
+  银     yín     yín     correct
+  行     háng    xíng    wrong
+
 $ pinyinjs explain 银行
 银行  yínháng
   yín     locked
@@ -43,6 +48,11 @@ $ pinyinjs explain 银行
 $ pinyinjs lookup 头发
 头发  tóu fa  n
 
+$ pinyinjs match --query bjdx 北京大学 我在北京大学学中文 上海大学
+[北京大学]  7.00
+我在[北京大学]学中文  6.33
+上海大学  no match
+
 $ pinyinjs syllable nǐhǎo
 nǐhǎo  nǐ hǎo
   nǐ        n + i, tone 3         nǐ  ni3  ni³
@@ -60,6 +70,9 @@ běijīng     běijīng   ㄅㄟˇ ㄐㄧㄥ     pei³-ching¹ běijīng   beeij
 | `convert`    | hanzi to pinyin                                     |
 | `html`       | the same, as HTML                                   |
 | `annotate`   | hanzi with its pinyin above, as ruby HTML           |
+| `segment`    | split text into words                               |
+| `match`      | filter text by a pinyin query, best first           |
+| `check`      | mark typed pinyin against the text                  |
 | `slug`       | hanzi to a URL-safe slug                            |
 | `script`     | 简体 ↔ 繁體 conversion                              |
 | `explain`    | each syllable, how settled it was, and what it beat |
@@ -144,9 +157,9 @@ Load the dictionary once and keep it; it is immutable and safe to share.
 
 | Tier       | Entries | Download (brotli) | Contains               |
 | ---------- | ------: | ----------------: | ---------------------- |
-| `core`     |  16,727 |             70 KB | single characters only |
-| `standard` |  66,727 |            376 KB | the most common words  |
-| `full`     | 461,555 |          2,381 KB | every word             |
+| `core`     |  16,970 |             70 KB | single characters only |
+| `standard` |  66,970 |            377 KB | the most common words  |
+| `full`     | 461,555 |          2,378 KB | every word             |
 
 `full` is the default. The tiers are nested, so a page can load `standard`
 first, convert with it, and reload as `full` arrives.
@@ -389,6 +402,192 @@ ruby rt {
 `ConvertedPiece.source` is what makes that possible: the characters each piece
 reads, or undefined where it reads on into the ones before it.
 
+## Segment
+
+Converting has to find the words before it can read them, because the unit a
+reading belongs to is the word — 行 is `xíng` or `háng` and only 银行 and 行长
+say which. `segment` returns that answer instead of throwing it away.
+
+```ts
+import { segment } from "@kensio/pinyinjs";
+
+segment(dictionary, "南京市长江大桥").map((found) => found.text);
+// ["南京市", "长江", "大桥"], not ["南京", "市长", "江大桥"]
+
+const found = segment(dictionary, "我要去北京。");
+found.map((one) => one.text); // ["我", "要", "去", "北京", "。"]
+found[3]?.partOfSpeech; // "ns", jieba's tag
+found[3]?.isProperNoun; // true
+found[3]?.at; // 3, in code points from the start of the text
+```
+
+Every stretch comes back in order, including the ones that were never Han, so
+the segments rejoin into exactly the text they came from — which is what makes
+it safe to rebuild a document, highlight in place, or index a corpus. Filter on
+`isKnown` for the stretches the dictionary recognised.
+
+What it does not apply is 分词连写, the word spacing written pinyin wants: 他看了
+segments as 他 / 看 / 了 and converts as `tā kànle`, because attaching an aspect
+particle to its verb is a fact about writing pinyin rather than about where the
+words are.
+
+## Match a pinyin query
+
+A search box on a Latin keyboard, filtering Chinese text. `match` returns where
+a query landed, or undefined where it did not land at all.
+
+```ts
+import { match } from "@kensio/pinyinjs";
+
+match(dictionary, "北京大学", "bjdx")?.ranges; // [{ at: 0, length: 4 }]
+match(dictionary, "北京大学", "beijing")?.ranges; // [{ at: 0, length: 2 }]
+match(dictionary, "北京大学", "nanjing"); // undefined
+```
+
+Every way anybody types it: full syllables joined or spaced, `beijing` and
+`bei jing`; initials, `bj`; the two mixed, `beij` and `bjing`; tones as digits
+where they are worth writing, `bei3jing1`; `v` or `u:` for ü; and the r of 儿化
+on the syllable it belongs to, so 玩儿 answers to `wanr`.
+
+**No index is built and none is needed.** The haystack is Chinese, so the query
+is tested as a path over each character's readings rather than the text being
+spelled out in advance — which is why every reading of a polyphone is
+matchable, where a default reading table only ever offers one of them:
+
+```ts
+match(dictionary, "银行", "yh")?.score; // 7, and 银行 is yínháng
+match(dictionary, "银行", "yx")?.score; // 5 — a reading 行 has, but not here
+```
+
+Both match, and the decoder's own reading is what ranks them, along with
+whether the match starts a word. Sort a filtered list by `score`, highest
+first. Ranges rather than a boolean, in code points, so a caller can highlight
+what matched — and there is more than one where the query stepped over
+something with no reading of its own, as 北京·大学 does.
+
+The `core` tier is enough, so a page that never loads a word list can still
+filter. See [matching](docs/matching/).
+
+## Go the other way: pinyin to hanzi
+
+`candidates` answers a pinyin query with the words it could be spelling, and
+`homophonesOf` lists the words read exactly like one you already have. Both run
+off a reverse index derived from the dictionary in memory.
+
+```ts
+import { candidates, homophonesOf, ReverseIndex } from "@kensio/pinyinjs";
+
+const index = ReverseIndex.of(dictionary);
+
+candidates(index, "shi", { limit: 5 }); // ["是", "时", "事", "使", "市"]
+candidates(index, "yinhang"); // ["銀行", "银行", "引吭", "引航", "印航"]
+candidates(index, "yínháng"); // ["銀行", "银行"], narrowed by tone
+homophonesOf(index, "长城"); // ["長城", "長程", "长程", "常程"]
+```
+
+This is the half of search with no haystack: `match` filters Chinese text you
+already hold, and this answers a query with nothing behind it but the
+dictionary. Pinyin-only lookup, a homophones section on a word page, or a
+browser input method for somebody with no Chinese keyboard.
+
+**Nothing is downloaded for it.** Shipping a reverse index of the `full` tier
+would add 1,995 KB to a 2,378 KB download and could not be compressed below
+about 1,474 KB, so the client derives it instead — 510 ms and 2.03 MB of heap on
+`full`, and 4 ms on `core`. `ReverseIndex.building` drives that build a slice at
+a time, and `serialise` hands it out of a worker, so nothing has to hold the
+main thread for it.
+
+Candidates come back likeliest first, because a posting is a dictionary position
+and a position indexes the frequency table. Tones may be written or left off,
+`v` and `u:` reach ü, and the r of 儿化 is optional, so 玩儿 answers to `wanr`
+and to `wan`. Both scripts are dictionary keys, so pass a script preference and
+the conversion tables to keep one writing of a word rather than both. See
+[candidates](docs/candidates/).
+
+## Check what somebody typed
+
+`check` marks a typed pinyin transcription against the Chinese it was written
+for, syllable by syllable.
+
+```ts
+import { check } from "@kensio/pinyinjs";
+
+const marked = check(dictionary, "银行", "yínxíng");
+marked.syllables.map((one) => one.verdict); // ["correct", "wrong"]
+marked.syllables[1]?.source; // "行", the character that was misread
+marked.score; // 0.5
+```
+
+Each syllable comes back as one of six verdicts, with `source` and `at` naming
+the characters it reads so a mistake can be shown against the text rather than
+against the answer:
+
+| Verdict    | Means                                          |
+| ---------- | ---------------------------------------------- |
+| `correct`  | right syllable, right tone                     |
+| `toneless` | right syllable, no tone written                |
+| `tone`     | right syllable, wrong tone                     |
+| `wrong`    | wrong syllable                                 |
+| `missing`  | a syllable of the reading that was not typed   |
+| `extra`    | a syllable typed that the reading does not use |
+
+**The point is being fair in the ways a string comparison cannot be.** All of
+these are a learner being marked wrong for being right, and all of them pass:
+
+```ts
+check(dictionary, "北京", "bei3jīng").isCorrect; // true, either notation, mixed
+check(dictionary, "行", "háng").isCorrect; // true, the decoder was guessing too
+check(dictionary, "你好", "ní hǎo").isCorrect; // true, said with sandhi
+check(dictionary, "不是", "bù shì").isCorrect; // true, written without it
+check(dictionary, "海鸥", "hǎiōu").isCorrect; // true, an apostrophe is not a sound
+check(dictionary, "我的书", "wǒ de shū").isCorrect; // true, pinyin marks no neutral tone
+```
+
+A guess is only forgiven where the library was guessing: 行 alone is chosen by a
+prior and nothing more, whereas 银行 settles both its syllables, so `yínxíng` is
+a real mistake. See [confidence](docs/confidence/).
+
+### Tones and word spacing
+
+Two axes an exercise may or may not be teaching yet, each reported always and
+each counted only when asked for.
+
+Tones left off are `toneless` rather than `tone`, because `Syllable.tone` knows
+the difference:
+
+```ts
+check(dictionary, "北京", "bei jing").isCorrect; // true
+check(dictionary, "北京", "bei jing", { tones: "required" }).isCorrect; // false
+```
+
+Word spacing is graded on `spacing`, separately from the syllable's own verdict,
+because it is a separate mistake: `yín háng` reads 银行 perfectly and writes it
+as two words.
+
+```ts
+const split = check(dictionary, "银行", "yín háng");
+split.syllables.map((one) => one.spacing); // ["correct", "split"]
+split.isCorrect; // true, spacing is not counted by default
+check(dictionary, "银行", "yín háng", { spacing: "required" }).isCorrect; // false
+```
+
+It is `split` for a word written as two and `joined` for two written as one, and
+it is tolerant in the same spirit as everything else — 分词连写 and the words the
+dictionary knows are two conventions this package writes, and a learner may have
+been taught either:
+
+```ts
+const graded = { spacing: "required" } as const;
+check(dictionary, "他看了", "tā kànle", graded).isCorrect; // true, 分词连写
+check(dictionary, "他看了", "tā kàn le", graded).isCorrect; // true, the words
+check(dictionary, "干干净净", "gāngān jìngjìng", graded).isCorrect; // true, a hyphen is a boundary
+check(dictionary, "我要去北京。", "wǒyàoqùběijīng", graded).isCorrect; // false
+```
+
+Takes every `convert` option besides, so an exercise that knows which sense of 长
+its own sentence uses can say so with `readings`. More in
+[checking](docs/checking/).
+
 ## Look words up
 
 ```ts
@@ -639,6 +838,12 @@ This is orthography and not translation — 软件 becomes 軟件, never 軟體.
 | `convertPieces(dictionary, text, ...)`               | the same, per syllable, with confidence           |
 | `convertToHtml(dictionary, text, ...)`               | the same, as HTML                                 |
 | `convertToAnnotatedHtml(dictionary, text, ...)`      | hanzi and pinyin together, as ruby HTML           |
+| `segment(dictionary, text)`                          | split text into words                             |
+| `match(dictionary, haystack, query)`                 | where a pinyin query matches a text               |
+| `candidates(index, query, options?)`                 | pinyin → the hanzi it could be                    |
+| `homophonesOf(index, word, options?)`                | the words read exactly like this one              |
+| `ReverseIndex.of(dictionary)`                        | derive the reading → words index                  |
+| `check(dictionary, text, typed, options?)`           | mark typed pinyin against the text                |
 | `slug(dictionary, text, options?)`                   | hanzi → a URL-safe slug                           |
 | `joinPieces(pieces)` / `toHtml(pieces)`              | render pieces                                     |
 | `isUncertain(confidence)`                            | was this syllable a guess?                        |
@@ -654,8 +859,9 @@ This is orthography and not translation — 软件 becomes 軟件, never 軟體.
 | `convertGreedily(...)`                               | the old longest-match decoder, kept as a baseline |
 
 Types (`Syllable`, `Tone`, `ConvertOptions`, `ConvertedPiece`,
-`ReadingConfidence`, `HtmlOptions`, `WordEntry`, `Tier`, `Locale`, `Script`) are
-exported alongside them.
+`ReadingConfidence`, `HtmlOptions`, `Segment`, `PinyinMatch`, `MatchRange`,
+`ReadingConfidence`, `HtmlOptions`, `Segment`, `PinyinCheck`, `CheckedSyllable`,
+`WordEntry`, `Tier`, `Locale`, `Script`) are exported alongside them.
 
 ## Development