tinyfix 0.0.0-stage → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Caleb Fahlgren
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,135 @@
1
- # Temporary Holding Version
1
+ <p align="center">
2
+ <img src=".github/banner.png" alt="tinyfix: autocorrect that reads both sides of a word" width="880">
3
+ </p>
2
4
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
5
+ Spell check can't tell that “form” was meant to be “from”, since both are real words. tinyfix can. It's a 2 MB model that reads the words on both sides of each word, and it runs in the browser, so nothing anyone types leaves the page.
6
+
7
+ ```bash
8
+ npm install tinyfix
9
+ ```
10
+
11
+ ```ts
12
+ import { loadTinyfix } from "tinyfix";
13
+
14
+ const tinyfix = await loadTinyfix();
15
+
16
+ await tinyfix.fix("I got a letter form the bank and did not recieve it.");
17
+ // "I got a letter from the bank and did not receive it."
18
+ ```
19
+
20
+ <p align="center">
21
+ <img src=".github/what-it-does.png" alt="You type: I got a letter form the bank. Their is a problem, and I am to tired to recieve it. You get: I got a letter from the bank. There is a problem, and I am too tired to receive it." width="880">
22
+ </p>
23
+
24
+ It's the autocorrect in [offlinepage.app](https://offlinepage.app), a writing app that works in airplane mode.
25
+
26
+ ## Getting the fixes
27
+
28
+ `check` returns what it would change and where, so you can show it your own way: underline it, ask first, or apply it.
29
+
30
+ ```ts
31
+ const text = "I got a letter form the bank and did not recieve it.";
32
+ const fixes = await tinyfix.check(text);
33
+ // [
34
+ // { from: 15, to: 19, original: "form", replacement: "from", kind: "context", confidence: 1 },
35
+ // { from: 41, to: 48, original: "recieve", replacement: "receive", kind: "spelling", confidence: 0.95 },
36
+ // ]
37
+
38
+ tinyfix.apply(text, fixes);
39
+ // "I got a letter from the bank and did not receive it."
40
+ ```
41
+
42
+ A fix is one of two kinds:
43
+
44
+ - `"spelling"`: the word isn't a word. “recieve” becomes “receive”.
45
+ - `"context"`: the word is real but doesn't fit the words around it. “form” becomes “from”.
46
+
47
+ ## API
48
+
49
+ | Call | What it does |
50
+ | ------------------------------- | ------------------------------------------------------------------------ |
51
+ | `loadTinyfix(options?)` | Downloads the model and word lists, and returns a checker. Call it once. |
52
+ | `tinyfix.check(text, options?)` | Returns the fixes for a text, in order. They never overlap. |
53
+ | `tinyfix.apply(text, fixes)` | Returns the text with those fixes made. |
54
+ | `tinyfix.fix(text, options?)` | `check`, then `apply`. |
55
+ | `tinyfix.ignore(word)` | Never change this word again. |
56
+ | `tinyfix.dispose()` | Frees the model. |
57
+
58
+ ### `loadTinyfix` options
59
+
60
+ | Option | Default | What it does |
61
+ | ----------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
62
+ | `baseUrl` | The [model repo](https://huggingface.co/paperweight/tinyfix) on Hugging Face | Where the four model files are, if you host them yourself |
63
+ | `model` | `true` | `false` skips the model. Only the word list is downloaded and only misspellings are fixed. |
64
+ | `mode` | `"balanced"` | `"careful"` makes fewer wrong changes and fixes fewer wrong words |
65
+ | `wasmPaths` | Found by your bundler | Where ONNX Runtime's `.wasm` files are, if you host those too |
66
+
67
+ ### `check` options
68
+
69
+ | Option | What it does |
70
+ | ------- | ---------------------------------------------------------------------------------------------- |
71
+ | `range` | `[from, to]`. Only words inside it are checked; the rest of the text is still read as context. |
72
+ | `kinds` | `["spelling"]`, `["context"]` or both. Both by default. |
73
+
74
+ ## Fixing as someone types
75
+
76
+ `check` treats the text as finished writing. To fix while someone types, check one word at a time with `range`, so you never re-check the whole page on a keystroke. offlinepage.app does this:
77
+
78
+ ```ts
79
+ // When a word is finished: is it misspelled?
80
+ await tinyfix.check(paragraph, { range: [wordStart, wordEnd], kinds: ["spelling"] });
81
+
82
+ // Two words later, when it has words on both sides: is it the wrong word?
83
+ await tinyfix.check(paragraph, { range: [earlierStart, earlierEnd], kinds: ["context"] });
84
+ ```
85
+
86
+ When someone undoes a fix, call `tinyfix.ignore(fix.original)` so it doesn't come back.
87
+
88
+ ## How well it works
89
+
90
+ <table>
91
+ <tr>
92
+ <td align="center" width="33%"><h3>94%</h3>of its fixes are right</td>
93
+ <td align="center" width="33%"><h3>40%</h3>of wrong words fixed</td>
94
+ <td align="center" width="33%"><h3>~3 ms</h3>per check in Chrome</td>
95
+ </tr>
96
+ </table>
97
+
98
+ On about 110,000 words of correct writing it wrongly changed about 1 word in 2,000. It would rather leave a typo alone than change a word you meant, so it misses more wrong words than it catches. The [model card](https://huggingface.co/paperweight/tinyfix) has the test sets, what it misses and how it was trained.
99
+
100
+ ## What it leaves alone
101
+
102
+ - Code in backticks, inline or fenced
103
+ - Web addresses, emails, file paths, `#tags` and words with digits
104
+ - Names, and words in ALL CAPS
105
+ - Anything you pass to `ignore`
106
+
107
+ A line break counts as the end of a sentence. Positions are JavaScript string indexes.
108
+
109
+ ## What it downloads
110
+
111
+ About 3.5 MB the first time, from Hugging Face: the 2.1 MB model, its 20,000-word vocabulary, a list of words people mix up, and an 80,000-word spelling list. Each version of this package loads one exact revision of those files, so it always behaves the same.
112
+
113
+ To serve them from your own site, copy the four files from the [model repo](https://huggingface.co/paperweight/tinyfix/tree/main) (`wrong-word.onnx`, `wrong-word-vocab.txt`, `wrong-word-confusions.txt`, `words.txt`) and point `baseUrl` at the folder:
114
+
115
+ ```ts
116
+ const tinyfix = await loadTinyfix({ baseUrl: "/models/" });
117
+ ```
118
+
119
+ The model runs on [ONNX Runtime Web](https://onnxruntime.ai/docs/tutorials/web/), which is installed with this package. On a page it runs in ONNX Runtime's own worker, so loading it doesn't freeze typing.
120
+
121
+ ## Where it runs
122
+
123
+ Browsers, Web Workers and Node 18 or later. English only.
124
+
125
+ ## Development
126
+
127
+ ```bash
128
+ npm test # downloads the model files once, then runs the tests
129
+ npm run check # types
130
+ npm run lint
131
+ ```
132
+
133
+ ## License
134
+
135
+ MIT, for this package and for the model.
@@ -0,0 +1,65 @@
1
+ import { WordTrie } from "./trie.js";
2
+ export interface ContextScorer {
3
+ /** Returns the log probability of each word appearing right after `textBefore`. */
4
+ score(textBefore: string, words: readonly string[]): Promise<number[]>;
5
+ }
6
+ export interface FixerWeights {
7
+ frequency: number;
8
+ context: number;
9
+ keepCost: number;
10
+ minConfidence: number;
11
+ }
12
+ export declare const DEFAULT_WEIGHTS: FixerWeights;
13
+ /** Use with a context scorer, so the model helps choose between spelling candidates. */
14
+ export declare const WITH_MODEL_WEIGHTS: FixerWeights;
15
+ export interface WordEntry {
16
+ word: string;
17
+ count: number;
18
+ }
19
+ export interface Candidate {
20
+ word: string;
21
+ cost: number;
22
+ logFrequency: number;
23
+ contextGain: number;
24
+ }
25
+ export interface Correction {
26
+ original: string;
27
+ replacement: string;
28
+ confidence: number;
29
+ /** True when the word was spelled correctly but didn't fit the sentence. */
30
+ usedContext?: boolean;
31
+ }
32
+ export interface FixerOptions {
33
+ words: readonly WordEntry[];
34
+ /** A trie already built from `words`, in the same order. Built here when missing. */
35
+ trie?: WordTrie;
36
+ scorer?: ContextScorer;
37
+ weights?: Partial<FixerWeights>;
38
+ maxCandidates?: number;
39
+ }
40
+ export declare function parseWordList(text: string): WordEntry[];
41
+ export declare function pickCandidate(candidates: readonly Candidate[], weights: FixerWeights): {
42
+ word: string;
43
+ confidence: number;
44
+ } | null;
45
+ export declare class Fixer {
46
+ private readonly words;
47
+ private readonly known;
48
+ private readonly trie;
49
+ private readonly scorer;
50
+ private readonly weights;
51
+ private readonly maxCandidates;
52
+ private readonly totalCount;
53
+ constructor(options: FixerOptions);
54
+ /** Treats a word as correct from now on, for example after the user undoes a fix. */
55
+ learn(word: string): void;
56
+ isKnown(word: string): boolean;
57
+ /** Ranks spelling candidates for an unknown word, or returns null when the word should be left alone. */
58
+ analyze(textBefore: string, word: string): Promise<Candidate[] | null>;
59
+ /** Returns a correction for the word just finished after `textBefore`, or null to keep it. */
60
+ correct(textBefore: string, word: string): Promise<Correction | null>;
61
+ private readonly undone;
62
+ /** Stops a fixed rule from firing again for a word the user restored. */
63
+ rememberUndo(word: string): void;
64
+ wasUndone(word: string): boolean;
65
+ }
@@ -0,0 +1,99 @@
1
+ import { applyFixedRule, isBritishSpelling, isCheckable, isNewerWord, matchCase } from "./rules.js";
2
+ import { WordTrie } from "./trie.js";
3
+ import { typoCost } from "./typoCost.js";
4
+ export const DEFAULT_WEIGHTS = { frequency: 0.5, context: 0, keepCost: 10, minConfidence: 0.7 };
5
+ /** Use with a context scorer, so the model helps choose between spelling candidates. */
6
+ export const WITH_MODEL_WEIGHTS = { ...DEFAULT_WEIGHTS, context: 0.25 };
7
+ export function parseWordList(text) {
8
+ return text.split("\n").flatMap((line) => {
9
+ const [word, count] = line.trim().split(/\s+/);
10
+ return word && count ? [{ word, count: Number(count) }] : [];
11
+ });
12
+ }
13
+ export function pickCandidate(candidates, weights) {
14
+ if (candidates.length === 0)
15
+ return null;
16
+ const scored = candidates.map((candidate) => ({
17
+ word: candidate.word,
18
+ score: -candidate.cost + weights.frequency * candidate.logFrequency + weights.context * candidate.contextGain,
19
+ }));
20
+ const best = scored.reduce((top, entry) => (entry.score > top.score ? entry : top));
21
+ const total = scored.reduce((sum, entry) => sum + Math.exp(entry.score - best.score), Math.exp(-weights.keepCost - best.score));
22
+ const confidence = 1 / total;
23
+ return confidence >= weights.minConfidence ? { word: best.word, confidence } : null;
24
+ }
25
+ export class Fixer {
26
+ words;
27
+ known = new Set();
28
+ trie;
29
+ scorer;
30
+ weights;
31
+ maxCandidates;
32
+ totalCount;
33
+ constructor(options) {
34
+ this.words = options.words;
35
+ this.words.forEach((entry) => this.known.add(entry.word));
36
+ this.trie = options.trie ?? new WordTrie(this.words.map((entry) => entry.word));
37
+ this.scorer = options.scorer;
38
+ this.weights = { ...DEFAULT_WEIGHTS, ...options.weights };
39
+ this.maxCandidates = options.maxCandidates ?? 12;
40
+ this.totalCount = this.words.reduce((sum, entry) => sum + entry.count, 0);
41
+ }
42
+ /** Treats a word as correct from now on, for example after the user undoes a fix. */
43
+ learn(word) {
44
+ this.known.add(word.toLowerCase());
45
+ }
46
+ isKnown(word) {
47
+ const lower = word.toLowerCase();
48
+ const isListed = (candidate) => this.known.has(candidate);
49
+ return isListed(lower) || isNewerWord(lower) || isBritishSpelling(lower, isListed);
50
+ }
51
+ /** Ranks spelling candidates for an unknown word, or returns null when the word should be left alone. */
52
+ async analyze(textBefore, word) {
53
+ if (!isCheckable(word, textBefore) || this.isKnown(word))
54
+ return null;
55
+ const lower = word.toLowerCase();
56
+ const maxEdits = lower.length <= 4 ? 1 : 2;
57
+ const ranked = this.trie
58
+ .search(lower, maxEdits)
59
+ .map(({ index }) => {
60
+ const entry = this.words[index];
61
+ return {
62
+ word: entry.word,
63
+ cost: typoCost(lower, entry.word),
64
+ logFrequency: Math.log(entry.count / this.totalCount),
65
+ contextGain: 0,
66
+ };
67
+ })
68
+ .sort((first, second) => second.logFrequency - second.cost - (first.logFrequency - first.cost))
69
+ .slice(0, this.maxCandidates);
70
+ if (!this.scorer || ranked.length === 0)
71
+ return ranked;
72
+ const scores = await this.scorer.score(textBefore, [
73
+ word,
74
+ ...ranked.map((candidate) => matchCase(word, candidate.word)),
75
+ ]);
76
+ const typedScore = scores[0];
77
+ return ranked.map((candidate, index) => ({ ...candidate, contextGain: scores[index + 1] - typedScore }));
78
+ }
79
+ /** Returns a correction for the word just finished after `textBefore`, or null to keep it. */
80
+ async correct(textBefore, word) {
81
+ const ruled = applyFixedRule(word);
82
+ if (ruled && ruled !== word && !this.wasUndone(word))
83
+ return { original: word, replacement: ruled, confidence: 1 };
84
+ const candidates = await this.analyze(textBefore, word);
85
+ if (!candidates)
86
+ return null;
87
+ const pick = pickCandidate(candidates, this.weights);
88
+ return pick ? { original: word, replacement: matchCase(word, pick.word), confidence: pick.confidence } : null;
89
+ }
90
+ undone = new Set();
91
+ /** Stops a fixed rule from firing again for a word the user restored. */
92
+ rememberUndo(word) {
93
+ this.undone.add(word.toLowerCase());
94
+ this.learn(word);
95
+ }
96
+ wasUndone(word) {
97
+ return this.undone.has(word.toLowerCase());
98
+ }
99
+ }
@@ -0,0 +1,10 @@
1
+ /** True for newer everyday words the word list doesn't have. */
2
+ export declare const isNewerWord: (lower: string) => boolean;
3
+ /** Recognizes British spellings, so "colour" isn't treated as a misspelling of "color". */
4
+ export declare function isBritishSpelling(lower: string, isKnown: (word: string) => boolean): boolean;
5
+ export declare function isSentenceStart(textBefore: string): boolean;
6
+ /** Applies typed casing to a replacement, so "Teh" becomes "The". */
7
+ export declare function matchCase(typed: string, replacement: string): string;
8
+ export declare function applyFixedRule(word: string): string | null;
9
+ /** Words the engine must leave alone: names, acronyms, identifiers and anything not plainly a word. */
10
+ export declare function isCheckable(word: string, textBefore: string): boolean;
@@ -0,0 +1,117 @@
1
+ const CONTRACTIONS = new Map(Object.entries({
2
+ arent: "aren't",
3
+ cant: "can't",
4
+ couldnt: "couldn't",
5
+ didnt: "didn't",
6
+ doesnt: "doesn't",
7
+ dont: "don't",
8
+ hadnt: "hadn't",
9
+ hasnt: "hasn't",
10
+ havent: "haven't",
11
+ im: "I'm",
12
+ isnt: "isn't",
13
+ ive: "I've",
14
+ shouldnt: "shouldn't",
15
+ thats: "that's",
16
+ theres: "there's",
17
+ theyre: "they're",
18
+ theyve: "they've",
19
+ wasnt: "wasn't",
20
+ werent: "weren't",
21
+ weve: "we've",
22
+ whats: "what's",
23
+ wont: "won't",
24
+ wouldnt: "wouldn't",
25
+ youre: "you're",
26
+ youve: "you've",
27
+ }));
28
+ const BRITISH_OUR_ENDING = /(.{2,})our(s|ed|ing|ite|ites|ful|able|ably|less|hood|hoods|ly|ist|ists|er|ers)?$/;
29
+ const BRITISH_ISE_ENDING = /(is|ys)(e|es|ed|ing|er|ers|ation|ations|able)$/;
30
+ const BRITISH_DOUBLED_L = /^(un|re|mis|pre)?(travel|cancel|model|label|signal|fuel|level|total|marvel|counsel|jewel|tunnel|channel|shovel|dial|equal|quarrel|rival|panel|funnel|pencil|spiral|trial)l(ed|ing|er|ers|or|ors)$/;
31
+ const MIN_LENGTH_FOR_ISE_RULE = 7;
32
+ /** "councillor", "panellist", "pedalling": a doubled l before an ending where American English has one. */
33
+ const BRITISH_DOUBLED_L_ENDING = /ll(ed|ing|er|ers|or|ors|ist|ists|ous)$/;
34
+ /** "epicentre", "kilometres", "centred": re where American English has er. */
35
+ const BRITISH_RE_ENDING = /([^aeiou])re(s|d)?$/;
36
+ const MIN_LENGTH_FOR_RE_RULE = 5;
37
+ /** "paediatric", "haemorrhage", "oesophagus": ae and oe where American English has e. */
38
+ const BRITISH_DIGRAPH = /(ae|oe)/;
39
+ const MIN_LENGTH_FOR_DIGRAPH_RULE = 7;
40
+ const BRITISH_WORDS = new Set(("theatre centre centred metre litre fibre calibre sombre spectre defence offence licence pretence catalogue analogue programme judgement " +
41
+ "ageing aluminium fulfil enrol skilful wilful jewellery tyre grey mould moulding plough sceptical sceptic cheque kerb draught manoeuvre " +
42
+ "paediatric anaesthetic foetus oestrogen moustache pyjamas storey acknowledgement instalment distil instil appal " +
43
+ "whisky enquiry amongst whilst")
44
+ .split(" ")
45
+ .flatMap((word) => [word, `${word}s`]));
46
+ /** Everyday words newer than the word list, which would otherwise be "corrected" (emoji → meiji, todo → too). */
47
+ const NEWER_WORDS = new Set(("https emoji todo selfie hashtag livestream chatbot vlog vlogger influencer fintech covid podcaster blogger unfollow " +
48
+ "unfriend retweet screenshare webinar onboarding offboarding skincare smartwatch earbud airpod wifi bluetooth " +
49
+ "async runtime frontend backend dataset username login logout signup checkout touchscreen spreadsheet " +
50
+ "workflow timeline toolbar dropdown popup tooltip codebase repo config dev devops cybersecurity crypto " +
51
+ "kombucha matcha ramen udon pho bao boba poke quinoa kale hummus falafel tahini sriracha kimchi gochujang " +
52
+ "tofu tempeh seitan acai chia granola brunch latte cappuccino espresso macchiato americano frappe smoothie " +
53
+ "barista crema robusta arabica croissant focaccia ciabatta brioche sourdough gelato prosecco charcuterie " +
54
+ "hygge wellness mindfulness hangry yeet meme binge rewatch preorder crowdfund carpool rideshare " +
55
+ "staycation glamping bae vibe vibes ghosting doomscroll doomscrolling cringe adulting")
56
+ .split(" ")
57
+ .flatMap((word) => [word, `${word}s`]));
58
+ /** True for newer everyday words the word list doesn't have. */
59
+ export const isNewerWord = (lower) => NEWER_WORDS.has(lower);
60
+ /** Recognizes British spellings, so "colour" isn't treated as a misspelling of "color". */
61
+ export function isBritishSpelling(lower, isKnown) {
62
+ if (BRITISH_WORDS.has(lower))
63
+ return true;
64
+ const doubled = BRITISH_DOUBLED_L.exec(lower);
65
+ if (doubled)
66
+ return true;
67
+ const withOr = lower.replace(BRITISH_OUR_ENDING, "$1or$2");
68
+ if (withOr !== lower && isKnown(withOr))
69
+ return true;
70
+ const singleL = lower.replace(BRITISH_DOUBLED_L_ENDING, "l$1");
71
+ if (singleL !== lower && isKnown(singleL))
72
+ return true;
73
+ const withEr = lower.length < MIN_LENGTH_FOR_RE_RULE
74
+ ? lower
75
+ : lower.replace(BRITISH_RE_ENDING, (_, before, ending = "") => `${before}er${ending === "d" ? "ed" : ending}`);
76
+ if (withEr !== lower && isKnown(withEr))
77
+ return true;
78
+ if (lower.length >= MIN_LENGTH_FOR_DIGRAPH_RULE &&
79
+ BRITISH_DIGRAPH.test(lower) &&
80
+ isKnown(lower.replace(BRITISH_DIGRAPH, "e")))
81
+ return true;
82
+ if (lower.length < MIN_LENGTH_FOR_ISE_RULE)
83
+ return false;
84
+ const withZ = lower.replace(BRITISH_ISE_ENDING, (_, stem, ending) => `${stem === "is" ? "iz" : "yz"}${ending}`);
85
+ return withZ !== lower && isKnown(withZ);
86
+ }
87
+ export function isSentenceStart(textBefore) {
88
+ return /(^|[.!?:\n]\s*|^\s*[-*•]\s+)$/.test(textBefore);
89
+ }
90
+ /** Applies typed casing to a replacement, so "Teh" becomes "The". */
91
+ export function matchCase(typed, replacement) {
92
+ const isCapitalized = typed[0] !== typed[0].toLowerCase();
93
+ return isCapitalized ? replacement[0].toUpperCase() + replacement.slice(1) : replacement;
94
+ }
95
+ export function applyFixedRule(word) {
96
+ if (word === "i")
97
+ return "I";
98
+ const lower = word.toLowerCase();
99
+ const contraction = CONTRACTIONS.get(lower);
100
+ if (!contraction)
101
+ return null;
102
+ return contraction.startsWith("I") ? contraction : matchCase(word, contraction);
103
+ }
104
+ /** Words the engine must leave alone: names, acronyms, identifiers and anything not plainly a word. */
105
+ export function isCheckable(word, textBefore) {
106
+ if (word.length < 3 || !/^[A-Za-z]+$/.test(word))
107
+ return false;
108
+ if (word === word.toUpperCase())
109
+ return false;
110
+ if (/[A-Z]/.test(word.slice(1)))
111
+ return false;
112
+ const isCapitalized = word[0] !== word[0].toLowerCase();
113
+ if (isCapitalized && !isSentenceStart(textBefore))
114
+ return false;
115
+ const previousToken = textBefore.trimEnd().split(/\s/).pop() ?? "";
116
+ return !/[/@\\_#]|\.\w/.test(previousToken);
117
+ }
@@ -0,0 +1,18 @@
1
+ export interface TrieMatch {
2
+ index: number;
3
+ distance: number;
4
+ }
5
+ /** Compact trie that finds dictionary words within a small edit distance of a query. */
6
+ export declare class WordTrie {
7
+ private readonly character;
8
+ private readonly firstChild;
9
+ private readonly nextSibling;
10
+ private readonly wordIndex;
11
+ constructor(words: readonly string[]);
12
+ /** Builds the same trie a few thousand words at a time, calling `pause` in between so a page stays responsive. */
13
+ static inSteps(words: readonly string[], pause: () => Promise<void>, step?: number): Promise<WordTrie>;
14
+ private insert;
15
+ private findOrCreateChild;
16
+ /** Returns every word within `maxEdits` insertions, deletions, substitutions or adjacent swaps. */
17
+ search(query: string, maxEdits: number): TrieMatch[];
18
+ }
@@ -0,0 +1,74 @@
1
+ /** Compact trie that finds dictionary words within a small edit distance of a query. */
2
+ export class WordTrie {
3
+ character = [0];
4
+ firstChild = [-1];
5
+ nextSibling = [-1];
6
+ wordIndex = [-1];
7
+ constructor(words) {
8
+ words.forEach((word, index) => this.insert(word, index));
9
+ }
10
+ /** Builds the same trie a few thousand words at a time, calling `pause` in between so a page stays responsive. */
11
+ static async inSteps(words, pause, step = 8000) {
12
+ const trie = new WordTrie([]);
13
+ for (let start = 0; start < words.length; start += step) {
14
+ const end = Math.min(words.length, start + step);
15
+ for (let index = start; index < end; index++)
16
+ trie.insert(words[index], index);
17
+ await pause();
18
+ }
19
+ return trie;
20
+ }
21
+ insert(word, index) {
22
+ let node = 0;
23
+ for (let position = 0; position < word.length; position++) {
24
+ node = this.findOrCreateChild(node, word.charCodeAt(position));
25
+ }
26
+ this.wordIndex[node] = index;
27
+ }
28
+ findOrCreateChild(parent, code) {
29
+ let child = this.firstChild[parent];
30
+ while (child !== -1) {
31
+ if (this.character[child] === code)
32
+ return child;
33
+ child = this.nextSibling[child];
34
+ }
35
+ const created = this.character.length;
36
+ this.character.push(code);
37
+ this.firstChild.push(-1);
38
+ this.nextSibling.push(this.firstChild[parent]);
39
+ this.wordIndex.push(-1);
40
+ this.firstChild[parent] = created;
41
+ return created;
42
+ }
43
+ /** Returns every word within `maxEdits` insertions, deletions, substitutions or adjacent swaps. */
44
+ search(query, maxEdits) {
45
+ const width = query.length + 1;
46
+ const matches = [];
47
+ const typed = Array.from({ length: query.length }, (_, position) => query.charCodeAt(position));
48
+ const walk = (node, previousCode, row, rowBefore) => {
49
+ for (let child = this.firstChild[node]; child !== -1; child = this.nextSibling[child]) {
50
+ const code = this.character[child];
51
+ const next = new Uint8Array(width);
52
+ next[0] = row[0] + 1;
53
+ let smallest = next[0];
54
+ for (let column = 1; column < width; column++) {
55
+ const substitution = row[column - 1] + (typed[column - 1] === code ? 0 : 1);
56
+ let value = Math.min(row[column] + 1, next[column - 1] + 1, substitution);
57
+ const isSwap = rowBefore && column > 1 && typed[column - 1] === previousCode && typed[column - 2] === code;
58
+ if (isSwap)
59
+ value = Math.min(value, rowBefore[column - 2] + 1);
60
+ next[column] = value;
61
+ smallest = Math.min(smallest, value);
62
+ }
63
+ const distance = next[width - 1];
64
+ if (this.wordIndex[child] !== -1 && distance <= maxEdits) {
65
+ matches.push({ index: this.wordIndex[child], distance });
66
+ }
67
+ if (smallest <= maxEdits)
68
+ walk(child, code, next, row);
69
+ }
70
+ };
71
+ walk(0, -1, Uint8Array.from({ length: width }, (_, column) => column), null);
72
+ return matches;
73
+ }
74
+ }
@@ -0,0 +1,2 @@
1
+ /** Weighted edit distance: a lower cost means the typo is a more believable slip of the fingers. */
2
+ export declare function typoCost(typed: string, intended: string): number;
@@ -0,0 +1,81 @@
1
+ const KEYBOARD_ROWS = ["qwertyuiop", "asdfghjkl", "zxcvbnm"];
2
+ const VOWELS = new Set("aeiou");
3
+ const COST = {
4
+ swap: 1.2,
5
+ doubledKey: 1.2,
6
+ adjacentKey: 1.6,
7
+ vowelForVowel: 2,
8
+ extraNearbyKey: 1.8,
9
+ missedKey: 2.4,
10
+ extraKey: 3,
11
+ wrongKey: 3.4,
12
+ firstLetterPenalty: 1.5,
13
+ };
14
+ function buildNeighbors() {
15
+ const neighbors = new Map();
16
+ KEYBOARD_ROWS.forEach((row, rowIndex) => {
17
+ [...row].forEach((key, keyIndex) => {
18
+ const near = new Set();
19
+ for (const [otherRow, shift] of [
20
+ [rowIndex, -1],
21
+ [rowIndex, 1],
22
+ [rowIndex - 1, 0],
23
+ [rowIndex - 1, 1],
24
+ [rowIndex + 1, -1],
25
+ [rowIndex + 1, 0],
26
+ ]) {
27
+ const other = KEYBOARD_ROWS[otherRow]?.[keyIndex + shift];
28
+ if (other)
29
+ near.add(other);
30
+ }
31
+ neighbors.set(key, near);
32
+ });
33
+ });
34
+ return neighbors;
35
+ }
36
+ const NEIGHBORS = buildNeighbors();
37
+ const isNearby = (first, second) => !!first && !!second && !!NEIGHBORS.get(first)?.has(second);
38
+ function substitutionCost(typed, intended) {
39
+ if (isNearby(typed, intended))
40
+ return COST.adjacentKey;
41
+ if (VOWELS.has(typed) && VOWELS.has(intended))
42
+ return COST.vowelForVowel;
43
+ return COST.wrongKey;
44
+ }
45
+ function extraKeyCost(typed, position) {
46
+ const key = typed[position];
47
+ if (key === typed[position - 1] || key === typed[position + 1])
48
+ return COST.doubledKey;
49
+ if (isNearby(key, typed[position - 1]) || isNearby(key, typed[position + 1]))
50
+ return COST.extraNearbyKey;
51
+ return COST.extraKey;
52
+ }
53
+ function missedKeyCost(intended, position) {
54
+ return intended[position] === intended[position - 1] ? COST.doubledKey : COST.missedKey;
55
+ }
56
+ /** A slip on a word's first letter is less likely, so it costs more. */
57
+ const start = (position) => (position === 0 ? COST.firstLetterPenalty : 0);
58
+ /** Weighted edit distance: a lower cost means the typo is a more believable slip of the fingers. */
59
+ export function typoCost(typed, intended) {
60
+ const rows = typed.length + 1;
61
+ const columns = intended.length + 1;
62
+ const table = Array.from({ length: rows }, () => new Array(columns).fill(0));
63
+ for (let row = 1; row < rows; row++)
64
+ table[row][0] = table[row - 1][0] + extraKeyCost(typed, row - 1) + start(row - 1);
65
+ for (let column = 1; column < columns; column++)
66
+ table[0][column] = table[0][column - 1] + missedKeyCost(intended, column - 1) + start(column - 1);
67
+ for (let row = 1; row < rows; row++) {
68
+ for (let column = 1; column < columns; column++) {
69
+ const typedKey = typed[row - 1];
70
+ const intendedKey = intended[column - 1];
71
+ const firstLetter = start(Math.min(row, column) - 1);
72
+ const replace = typedKey === intendedKey ? 0 : substitutionCost(typedKey, intendedKey) + firstLetter;
73
+ let best = Math.min(table[row - 1][column] + extraKeyCost(typed, row - 1) + firstLetter, table[row][column - 1] + missedKeyCost(intended, column - 1) + firstLetter, table[row - 1][column - 1] + replace);
74
+ const swapped = row > 1 && column > 1 && typedKey === intended[column - 2] && typed[row - 2] === intendedKey;
75
+ if (swapped)
76
+ best = Math.min(best, table[row - 2][column - 2] + COST.swap);
77
+ table[row][column] = best;
78
+ }
79
+ }
80
+ return table[rows - 1][columns - 1];
81
+ }
@@ -0,0 +1,64 @@
1
+ import type * as Ort from "onnxruntime-web";
2
+ import type { ContextScorer, Correction } from "./fixer.js";
3
+ export interface WrongWordOptions {
4
+ ort: typeof Ort;
5
+ session: Ort.InferenceSession;
6
+ /** Lines of the model's vocab.txt, most common word first. */
7
+ vocabulary: readonly string[];
8
+ /** Lines of the model's confusions.txt: a typed word, then words people meant by it. */
9
+ confusions?: readonly string[];
10
+ settings?: WrongWordSettings;
11
+ }
12
+ export interface WrongWordSettings {
13
+ /** A nearby word must beat the typed word by at least this much. */
14
+ threshold: number;
15
+ /** How strongly an unlikely slip of the fingers counts against a nearby word. */
16
+ costWeight: number;
17
+ }
18
+ /** Tuned on a dev split to wrongly change at most about 1 correct word in 2,000. */
19
+ export declare const CAREFUL_WRONG_WORD: WrongWordSettings;
20
+ /**
21
+ * On the lab bench's test sets this fixes about 43% of wrong words and wrongly changes about 1 correct word in 1,750,
22
+ * about the same as the model before it. A threshold of 2.75 fixes about 1 point more and changes about 1 in 1,550.
23
+ */
24
+ export declare const EAGER_WRONG_WORD: WrongWordSettings;
25
+ export interface WrongWordFinder {
26
+ check(textBefore: string, word: string, textAfter: string): Promise<Correction | null>;
27
+ }
28
+ export interface NearbyWord {
29
+ word: string;
30
+ /** How much better this word fits than the typed word, as a log-odds difference. */
31
+ margin: number;
32
+ /** How unlikely it is to type the given word when meaning this one. */
33
+ cost: number;
34
+ /** True when the word is a known mix-up for the typed word. */
35
+ isKnownMixup: boolean;
36
+ }
37
+ /** True for pairs like "services" and "service" or "continued" and "continues". */
38
+ export declare function differsOnlyInEnding(first: string, second: string): boolean;
39
+ /** Compares a correctly spelled word with nearby real words, using the words on both sides of it. */
40
+ export declare class WrongWordChecker implements WrongWordFinder, ContextScorer {
41
+ private readonly options;
42
+ settings: WrongWordSettings;
43
+ private readonly ids;
44
+ private readonly targets;
45
+ private readonly trie;
46
+ private readonly neighbors;
47
+ private readonly knownMixups;
48
+ constructor(options: WrongWordOptions);
49
+ private tokenId;
50
+ private nearbyIds;
51
+ /** Returns nearby words with how well each fits, or null when the word isn't one the model can judge. */
52
+ compare(textBefore: string, word: string, textAfter: string, onlyKnownMixups?: boolean): Promise<NearbyWord[] | null>;
53
+ /** Scores spelling candidates by how well they fit the text before them. The first word is the misspelling itself. */
54
+ score(textBefore: string, words: readonly string[]): Promise<number[]>;
55
+ /** Runs the model once per row of candidates, returning scores as one flat array of CANDIDATES per row. */
56
+ private runModel;
57
+ /** Short words and words with an apostrophe are left to the list of known mix-ups ("to" and "too", "it's" and "its"). */
58
+ private hasKnownMixups;
59
+ /** Whether a word is judged at all, and if so whether only against its known mix-ups. Null means leave it alone. */
60
+ judgedAgainst(word: string, textBefore: string): "nearby" | "knownMixups" | null;
61
+ private value;
62
+ /** Returns a fix when a nearby word fits the sentence clearly better than the one typed. */
63
+ check(textBefore: string, word: string, textAfter: string): Promise<Correction | null>;
64
+ }
@@ -0,0 +1,208 @@
1
+ import { isCheckable, isSentenceStart, matchCase } from "./rules.js";
2
+ import { WordTrie } from "./trie.js";
3
+ import { typoCost } from "./typoCost.js";
4
+ const LEFT_WINDOW = 16;
5
+ const RIGHT_WINDOW = 4;
6
+ const MAX_NEIGHBORS = 8;
7
+ const CANDIDATES = MAX_NEIGHBORS + 1;
8
+ const PAD = 0;
9
+ const UNKNOWN = 1;
10
+ const NUMBER = 2;
11
+ const START = 3;
12
+ const NAME = 4;
13
+ const SPECIAL_TOKENS = 5;
14
+ /** Known mix-ups like "their" and "there" are likely even when they're far apart on the keyboard. */
15
+ const MAX_KNOWN_MIXUP_COST = 1.5;
16
+ /** A known mix-up needs to clear a lower bar than other nearby words, which are mostly slips of the fingers. */
17
+ const KNOWN_MIXUP_BONUS = 1;
18
+ /** For a word with known mix-ups, any other nearby word needs to clear a higher bar: "your" is rarely meant as "you". */
19
+ const OTHER_WORD_PENALTY = 1;
20
+ /** A word that only differs by an ending ("examples" and "example") needs to clear a higher bar, unless it's a known mix-up. */
21
+ const INFLECTION_PENALTY = 1;
22
+ const INFLECTION_ENDINGS = new Set(["", "s", "es", "d", "ed", "ing", "r", "er", "ly"]);
23
+ const MIN_STEM = 3;
24
+ const TOKEN = /[A-Za-z]+(?:'[A-Za-z]+)?|\d+(?:[.,]\d+)*|\n|[^\s\w]/g;
25
+ const SENTENCE_OPENERS = new Set([".", "!", "?", ":", '"']);
26
+ const WORD = /^[a-z]+(?:'[a-z]+)?$/;
27
+ const LISTED_WORD = /^[A-Za-z]+(?:'[A-Za-z]+)?$/;
28
+ /** Tuned on a dev split to wrongly change at most about 1 correct word in 2,000. */
29
+ export const CAREFUL_WRONG_WORD = { threshold: 5, costWeight: 0.5 };
30
+ /**
31
+ * On the lab bench's test sets this fixes about 43% of wrong words and wrongly changes about 1 correct word in 1,750,
32
+ * about the same as the model before it. A threshold of 2.75 fixes about 1 point more and changes about 1 in 1,550.
33
+ */
34
+ export const EAGER_WRONG_WORD = { threshold: 3, costWeight: 1.5 };
35
+ const pad = (ids, length) => [
36
+ ...new Array(length - ids.length).fill(PAD),
37
+ ...ids,
38
+ ];
39
+ /** True for pairs like "services" and "service" or "continued" and "continues". */
40
+ export function differsOnlyInEnding(first, second) {
41
+ let shared = 0;
42
+ while (shared < Math.min(first.length, second.length) && first[shared] === second[shared])
43
+ shared++;
44
+ for (let stem = shared; stem >= MIN_STEM; stem--) {
45
+ if (INFLECTION_ENDINGS.has(first.slice(stem)) && INFLECTION_ENDINGS.has(second.slice(stem)))
46
+ return true;
47
+ }
48
+ return false;
49
+ }
50
+ /**
51
+ * Lowercases words, except capitalized words mid-sentence, which become a single name token. A line break ends a
52
+ * sentence, so the end of the paragraph before reads like the sentence before ("Notes" then "Their is..." isn't one run).
53
+ */
54
+ function tokenize(text, startsSentence) {
55
+ const tokens = [];
56
+ for (const raw of text.replaceAll("’", "'").match(TOKEN) ?? []) {
57
+ if (raw === "\n") {
58
+ if (tokens.length > 0 && !SENTENCE_OPENERS.has(tokens.at(-1)))
59
+ tokens.push(".");
60
+ continue;
61
+ }
62
+ const previous = tokens.at(-1);
63
+ const atSentenceStart = previous === undefined ? startsSentence : SENTENCE_OPENERS.has(previous);
64
+ const isPronoun = raw === "I" || raw.startsWith("I'");
65
+ tokens.push(/^[A-Z]/.test(raw) && !atSentenceStart && !isPronoun ? "<name>" : raw.toLowerCase());
66
+ }
67
+ return tokens;
68
+ }
69
+ /** Compares a correctly spelled word with nearby real words, using the words on both sides of it. */
70
+ export class WrongWordChecker {
71
+ options;
72
+ settings;
73
+ ids = new Map();
74
+ targets = [];
75
+ trie;
76
+ neighbors = new Map();
77
+ knownMixups = new Map();
78
+ constructor(options) {
79
+ this.options = options;
80
+ this.settings = options.settings ?? CAREFUL_WRONG_WORD;
81
+ options.vocabulary.forEach((token, index) => {
82
+ this.ids.set(token, index);
83
+ if (index >= SPECIAL_TOKENS && token.length >= 2 && WORD.test(token))
84
+ this.targets.push(index);
85
+ });
86
+ this.trie = new WordTrie(this.targets.map((index) => options.vocabulary[index]));
87
+ for (const line of options.confusions ?? []) {
88
+ const [typed, ...meant] = line.split(" ").map((token) => this.ids.get(token));
89
+ if (typed !== undefined)
90
+ this.knownMixups.set(typed, meant.filter((id) => id !== undefined));
91
+ }
92
+ }
93
+ tokenId(token) {
94
+ if (token === "<name>")
95
+ return NAME;
96
+ if (/^\d/.test(token))
97
+ return NUMBER;
98
+ return this.ids.get(token) ?? UNKNOWN;
99
+ }
100
+ nearbyIds(id) {
101
+ const cached = this.neighbors.get(id);
102
+ if (cached)
103
+ return cached;
104
+ const oneEdit = this.trie
105
+ .search(this.options.vocabulary[id], 1)
106
+ .map((match) => this.targets[match.index])
107
+ .filter((other) => other !== id)
108
+ .sort((first, second) => first - second);
109
+ const found = [...new Set([...(this.knownMixups.get(id) ?? []), ...oneEdit])].slice(0, MAX_NEIGHBORS);
110
+ this.neighbors.set(id, found);
111
+ return found;
112
+ }
113
+ /** Returns nearby words with how well each fits, or null when the word isn't one the model can judge. */
114
+ async compare(textBefore, word, textAfter, onlyKnownMixups = false) {
115
+ const typed = this.ids.get(word.toLowerCase());
116
+ if (typed === undefined || typed < SPECIAL_TOKENS)
117
+ return null;
118
+ const nearby = onlyKnownMixups ? (this.knownMixups.get(typed) ?? []) : this.nearbyIds(typed);
119
+ if (nearby.length === 0)
120
+ return null;
121
+ const scores = await this.runModel(textBefore, textAfter, [[typed, ...nearby]]);
122
+ const mixups = this.knownMixups.get(typed) ?? [];
123
+ return nearby.map((id, index) => {
124
+ const nearbyWord = this.options.vocabulary[id];
125
+ const cost = typoCost(word.toLowerCase(), nearbyWord);
126
+ const isKnownMixup = mixups.includes(id);
127
+ return {
128
+ word: nearbyWord,
129
+ margin: scores[index + 1] - scores[0],
130
+ cost: isKnownMixup ? Math.min(cost, MAX_KNOWN_MIXUP_COST) : cost,
131
+ isKnownMixup,
132
+ };
133
+ });
134
+ }
135
+ /** Scores spelling candidates by how well they fit the text before them. The first word is the misspelling itself. */
136
+ async score(textBefore, words) {
137
+ const ids = [UNKNOWN, ...words.slice(1).map((word) => this.ids.get(word.toLowerCase()) ?? UNKNOWN)];
138
+ const rows = [];
139
+ for (let start = 0; start < ids.length; start += MAX_NEIGHBORS)
140
+ rows.push([PAD, ...ids.slice(start, start + MAX_NEIGHBORS)]);
141
+ const scores = await this.runModel(textBefore, "", rows);
142
+ return ids.map((_, index) => scores[Math.floor(index / MAX_NEIGHBORS) * CANDIDATES + (index % MAX_NEIGHBORS) + 1]);
143
+ }
144
+ /** Runs the model once per row of candidates, returning scores as one flat array of CANDIDATES per row. */
145
+ async runModel(textBefore, textAfter, rows) {
146
+ const before = tokenize(textBefore, true).map((token) => this.tokenId(token));
147
+ const left = before.length >= LEFT_WINDOW ? before.slice(-LEFT_WINDOW) : pad([START, ...before], LEFT_WINDOW);
148
+ const right = pad(tokenize(textAfter, false)
149
+ .slice(0, RIGHT_WINDOW)
150
+ .map((token) => this.tokenId(token))
151
+ .reverse(), RIGHT_WINDOW);
152
+ const tiled = (values) => rows.flatMap(() => values);
153
+ const tensor = (values, width) => new this.options.ort.Tensor("int64", BigInt64Array.from(values, BigInt), [rows.length, width]);
154
+ const output = await this.options.session.run({
155
+ left: tensor(tiled(left), LEFT_WINDOW),
156
+ right: tensor(tiled(right), RIGHT_WINDOW),
157
+ candidates: tensor(rows.flatMap((row) => [...row, ...pad([], CANDIDATES - row.length)]), CANDIDATES),
158
+ });
159
+ return output.scores.data;
160
+ }
161
+ /** Short words and words with an apostrophe are left to the list of known mix-ups ("to" and "too", "it's" and "its"). */
162
+ hasKnownMixups(word, textBefore) {
163
+ const id = this.ids.get(word.toLowerCase());
164
+ if (id === undefined || !this.knownMixups.has(id) || !LISTED_WORD.test(word))
165
+ return false;
166
+ if (word.length > 1 && word === word.toUpperCase())
167
+ return false;
168
+ const isCapitalized = word[0] !== word[0].toLowerCase();
169
+ return !isCapitalized || isSentenceStart(textBefore);
170
+ }
171
+ /** Whether a word is judged at all, and if so whether only against its known mix-ups. Null means leave it alone. */
172
+ judgedAgainst(word, textBefore) {
173
+ if (isCheckable(word, textBefore))
174
+ return "nearby";
175
+ return this.hasKnownMixups(word, textBefore) ? "knownMixups" : null;
176
+ }
177
+ value(item, typed, typedHasMixups) {
178
+ const value = item.margin - this.settings.costWeight * item.cost;
179
+ if (item.isKnownMixup)
180
+ return value + KNOWN_MIXUP_BONUS;
181
+ let penalty = typedHasMixups ? OTHER_WORD_PENALTY : 0;
182
+ if (differsOnlyInEnding(typed, item.word))
183
+ penalty += INFLECTION_PENALTY;
184
+ return value - penalty;
185
+ }
186
+ /** Returns a fix when a nearby word fits the sentence clearly better than the one typed. */
187
+ async check(textBefore, word, textAfter) {
188
+ const judged = this.judgedAgainst(word, textBefore);
189
+ if (!judged)
190
+ return null;
191
+ const nearby = await this.compare(textBefore, word, textAfter, judged === "knownMixups");
192
+ if (!nearby)
193
+ return null;
194
+ const typedHasMixups = nearby.some((item) => item.isKnownMixup);
195
+ const ranked = nearby
196
+ .map((item) => ({ ...item, value: this.value(item, word.toLowerCase(), typedHasMixups) }))
197
+ .sort((first, second) => second.value - first.value);
198
+ const best = ranked[0];
199
+ if (best.value < this.settings.threshold)
200
+ return null;
201
+ return {
202
+ original: word,
203
+ replacement: matchCase(word, best.word),
204
+ confidence: 1 / (1 + Math.exp(-best.margin)),
205
+ usedContext: true,
206
+ };
207
+ }
208
+ }
@@ -0,0 +1,51 @@
1
+ /** One change tinyfix would make to a text. */
2
+ export interface Fix {
3
+ /** Where the word starts in the text, as a string index. */
4
+ from: number;
5
+ /** Where the word ends. */
6
+ to: number;
7
+ /** What was typed. */
8
+ original: string;
9
+ /** What it should be. */
10
+ replacement: string;
11
+ /** A misspelling, or a correctly spelled word that doesn't fit the words around it. */
12
+ kind: "spelling" | "context";
13
+ /** How sure tinyfix is, from 0 to 1. */
14
+ confidence: number;
15
+ }
16
+ export interface LoadOptions {
17
+ /** Where the model and word lists are. Defaults to the tinyfix repo on Hugging Face, at a fixed revision. */
18
+ baseUrl?: string | URL;
19
+ /** Pass false to skip the model: nothing but the word list is downloaded, and only misspellings are fixed. */
20
+ model?: boolean;
21
+ /**
22
+ * How sure tinyfix must be before it changes a correctly spelled word. "balanced" is what offlinepage.app uses and
23
+ * what the published numbers describe. "careful" makes fewer wrong changes and fixes fewer wrong words.
24
+ */
25
+ mode?: "balanced" | "careful";
26
+ /** Where ONNX Runtime's .wasm files are, for apps that host them. Most bundlers find them without this. */
27
+ wasmPaths?: string;
28
+ }
29
+ export interface CheckOptions {
30
+ /** Only words inside this range of the text are checked. The rest of the text is still read as context. */
31
+ range?: readonly [from: number, to: number];
32
+ /** Only these kinds of fix are looked for. Both, by default. */
33
+ kinds?: ReadonlyArray<Fix["kind"]>;
34
+ }
35
+ export interface Tinyfix {
36
+ /** Finds the fixes for a text. They come back in order and never overlap. */
37
+ check(text: string, options?: CheckOptions): Promise<Fix[]>;
38
+ /** Returns the text with the fixes made. Throws if a fix doesn't match the text it's applied to. */
39
+ apply(text: string, fixes: readonly Fix[]): string;
40
+ /** Checks a text and makes every fix. */
41
+ fix(text: string, options?: CheckOptions): Promise<string>;
42
+ /** Never change this word again. Use it when someone undoes a fix, or for words of their own. */
43
+ ignore(word: string): void;
44
+ /** Frees the model. The checker can't be used afterwards. */
45
+ dispose(): Promise<void>;
46
+ }
47
+ /**
48
+ * Loads tinyfix: about 3.5 MB of model and word lists, from Hugging Face unless `baseUrl` says otherwise. Call it
49
+ * once and keep the result.
50
+ */
51
+ export declare function loadTinyfix(options?: LoadOptions): Promise<Tinyfix>;
package/dist/index.js ADDED
@@ -0,0 +1,163 @@
1
+ import * as ort from "onnxruntime-web";
2
+ import { Fixer, WITH_MODEL_WEIGHTS, parseWordList } from "./engine/fixer.js";
3
+ import { WordTrie } from "./engine/trie.js";
4
+ import { CAREFUL_WRONG_WORD, EAGER_WRONG_WORD, WrongWordChecker } from "./engine/wrongWord.js";
5
+ /** The revision of huggingface.co/paperweight/tinyfix this version of the package was tested with. */
6
+ const MODEL_REVISION = "8efdf5cd91b32584f4077aa0e8ecaf48f7651bef";
7
+ const DEFAULT_BASE_URL = `https://huggingface.co/paperweight/tinyfix/resolve/${MODEL_REVISION}/`;
8
+ /**
9
+ * Sure enough to change a misspelled word, with the model helping to choose. offlinepage.app uses the same bar: it
10
+ * fixes more misspellings than the stricter default, and is what lets a doubled-key slip like "sporrt" become "sport".
11
+ */
12
+ const SPELLING_MIN_CONFIDENCE = 0.55;
13
+ /** How much text before a word the engine is given. It reads 16 words back, so this is plenty. */
14
+ const CONTEXT_BEFORE = 600;
15
+ const CONTEXT_AFTER = 200;
16
+ const WORD = /[A-Za-z']+/g;
17
+ /** A fenced code block. An unclosed one runs to the end of the text. */
18
+ const FENCED_CODE = /```[\s\S]*?(?:```|$)/g;
19
+ /** Inline code. An unclosed one runs to the end of its line. */
20
+ const INLINE_CODE = /`[^`\n]*(?:`|$)/gm;
21
+ /** Parts of a web address, an email, a path, a tag or an identifier: not prose. */
22
+ const NOT_PROSE = /[@/\\_#\d]|\.\w/;
23
+ const lines = (text) => text.split("\n").filter(Boolean);
24
+ /** Lets a page draw and handle input between pieces of setup, so loading never freezes it. */
25
+ const nextTask = () => new Promise((resolve) => setTimeout(resolve));
26
+ async function download(base, file) {
27
+ const response = await fetch(new URL(file, base));
28
+ if (!response.ok)
29
+ throw new Error(`tinyfix couldn't load ${file} from ${base} (${response.status}).`);
30
+ return response;
31
+ }
32
+ const downloadText = (base, file) => download(base, file).then((response) => response.text());
33
+ /** The start and end of each stretch of code in a text. */
34
+ function codeRanges(text) {
35
+ return [FENCED_CODE, INLINE_CODE].flatMap((pattern) => [...text.matchAll(pattern)].map((match) => [match.index, match.index + match[0].length]));
36
+ }
37
+ /** The run of non-space characters around a word: "john@form.com" for the "form" inside it. */
38
+ function tokenAround(text, from, to) {
39
+ let start = from;
40
+ while (start > 0 && !/\s/.test(text[start - 1]))
41
+ start--;
42
+ let end = to;
43
+ while (end < text.length && !/\s/.test(text[end]))
44
+ end++;
45
+ return text.slice(start, end);
46
+ }
47
+ function apply(text, fixes) {
48
+ let result = text;
49
+ // Last fix first, so earlier positions stay valid as the text changes length.
50
+ for (const fix of [...fixes].sort((first, second) => second.from - first.from)) {
51
+ if (text.slice(fix.from, fix.to) !== fix.original) {
52
+ throw new Error(`This fix expects “${fix.original}” at ${fix.from}, but the text has “${text.slice(fix.from, fix.to)}”.`);
53
+ }
54
+ result = result.slice(0, fix.from) + fix.replacement + result.slice(fix.to);
55
+ }
56
+ return result;
57
+ }
58
+ /**
59
+ * Loads tinyfix: about 3.5 MB of model and word lists, from Hugging Face unless `baseUrl` says otherwise. Call it
60
+ * once and keep the result.
61
+ */
62
+ export async function loadTinyfix(options = {}) {
63
+ const base = String(options.baseUrl ?? DEFAULT_BASE_URL).replace(/\/?$/, "/");
64
+ const absoluteBase = typeof location === "undefined" ? base : new URL(base, location.href).href;
65
+ const useModel = options.model !== false;
66
+ const [words, modelFiles] = await Promise.all([
67
+ downloadText(absoluteBase, "words.txt"),
68
+ useModel
69
+ ? Promise.all([
70
+ download(absoluteBase, "wrong-word.onnx").then((response) => response.arrayBuffer()),
71
+ downloadText(absoluteBase, "wrong-word-vocab.txt"),
72
+ downloadText(absoluteBase, "wrong-word-confusions.txt"),
73
+ ])
74
+ : null,
75
+ ]);
76
+ let session = null;
77
+ let wrongWord = null;
78
+ if (modelFiles) {
79
+ const [model, vocabulary, confusions] = modelFiles;
80
+ // One thread, so a page needs no cross-origin isolation headers.
81
+ ort.env.wasm.numThreads = 1;
82
+ if (options.wasmPaths)
83
+ ort.env.wasm.wasmPaths = options.wasmPaths;
84
+ // On a page, ONNX Runtime runs the model in its own worker, so starting it doesn't block typing.
85
+ if (typeof document !== "undefined")
86
+ ort.env.wasm.proxy = true;
87
+ session = await ort.InferenceSession.create(new Uint8Array(model), { executionProviders: ["wasm"] });
88
+ await nextTask();
89
+ wrongWord = new WrongWordChecker({
90
+ ort,
91
+ session,
92
+ vocabulary: lines(vocabulary),
93
+ confusions: lines(confusions),
94
+ settings: options.mode === "careful" ? CAREFUL_WRONG_WORD : EAGER_WRONG_WORD,
95
+ });
96
+ }
97
+ await nextTask();
98
+ const wordList = parseWordList(words);
99
+ await nextTask();
100
+ const trie = await WordTrie.inSteps(wordList.map((entry) => entry.word), nextTask);
101
+ const fixer = wrongWord
102
+ ? new Fixer({
103
+ words: wordList,
104
+ trie,
105
+ scorer: wrongWord,
106
+ weights: { ...WITH_MODEL_WEIGHTS, minConfidence: SPELLING_MIN_CONFIDENCE },
107
+ })
108
+ : new Fixer({ words: wordList, trie });
109
+ async function check(text, checkOptions = {}) {
110
+ const [rangeFrom, rangeTo] = checkOptions.range ?? [0, text.length];
111
+ const kinds = checkOptions.kinds ?? ["spelling", "context"];
112
+ const wantsSpelling = kinds.includes("spelling");
113
+ const wantsContext = kinds.includes("context") && wrongWord !== null;
114
+ const code = codeRanges(text);
115
+ const fixes = [];
116
+ // The text up to the word being checked, with the fixes found so far already made. A later word is judged
117
+ // against corrected text, as it would be while typing.
118
+ let fixedSoFar = "";
119
+ let consumed = 0;
120
+ for (const match of text.matchAll(WORD)) {
121
+ const original = match[0];
122
+ const from = match.index;
123
+ const to = from + original.length;
124
+ if (to <= rangeFrom)
125
+ continue;
126
+ if (from >= rangeTo)
127
+ break;
128
+ if (code.some(([start, end]) => from < end && to > start))
129
+ continue;
130
+ if (fixer.wasUndone(original) || NOT_PROSE.test(tokenAround(text, from, to)))
131
+ continue;
132
+ const before = (fixedSoFar + text.slice(consumed, from)).slice(-CONTEXT_BEFORE);
133
+ let correction = wantsSpelling ? await fixer.correct(before, original) : null;
134
+ if (!correction && wantsContext && wrongWord) {
135
+ const lineEnd = text.indexOf("\n", to);
136
+ const after = text.slice(to, Math.min(lineEnd === -1 ? text.length : lineEnd, to + CONTEXT_AFTER));
137
+ correction = await wrongWord.check(before, original, after);
138
+ }
139
+ if (!correction || correction.replacement === original)
140
+ continue;
141
+ fixes.push({
142
+ from,
143
+ to,
144
+ original,
145
+ replacement: correction.replacement,
146
+ kind: correction.usedContext ? "context" : "spelling",
147
+ confidence: Math.max(0, Math.min(1, correction.confidence)),
148
+ });
149
+ fixedSoFar = (fixedSoFar + text.slice(consumed, from)).slice(-CONTEXT_BEFORE) + correction.replacement;
150
+ consumed = to;
151
+ }
152
+ return fixes;
153
+ }
154
+ return {
155
+ check,
156
+ apply,
157
+ fix: async (text, checkOptions) => apply(text, await check(text, checkOptions)),
158
+ ignore: (word) => fixer.rememberUndo(word),
159
+ dispose: async () => {
160
+ await session?.release();
161
+ },
162
+ };
163
+ }
package/package.json CHANGED
@@ -1,6 +1,57 @@
1
1
  {
2
2
  "name": "tinyfix",
3
- "version": "0.0.0-stage",
4
- "stub": true,
5
- "description": "Temporary package placeholder for staged publishing"
6
- }
3
+ "version": "0.1.0",
4
+ "description": "Autocorrect that reads both sides of a word. A 2 MB model that runs in the browser and fixes “form” when you meant “from”.",
5
+ "keywords": [
6
+ "autocorrect",
7
+ "browser",
8
+ "offline",
9
+ "onnx",
10
+ "spellcheck",
11
+ "spelling",
12
+ "typo"
13
+ ],
14
+ "homepage": "https://github.com/cfahlgren1/tinyfix#readme",
15
+ "license": "MIT",
16
+ "author": "Caleb Fahlgren",
17
+ "repository": {
18
+ "type": "git",
19
+ "url": "git+https://github.com/cfahlgren1/tinyfix.git"
20
+ },
21
+ "files": [
22
+ "dist"
23
+ ],
24
+ "type": "module",
25
+ "sideEffects": false,
26
+ "main": "./dist/index.js",
27
+ "types": "./dist/index.d.ts",
28
+ "exports": {
29
+ ".": {
30
+ "types": "./dist/index.d.ts",
31
+ "default": "./dist/index.js"
32
+ }
33
+ },
34
+ "scripts": {
35
+ "build": "tsc -p tsconfig.build.json",
36
+ "check": "tsc --noEmit",
37
+ "lint": "oxlint",
38
+ "format": "oxfmt",
39
+ "format:check": "oxfmt --check",
40
+ "models": "node scripts/download-models.mjs",
41
+ "test": "npm run models && tsx --test test/*.test.ts",
42
+ "prepublishOnly": "npm run check && npm run lint && npm test && npm run build"
43
+ },
44
+ "dependencies": {
45
+ "onnxruntime-web": "^1.20.1"
46
+ },
47
+ "devDependencies": {
48
+ "@types/node": "^22.10.0",
49
+ "oxfmt": "latest",
50
+ "oxlint": "latest",
51
+ "tsx": "^4.19.0",
52
+ "typescript": "~5.9"
53
+ },
54
+ "engines": {
55
+ "node": ">=18"
56
+ }
57
+ }